You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

NeuCodec 🎧

Click the image above to see NeuCodec in action on Youtube!

Created by Neuphonic - building faster, smaller, on-device voice AI

A lightweight neural codec that encodes audio at just 0.8 kbps - perfect for researchers and builders who need something that just works for training high quality text-to-speech models.

Key Features

🔊 Low bit-rate compression - a speech codec that compresses and reconstructs audio with near-inaudible reconstruction loss
🎼 Upsamples from 16kHz → 24kHz
🌍 Ready for real-world use - train your own SpeechLMs without needing to build your own codec
🏢 Commercial use permitted - use it in your own tools or products
📊 Released with large pre-encoded datasets - we’ve compressed Emilia-YODAS from 1.7TB to 41GB using NeuCodec, significantly reducing the compute requirements needed for training

Model Details

NeuCodec is a Finite Scalar Quantisation (FSQ) based 0.8kbps audio codec for speech tokenization. It takes advantage of the following features:

FSQ quantisation resulting in a single codebook, making it ideal for downstream modeling with Speech Language Models.
Trained with CC data such that there are no Non-Commercial data restrictions.
At 50 tokens/sec and 16 bits per token, the overall bit-rate is 0.8kbps.
The codec takes in 16kHz input and outputs 24kHz using an upsampling decoder.
The FSQ encoding scheme allows for bit-level error resistance suitable for unreliable and noisy channels.

NeuCodec is largely based on extending the work of X-Codec2.0.

Developed by: Neuphonic
Model type: Neural Audio Codec
License: apache-2.0
Repository: https://github.com/neuphonic/neucodec
Paper: arXiv
Pre-encoded Datasets:
- Emilia-YODAS-EN
- More coming soon!

Get Started

Use the code below to get started with the model.

To install from pypi in a dedicated environment, using Python 3.10 or above:

conda create -n neucodec python=3.10
conda activate neucodec
pip install neucodec

Then, to use in python:

import librosa
import torch
import torchaudio
from torchaudio import transforms as T
from neucodec import NeuCodec
 
model = NeuCodec.from_pretrained("neuphonic/neucodec")
model.eval().cuda()   
 
y, sr = torchaudio.load(librosa.ex("libri1"))
if sr != 16_000:
    y = T.Resample(sr, 16_000)(y)[None, ...] # (B, 1, T_16)

with torch.no_grad():
    fsq_codes = model.encode_code(y)
    # fsq_codes = model.encode_code(librosa.ex("libri1")) # or directly pass your filepath!
    print(f"Codes shape: {fsq_codes.shape}")  
    recon = model.decode_code(fsq_codes).cpu() # (B, 1, T_24)

torchaudio.save("reconstructed.wav", recon[0, :, :], 24_000)