FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs

FuseCodec proposes a speech tokenization framework that fuses semantic and contextual information into neural codecs using three novel strategies—latent fusion, global supervision, and temporal alignment—to enable state-of-the-art discrete speech representations for tasks like zero-shot TTS.

➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐨𝐮𝐫 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐓𝐨𝐤𝐞𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤:

🧠 𝑳𝒂𝒕𝒆𝒏𝒕 𝑹𝒆𝒑𝒓𝒆𝒔𝒆𝒏𝒕𝒂𝒕𝒊𝒐𝒏 𝑭𝒖𝒔𝒊𝒐𝒏 (FuseCodec-Fusion):
FuseCodec fuses semantic representations from self-supervised speech models and contextual representations from pretrained language models directly into the encoder’s latent space using cross-modal multi-head attention followed by additive fusion with stochastic dropout, enabling robust and unified token learning without modifying the core codec architecture.

📡 𝑮𝒍𝒐𝒃𝒂𝒍 𝑺𝒆𝒎𝒂𝒏𝒕𝒊𝒄-𝑪𝒐𝒏𝒕𝒆𝒙𝒕𝒖𝒂𝒍 𝑺𝒖𝒑𝒆𝒓𝒗𝒊𝒔𝒊𝒐𝒏 (FuseCodec-Distill):
FuseCodec supervises the quantized RVQ tokens by aligning them with global pooled representations from both modalities using cosine-based timestep-wise distillation loss, which enhances temporal coherence, linguistic grounding, and perceptual quality across the quantized space, outperforming existing tokenizers in WER, PESQ, and UTMOS.

🧭 𝑻𝒆𝒎𝒑𝒐𝒓𝒂𝒍𝒍𝒚 𝑨𝒍𝒊𝒈𝒏𝒆𝒅 𝑪𝒐𝒏𝒕𝒆𝒙𝒕𝒖𝒂𝒍 𝑺𝒖𝒑𝒆𝒓𝒗𝒊𝒔𝒊𝒐𝒏 (FuseCodec-ContextAlign):
By dynamically aligning contextual embeddings to speech tokens through a content-similarity-based windowed matching algorithm, FuseCodec enforces fine-grained cross-modal alignment via cosine supervision, improving intelligibility and interpretability while preserving local linguistic structure without degrading audio quality.