Scaling Analysis of Interleaved Speech-Text Language Models

The model was presented in the paper Scaling Analysis of Interleaved Speech-Text Language Models.

Paper abstract

Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. They predict that SLMs require much more compute and data compared to text, leading some to question the feasibility of training high-quality SLMs. However, modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. This raises the question - Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper we answer a resounding yes! We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends. We see that under this setup SLMs scale more efficiently with compute. Additionally, our results indicate that the scaling-dynamics are significantly different than textless-SLMs, suggesting one should allocate notably more of the compute budget for increasing model size over training tokens. We also study the role of synthetic data and TextLM model families in unlocking this potential. Results suggest, that our scaled up model achieves comparable performance with leading models on speech semantic metrics while using less compute and data than other approaches.

Model Card for Model ID

This is a Speech Language Model (SLM) trained for generating speech or text continuations over discrete Hubert tokens given speech-text prompts.

Model Details

Model Description

This Speech Language Model, introduced in "Scaling Analysis of Interleaved Speech-Text Language Models", focuses on scaling analysis of interleaved speech-text SLMs. It was fine-tuned from Qwen/Qwen2.5-7B by extending its vocabulary with 500 speech tokens extracted from the 11-th layer of mhubert-25hz.

Developed by: SLP-RL
Model type: SpeechLM
License: MIT
Finetuned from model: Qwen/Qwen2.5-7B

Model Sources

Repository: https://github.com/slp-rl/slamkit
Paper: https://arxiv.org/abs/2504.02398
Demo: https://pages.cs.huji.ac.il/adiyoss-lab/sims/

Uses

This base SpeechLM can be used to generate continuations for speech segments, or cross-modal e.g generate a text contiuation to a speech prompt, or as a base for further tuning. See the SlamKit codebase for more details on usage, and checkout the demo page for some generation examples

Out-of-Scope Use

This model was trained on diverse speech datasets, as such the outputs should not be treated as factual in any way.

How to Get Started with the Model

We refer users to the official repository for full usage explanations - github.

Training Details

We highly encourage users to read the full paper, for full training details.

Compute Infrastructure

Hardware

This model was trained using 8 Nvidia H100 GPUs.

Software

The model was trained using the SlamKit codebase which builds upon 🤗transformers extending it to support easy and efficient training of Speech Language Models.

Citation

BibTeX:

@misc{maimon2025scaling,
      title={Scaling Analysis of Interleaved Speech-Text Language Models}, 
      author={Gallil Maimon and Michael Hassid and Amit Roth and Yossi Adi},
      year={2025},
      eprint={2504.02398},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.02398}, 
}

slprl
/

SIMS-7B