|
--- |
|
license: cc-by-nc-4.0 |
|
--- |
|
|
|
# SpireLM |
|
Spire is a 7B parameter decoder-only model with strong abilities in machine translation, automatic speech recognition, and speech translation. [SpireBase](https://huggingface.co/utter-project/SpireBase) was created by applying speech-centric continued pretraining to [TowerBase-7B-v0.1](https://huggingface.co/Unbabel/TowerBase-7B-v0.1), which was itself created by applying continued pretraining to [Llama 2](https://huggingface.co/meta-llama/Llama-2-7b). |
|
|
|
## Model Checkpoints |
|
We release our checkpoints through Hugging Face. All of our models can be loaded as `LlamaForCausalLM` instances, allowing inference to be performed with [vLLM](https://github.com/vllm-project/vllm). For further details on the models, check [the paper](https://arxiv.org/abs/2503.10620). |
|
|
|
| Model | Path | |
|
| ----- | ---- | |
|
| SpireBase | [utter-project/SpireBase](https://huggingface.co/utter-project/SpireBase) | |
|
| SpireFull | [utter-project/SpireFull](https://huggingface.co/utter-project/SpireFull) | |
|
| SpireNoBlocks | [utter-project/SpireNoBlocks](https://huggingface.co/utter-project/SpireNoBlocks) | |
|
| SpireNoPseudo | [utter-project/SpireNoBlocks](https://huggingface.co/utter-project/SpireNoPseudo) | |
|
| TowerFull | [utter-project/TowerFull](https://huggingface.co/utter-project/TowerFull) | |
|
|
|
## Tokenizing Speech |
|
The core of our approach to speech is *discretization* - continuous speech signals are converted into sequences of tokens, which can then be processed alongside text. Our discretization system consists of a few steps: |
|
|
|
1. HuBERT Large ([fairseq download](https://dl.fbaipublicfiles.com/hubert/hubert_large_ll60k.pt)) converts 16kHz .wav files into into a sequence of feature vectors, one for each 20ms frame. We use the representations from layer 22. |
|
2. Our k-means model ([download](https://huggingface.co/utter-project/SpireKMeans/resolve/main/kmeans_model)) maps each frame to one of 5000 clusters. |
|
3. The sequences of cluster IDs are deduplicated, such that consecutive frames with the same label are collapsed into a single token. This usually shortens the sequence length by about 30%. |
|
|
|
The `spire` package implements this pipeline. Assuming you have downloaded both of these files, you can use it like so: |
|
|
|
``` |
|
from datasets import load_dataset |
|
from spire.dsus import Labeler |
|
from spire.utils import fix_fleurs_path |
|
|
|
fleurs = load_dataset("google/fleurs", "en_us") |
|
wav = fix_fleurs_path(fleurs["validation"][29], "validation") |
|
|
|
labeler = Labeler("hubert_large_ll60k.pt", "kmeans_model") |
|
speech_tokens = labeler.label(wav) |
|
print(speech_tokens) |
|
``` |
|
|
|
The output will not be very readable, as it consists of a sequence of Unicode [private use area](https://en.wikipedia.org/wiki/Private_Use_Areas) characters. However, these characters are known to the Spire tokenizer and can be combined with text: |
|
|
|
TODO: add ASR/ST examples with this sequence |
|
|
|
## Reproducing our Inference Results |
|
TODO: ducttape example |
|
|
|
## Reproducing our Training |
|
|
|
## Citation |
|
If you use Spire, please cite our work: |
|
``` |
|
@misc{spire, |
|
title={From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM}, |
|
author={Kshitij Ambilduke and Ben Peters and Sonal Sannigrahi and Anil Keshwani and Tsz Kin Lam and Bruno Martins and Marcely Zanon Boito and André F. T. Martins}, |
|
year={2025}, |
|
eprint={2503.10620}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2503.10620} |
|
} |
|
``` |
