CoRal-project/roest-wav2vec2-315m-v2

Røst-wav2vec2-315m-v2

This is a Danish state-of-the-art speech recognition model, trained as part of the CoRal project by Alvenir.

This repository contains a Wav2Vec2 model trained on the CoRal-v2 dataset soon to be released. The CoRal-v2 dataset includes a rich variety of Danish conversational and read-aloud data, distributed across diverse age groups, genders, and dialects. The model is designed for automatic speech recognition (ASR).

Try it out in our interactive demo!

Quick Start

Start by installing the required libraries:

$ pip install transformers kenlm pyctcdecode

Next you can use the model using the transformers Python package as follows:

>>> from transformers import pipeline
>>> audio = get_audio()  # 16kHz raw audio array
>>> transcriber = pipeline(model="CoRal-project/roest-wav2vec2-315m-v2")
>>> transcriber(audio)
{'text': 'your transcription'}

---

Transcription Examples

Explore the following audio samples along with their transcriptions and accuracy metrics. Each example showcases the model's performance with different Danish dialects.

Example 1 - Vestjysk Dialect

Audio Sample:

Model Transcription:
det blev til yderlig ti mål i den første sæson på trods af en position som back

Target Transcription:
det blev til yderligere ti mål i den første sæson på trods af en position som back

• Character Error Rate (CER): 3.7%
• Word Error Rate (WER): 5.9%

Example 2 - Sønderjysk Dialect

Audio Sample:

Model Transcription:
en arkitektoniske udformning af pladser forslagene iver benzen

Target Transcription:
den arkitektoniske udformning af pladsen er forestået af ivar bentsen

• Character Error Rate (CER): 20.3%
• Word Error Rate (WER): 60.0%

Example 3 - Nordsjællandsk Dialect

Audio Sample:

Model Transcription:
østrig og ungarn samarbejder om søen gennem den østrigske og ungarske vandkommission

Target Transcription:
østrig og ungarn samarbejder om søen gennem den østrigske og ungarske vandkommission

• Character Error Rate (CER): 0.0%
• Word Error Rate (WER): 0.0%

Example 4 - Lollandsk Dialect

Audio Sample:

Model Transcription:
det er produceret af thomas helme og indspillede i easy sound recording studio i københavn

Target Transcription:
det er produceret af thomas helmig og indspillet i easy sound recording studio i københavn

• Character Error Rate (CER): 4.4%
• Word Error Rate (WER): 13.3%

---

Model Details

Wav2Vec2 is a state-of-the-art model architecture for speech recognition, leveraging self-supervised learning from raw audio data. The pre-trained Wav2Vec2-XLS-R-300M has been fine-tuned for automatic speech recognition with the CoRal-v2 dataset dataset to enhance its performance in recognizing Danish speech with consideration to different dialects. The model was trained for 30K steps using the training setup in the CoRaL repository by running:

python src/scripts/finetune_asr_model.py \
  model=wav2vec2-small \
  max_steps=30000 \
  datasets.coral_conversation_internal.id=CoRal-project/coral-v2 \
  datasets.coral_readaloud_internal.id=CoRal-project/coral-v2

The model is evaluated using a Language Model (LM) as post-processing. The utilized LM is the one trained and used by CoRal-project/roest-wav2vec2-315m-v1.

The model was trained on the CoRal-v2 dataset, including both the conversational and read-aloud subset. This dataset consists of Danish speech across a variety of dialects, age groups and gender distinctions. Note that the dataset, and thus also this model, is licensed under a custom license, adapted from OpenRAIL-M, which allows commercial use with few restrictions (speech synthesis and biometric identification) - see license.

---

Evaluation

The model was evaluated using the following metrics:

• Character Error Rate (CER): The percentage of characters incorrectly transcribed.
• Word Error Rate (WER): The percentage of words incorrectly transcribed.

Conversational CoRal Performance

The model was firstly evaluated on a tentative version of the coral-v2 conversation dataset.

The results are tentative as the test set only includes 5 unique speakers, of which 4 are women. The test set includes 2 speakers with 'Fynsk' dialect, 1 with 'Sønderjysk', 1 with 'Non-native' and 1 'Nordjysk'.

Note that the high generalization error on conversation data for models trained on read-aloud data is still being analyzed.

Model	Number of parameters	Finetuned on data of type	CoRal-v2::conversation CER	CoRal-v2::conversation WER
CoRal-project/roest-wav2vec2-1B-v2	1B	Read-aloud and conversation	23.9%	36.7%
CoRal-project/roest-wav2vec2-315M-v2 (This model)	315M	Read-aloud and conversation	24.2%	37.7%
CoRal-project/roest-whisper-large-v1	1540M	Read-aloud	138%	121%
CoRal-project/roest-wav2vec2-315m-v1	315M	Read-aloud	123%	80.5%
mhenrichsen/hviske-v2	1540M	Read-aloud	78.2%	72.6%
openai/whisper-large-v3	1540M	-	46.4 %	57.4%

Read-aloud CoRal Performance

Model	Number of parameters	Finetuned on data of type	CoRal CER	CoRal WER
CoRal-project/roest-wav2vec2-1B-v2	1B	Read-aloud and conversation	6.5% ± 0.2%	16.4% ± 0.4%
CoRal-project/roest-wav2vec2-315M-v2 (This model)	315M	Read-aloud and conversation	6.5% ± 0.2%	16.3% ± 0.4%
CoRal-project/roest-whisper-large-v1	1540M	Read-aloud	4.3% ± 0.2%	10.4% ± 0.3%
CoRal-project/roest-wav2vec2-315M-v1	315M	Read-aloud	6.6% ± 0.2%	17.0% ± 0.4%
mhenrichsen/hviske-v2	1540M	Read-aloud	4.7% ± 0.2%	11.8% ± 0.3%
openai/whisper-large-v3	1540M	-	11.4% ± 0.3%	28.3% ± 0.6%

OBS! Benchmark for hviske-v2 has been re-evaluated and the confidence interval is larger than reported in the model card.

Detailed CER scores in % of evaluation across demographics on the CoRal test data

Category	Røst-whisper-large-v1	Røst-wav2vec2-315m-v1	Røst-wav2vec2-315m-v2	Røst-wav2vec2-1B-v2
female	5.1	7.4	7.2	7.3
male	3.6	5.8	5.7	5.8
0-25	3.4	5.4	5.3	5.1
25-50	4.0	6.2	6.0	5.7
50+	5.0	7.5	7.4	7.8
Bornholmsk	3.8	6.8	6.1	6.2
Fynsk	5.1	7.4	7.2	6.9
Københavnsk	1.9	3.3	3.2	3.0
Non-native	4.8	7.8	7.5	7.3
Nordjysk	1.6	2.6	2.8	2.6
Sjællandsk	3.0	4.4	4.5	3.9
Sydømål	4.1	6.4	6.4	6.5
Sønderjysk	8.8	11.9	11.6	12.6
Vestjysk	6.4	10.1	9.8	10.5
Østjysk	2.6	4.0	4.1	3.8
Overall	4.3	6.6	6.5	6.5

Detailed WER scores in % of evaluation across demographics on the CoRal test data

Category	Røst-whisper-large-v1	Røst-wav2vec2-315m-v1	Røst-wav2vec2-315m-v2	Røst-wav2vec2-1B-v2
female	11.5	18.5	17.7	17.8
male	9.4	15.5	14.9	15.0
0-25	9.0	14.7	14.0	13.7
25-50	10.1	16.6	15.8	15.3
50+	11.3	18.2	17.7	18.5
Bornholmsk	9.8	17.7	15.7	16.4
Fynsk	12.1	18.3	17.7	16.7
Københavnsk	5.9	10.2	10.0	9.5
Non-native	12.2	20.9	19.4	19.4
Nordjysk	4.5	7.7	7.5	7.3
Sjællandsk	7.6	12.6	12.7	11.0
Sydømål	10.0	14.9	15.3	14.4
Sønderjysk	17.5	26.0	25.4	27.8
Vestjysk	15.0	26.3	25.2	26.7
Østjysk	7.5	11.7	11.3	10.8
Overall	10.4	17.0	16.3	16.4

Experiments with Røst-wav2vec2 with and without language model

The inclusion of a post-processing language model can affect the performance significantly. The Røst-v1 and Røst-v2 models are using the same Language Model (LM). The utilized LM is the one trained and used by CoRal-project/roest-wav2vec2-315m-v1.

Model	Number of parameters	Finetuned on data of type	Postprocessed with Language Model	CoRal CER	CoRal WER
CoRal-project/roest-wav2vec2-1B-v2	1B	Read-aloud and conversation	Yes	6.5% ± 0.2%	16.4% ± 0.4%
CoRal-project/roest-wav2vec2-1B-v2	1B	Read-aloud and conversation	No	8.1% ± 0.2%	23.9% ± 0.4%
CoRal-project/roest-wav2vec2-315M-v2 (This model)	315M	Read-aloud and conversation	Yes	6.5% ± 0.2%	16.3% ± 0.4%
CoRal-project/roest-wav2vec2-315M-v2	315M	Read-aloud and conversation	No	8.2% ± 0.2%	25.1% ± 0.4%
CoRal-project/roest-wav2vec2-315m-v1	315M	Read-aloud	Yes	6.6% ± 0.2%	17.0% ± 0.4%
CoRal-project/roest-wav2vec2-315m-v1	315M	Read-aloud	No	8.6% ± 0.2%	26.3% ± 0.5%

Here are the results of the Røst-Wav2Vec2-315m models on different Danish dialects in the test set:

	Røst-v1		Røst-v1		Røst-v2		Røst-v2
LM	No		Yes		No		Yes
-------------	---------	---------	---------	---------	---------	---------	---------	---------
Dialect	CER (%)	WER (%)	CER (%)	WER (%)	CER (%)	WER (%)	CER (%)	WER (%)
Vestjysk	12.7	37.1	10.1	26.3	12.2	36.3	9.82	25.2
Sønderjysk	14.7	37.8	11.9	26.0	14.2	36.2	11.6	25.4
Bornholmsk	9.32	29.9	6.79	17.7	8.08	26.7	6.12	15.7
Østjysk	5.51	18.7	3.97	11.7	5.39	18.0	4.06	11.3
Nordjysk	3.86	13.6	2.57	7.72	3.80	13.5	2.75	7.51
Københavnsk	5.27	18.8	3.31	10.2	5.02	17.7	3.20	9.98
Fynsk	9.41	28.6	7.43	18.3	8.86	27.0	7.20	17.7
Non-native	10.6	33.2	7.84	20.9	10.0	31.6	7.46	19.4
Sjællandsk	5.82	19.5	4.44	12.6	5.70	18.6	4.48	12.7
Sydømål	7.09	20.7	6.38	14.9	6.96	20.4	6.44	15.3

Performance on Other Datasets

The model was also tested against other datasets to evaluate generalizability:

	Røst-whisper-large-v1		Røst-wav2vec2-315M-v1		Røst-wav2vec2-315M-v2		Røst-wav2vec2-1B-v2
Evaluation Dataset	WER %	CER %	WER %	CER %	WER %	CER %	WER %	CER %
CoRal	10.4	4.3	17.0	6.6	16.3	6.5	16.4	6.5
NST-da	29.8	14.5	29.7	13.9	26.1	11.9	12.4	4.9
CommonVoice17	15.6	8.2	16.7	6.6	14.4	5.4	26.3	10.9
Fleurs-da_dk	12.6	5.1	16.6	6.3	15.6	6.1	13.7	5.5

OBS! The vocab used for training incudes numerals (0,1,2,..,9), which are translated to text in a post-processing step. If the model misses spaces the numbers are interpreted as one, which especially affects the NST score as this dataset contains many numerals.

---

Note on comparing Whisper and Wav2Vec2 models

The Whisper models detailed in this model card exhibit significantly lower Character Error Rates (CER) and Word Error Rates (WER) compared to the Wav2Vec2 models. Whisper utilizes a transformer-based architecture with additional layers that enhance contextual understanding. In contrast, Wav2Vec2 models employ shorter context windows that focus on sound prediction. The Røst-Wav2Vec2 models incorporate a straightforward language model during post-processing, which addresses errors based on statistical language patterns. Introducing a more complex, contextual post-processing language model might enable a better comparison between these model types, which the CoRal project plans to explore in future releases.

The Røst-Whisper model excels in read-aloud data, leveraging its embedded contextual framework to achieve more robust recognition within this context. However, Wav2Vec2 models appear to generalize more effectively across various speech recognition tasks, whereas Whisper models incur higher error rates in conversational data. It’s important to note that the CoRal-v2 conversation dataset, being tentative and featuring limited speaker diversity, might influence these results.

---

Training curves

---

Creators and Funders

This model has been trained and the model card written by Marie Juhl Jørgensen and Søren Vejlgaard Holm at Alvenir.

The CoRal project is funded by the Danish Innovation Fund and consists of the following partners:

• Alexandra Institute
• University of Copenhagen
• Agency for Digital Government
• Alvenir
• Corti

We would like specifically to thank Dan Saattrup Nielsen, Alexandra Institute for (among other things) the repository work and Simon Leminen Madsen, Alexandra Institute for modelling work.

Citation

@misc{roest-wav2vec2-315m-v2,
  author    = {Marie Juhl Jørgensen, Søren Vejlgaard Holm, Martin Carsten Nielsen, Dan Saattrup Nielsen, Sif Bernstorff Lehmann, Simon Leminen Madsen and Torben Blach},
  title     = {Røst-wav2vec-315m-v2: A Danish state-of-the-art speech recognition model trained on varied demographics and dialects},
  year      = {2025},
  url       = {https://huggingface.co/CoRal-project/roest-wav2vec2-315m-v2},
}