library_name: transformers
datasets:
- FBK-MT/Speech-MASSIVE
language:
- pl
metrics:
- wer
- bleu
base_model:
- openai/whisper-tiny
pipeline_tag: automatic-speech-recognition
Model Card
Model Details
Model Description
This model is a fine-tuned version of OpenAI's Whisper-Tiny ASR model, optimized for transcribing Polish voice commands. The fine-tuning process utilized the MASSIVE Speech dataset to enhance the model's performance on Polish utterances. The Whisper-Tiny model is a transformer-based encoder-decoder architecture, pre-trained on 680,000 hours of labeled speech data.
- Developed by: gs224
- Language(s) (NLP): Polish
- Finetuned from model: Whisper-tiny
Link to the training code: https://github.com/gs224/Fine-tuning-Whisper-for-Polish-voice-commands
Uses
The model can be used for automatic transcription of Polish speech-to-text tasks, including voice command recognition.
Out-of-Scope Use
The model may not perform well on languages or domains it was not fine-tuned for, and it is not suitable for sensitive applications requiring very high accuracy.
Bias, Risks, and Limitations
The fine-tuning was performed on a relatively small subset of Polish voice data with limited epochs, leading to potential underperformance in certain dialects or accents. The presence of capital letters and punctuation in the ground-truth transcription may affect the Word Error Rate (WER) score.
Recommendations
Future improvements could include training on larger datasets, more diverse utterances, and addressing case sensitivity and punctuation in ground-truth labels.
Training Details
Training Data
https://huggingface.co/datasets/FBK-MT/Speech-MASSIVE-test
Evaluation
Word Error Rate (WER)
Testing Data, Factors & Metrics
Metrics
WER, a typical metrics for ASR.
Results
Word Error Rate on the test set:
| Base model | Fine-tuned model |
|---|---|
| 0.8435 | 0.3176 |
Example sentences:
| Reference | Base model | Fine-tuned model |
|---|---|---|
| wyślij maila do mojego brata i przypomnij o rocznicy ślubu | wysli myę latą mojego biata i przypamni o nici ślubu | wyślij maila do mojego bryata i przypomnij mi o lepszy ślubu |
| przypomnij mi o jutrzejszym spotkaniu godzinę wcześniej | przypomnij mi o jutrzejszym spotkaniu godzinę wcześniej | przypomnij mi o jutrzejszym spotkaniu godzina wcześniej |
| graj plejlistę boba dylana | gra i play listę boba dylana | graj playlistę boba delana |
| graj ale jazz autorki sanah | grei, al het rust autoorkisana | graj ale jazz autorki sanah |
| olly posłuchajmy sto jeden i trzy f. m. | oli posłuchajmy sto jeden i trzefam | olly posłuchaj we z to jeden i trzy f. m. |