Whisper-Small-hindi

This is a fine-tuned version of openai/whisper-small, fine-tuned on the following datasets:

Shrutilipi (AI4Bharat): Shrutilipi is a labelled ASR corpus obtained by mining parallel audio and text pairs at the document scale from All India Radio news bulletins for 12 Indian languages - Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu. The corpus has over 6400 hours of data across all languages. Out of which hindi is ~ 1600 hours
IITM Madras SpringLab (CC BY 4.0 License): This data was collected on payment basis using the following vendors -- Mediscribe India, Desicrew, and Crescendo. Preliminary checking of quality of transcriptions was done by our partners at KL University as well as by SPRING Lab members. The data consists mostly of mock conversations as well as monolgues on different topics.

The model is trained on around 2500 hours of hindi speech & optimized for ASR tasks in hindi, with a particular focus on high-accuracy transcription.

How to use

The Whisper model is intrinsically designed to work on audio samples of up to 30s in duration. However, by using a chunking algorithm, it can be used to transcribe audio samples of up to arbitrary length. This is possible through Transformers pipeline method. Chunking is enabled by setting chunk_length_s=30 when instantiating the pipeline. With chunking enabled, the pipeline can be run with batched inference. It can also be extended to predict sequence level timestamps by passing return_timestamps=True:

>>> import torch
>>> from transformers import pipeline
>>> from datasets import load_dataset

>>> device = "cuda:0" if torch.cuda.is_available() else "cpu"

>>> asr_pipe = pipe(
>>>     "automatic-speech-recognition",
>>>     model="collabora/whisper-small-hindi",
>>>     chunk_length_s=30,
>>>     device=device
>>> )

>>> ds = load_dataset("mozilla-foundation/common_voice_11_0", "hi", split="validation")
>>> sample = ds[0]["audio"]
>>> prediction = asr_pipe(sample.copy(), batch_size=8, return_timestamps=True)["text"]
हमने उस उम्मीदवार को चुना।

Intended Use

The model is designed for high quality transcription in Hindi.
And is suitable for academic use in ASR related tasks.

Limitations

May not perform well on noisy or low-quality audio.
Focused primarily on Hindi.

Model Performance

Whisper Normalization is counter-productive for hindi since it takes the meaning out of a sentence for e.g. consider the hindi phrase:

'राजनीतिज्ञों ने कहा कि उन्होंने निर्णायक मत को अनावश्यक रूप से निर्धारित करने के लिए अफगान संविधान में काफी अस्पष्टता पाई थी'

After whisper normalization:

'र जन त जञ न कह क उनह न न रण यक मत क अन वशयक रप स न रध र त करन क ल ए अफग न सव ध न म क फ असपषटत प ई थ'

So, we use indic-normalization for evaluation. Indic-norm produces the below output:

'राजनीतिज्ञों ने कहा कि उन्होंने निर्णायक मत को अनावश्यक रूप से निर्धारित करने के लिए अफगान संविधान में काफी अस्पष्टता पाई थी'

openai-whisper/small baseline results on google/fleurs -- hindi:

Word Error Rate (WER) with whisper norm: 67.37 % 
Word Error Rate (WER) with indic norm: 89.73 %

The model achieves the following benchmarks on the held out test set google/fleurs -- hindi:

Word Error Rate (WER) with whisper norm: 10.11 % 
Word Error Rate (WER) with indic norm: 17.35 %

Indic normalization retains diacritics and complex characters in Hindi text, which can increase the Word Error Rate (WER) when compared to Whisper's default normalization but produces more semantically accurate transcriptions.

Acknowledgments

We thank the contributors and organizations behind the datasets:

AI4Bharat for the Shrutilipi dataset.
IIT Madras SpringLab for their springx-hindi dataset.
IndicNLP Library by Anoop Kunchukuttan for providing normalization tools that were crucial for evaluation.

BibTeX entry and citation info

Model Citation

@misc{whisper-small-hindi,
  title = {Whisper-Small Fine-Tuned on Hindi},
  author = {Collabora Ltd.},
  year = {2025},
  publisher = {Hugging Face},
  note = {Fine-tuned using Shrutilipi and IITM Madras SpringLab datasets},
  howpublished = {\url{https://huggingface.co/collabora/whisper-small-hindi/}},
}

IndicNLP Library Citation

@misc{kunchukuttan2020indicnlp,
author = "Anoop Kunchukuttan",
title = "{The IndicNLP Library}",
year = "2020",
howpublished={\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}
}

AI4Bharat - Shrutilipi dataset

@misc{https://doi.org/10.48550/arxiv.2208.12666,
  doi = {10.48550/ARXIV.2208.12666},
  url = {https://arxiv.org/abs/2208.12666},
  author = {Bhogale, Kaushal Santosh and Raman, Abhigyan and Javed, Tahir and Doddapaneni, Sumanth and Kunchukuttan, Anoop and Kumar, Pratyush and Khapra, Mitesh M.},
  title = {Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}