added COMET scores

Signed-off-by: Kunal Dhawan <[email protected]>

README.md

@@ -266,11 +266,11 @@ img {
 </style>
 
 ## Description:
-NVIDIA NeMo Canary [1] is a family of multilingual multi-tasking models that achieves state-of-the art performance on multiple speech benchmarks. With 883 million parameters and running at more then 900 RTFx (on open-asr-leaderboard sets), canary-1b-flash supports automatic speech-to-text recognition (ASR) in 4 languages (English, German, French, Spanish) and translation from English to German/French/Spanish and from German/French/Spanish to English with or without punctuation and capitalization (PnC). In addition to this, canary-1b-flash also supports functionality for word-level and segment-level timestamps for English, German, French, and Spanish. This model is released under the permissive CC-BY-4.0 license and is available for commercial use.
+NVIDIA NeMo Canary Flash [1] is a family of multilingual multi-tasking models based on Canary architecture [2] that achieves state-of-the art performance on multiple speech benchmarks. With 883 million parameters and running at more then 900 RTFx (on open-asr-leaderboard datasets), canary-1b-flash supports automatic speech-to-text recognition (ASR) in 4 languages (English, German, French, Spanish) and translation from English to German/French/Spanish and from German/French/Spanish to English with or without punctuation and capitalization (PnC). In addition to this, canary-1b-flash also supports functionality for word-level and segment-level timestamps for English, German, French, and Spanish. This model is released under the permissive CC-BY-4.0 license and is available for commercial use.
 
 
 ## Model Architecture:
-Canary is an encoder-decoder model with FastConformer [2] Encoder and Transformer Decoder [3]. With audio features extracted from the encoder, task tokens such as \<target language\>, \<task\>, \<toggle timestamps\> and \<toggle PnC\> are fed into the Transformer Decoder to trigger the text generation process. Canary uses a concatenated tokenizer [4] from individual SentencePiece [5] tokenizers of each language, which makes it easy to scale up to more languages. The canary-1b-flash model has 32 encoder layers and 4 layers of decoder layers, leading to a total of 883M parameters. For more details about the architecture, please refer to [9].
+Canary is an encoder-decoder model with FastConformer [3] Encoder and Transformer Decoder [4]. With audio features extracted from the encoder, task tokens such as \<target language\>, \<task\>, \<toggle timestamps\> and \<toggle PnC\> are fed into the Transformer Decoder to trigger the text generation process. Canary uses a concatenated tokenizer [5] from individual SentencePiece [6] tokenizers of each language, which makes it easy to scale up to more languages. The canary-1b-flash model has 32 encoder layers and 4 layers of decoder layers, leading to a total of 883M parameters. For more details about the architecture, please refer to [1].
 
 ## NVIDIA NeMo
 
@@ -357,7 +357,7 @@ output = canary_model.transcribe(
 
 ## Software Integration:
 **Runtime Engine(s):** 
-* NeMo - 2.1.0 or higher <br>
+* NeMo - 2.3.0 or higher <br>
 
 **Supported Hardware Microarchitecture Compatibility:** <br>
 * [NVIDIA Ampere] <br>
@@ -452,11 +452,11 @@ Noise Robustness:
 * [Librispeech](https://www.openslr.org/12)
 
 Model Fairness:
-* [Casual Conversations Dataset](https://arxiv.org/pdf/2104.02821)
+* [Casual Conversations Dataset](https://arxiv.org/abs/2104.02821)
 
 ## Training
 
-canary-1b-flash is trained using the NVIDIA NeMo toolkit [6] for a total of 200K steps with 2D bucketing [9] and optimal batch sizes set using OOMptimizer [7].The model is trained on 128 NVIDIA A100 80GB GPUs. 
+canary-1b-flash is trained using the NVIDIA NeMo toolkit [7] for a total of 200K steps with 2D bucketing [1] and optimal batch sizes set using OOMptimizer [8].The model is trained on 128 NVIDIA A100 80GB GPUs. 
 The model can be trained using this [example script](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_multitask/speech_to_text_aed.py) and [base config](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/speech_multitask/fast-conformer_aed.yaml).
 
 The tokenizers for these models were built using the text transcripts of the train set with this [script](https://github.com/NVIDIA/NeMo/blob/main/scripts/tokenizers/process_asr_text_tokenizer.py).
@@ -470,7 +470,7 @@ The tokenizers for these models were built using the text transcripts of the tra
 
 ## Performance
 
-In both ASR and AST experiments, predictions were generated using beam search with width 5 and length penalty 1.0.
+For ASR and AST experiments, predictions were generated using greedy decoding. Note that utterances shorter than 1 second are symmetrically zero-padded upto 1 second during evaluation.  
 
 ### ASR Performance (w/o PnC) 
 
@@ -480,67 +480,84 @@ WER on [HuggingFace OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/
 
 | **Version** | **Model**     | **RTFx**   | **AMI**   | **GigaSpeech**   | **LS Clean**   | **LS Other**   | **Earnings22**   | **SPGISpech**   | **Tedlium**   | **Voxpopuli**   |
 |:---------:|:-----------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|
-| 2.2.0  | canary-1b-flash | 928.19 | 13.08 | 9.88 | 1.48 | 2.87 | 12.77 | 1.95 | 3.09 | 5.64 |
+| 2.3.0  | canary-1b-flash | 928.19 | 13.08 | 9.88 | 1.48 | 2.87 | 12.77 | 1.95 | 3.09 | 5.64 |
 
 WER on [MLS](https://huggingface.co/datasets/facebook/multilingual_librispeech) test set:
 
 | **Version** | **Model**  | **De**   | **Es**   | **Fr**   |
 |:---------:|:-----------:|:------:|:------:|:------:|
-| 2.2.0   | canary-1b-flash | 4.36 | 2.69 | 4.47 |
+| 2.3.0   | canary-1b-flash | 4.36 | 2.69 | 4.47 |
 
 WER on [MCV-16.1](https://commonvoice.mozilla.org/en/datasets) test set:
 | **Version** | **Model**  |  **En**   | **De**   | **Es**   | **Fr**   |
 |:---------:|:-----------:|:------:|:------:|:------:|:------:|
-| 2.2.0   | canary-1b-flash | 6.99 | 4.03 | 3.31 | 5.88 |
+| 2.3.0   | canary-1b-flash | 6.99 | 4.03 | 3.31 | 5.88 |
 
 
 More details on evaluation can be found at [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
 
 ### AST Performance
 
-We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html), and use native annotations with punctuation and capitalization in the datasets.
+We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html) and [COMET score](https://aclanthology.org/2020.emnlp-main.213/), and use native annotations with punctuation and capitalization in the datasets.
 
-BLEU score on [FLEURS](https://huggingface.co/datasets/google/fleurs) test set:
+[FLEURS](https://huggingface.co/datasets/google/fleurs) test set:
 
+BLEU score:
 | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
 |:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
-| 2.2.0       | canary-1b-flash | 	32.27   |  22.6   |   41.22    |   35.5   |   23.32    |   33.42    |
+| 2.3.0       | canary-1b-flash | 	32.27   |  22.6   |   41.22    |   35.5   |   23.32    |   33.42    |
 
+COMET score:
+| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
+|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
+| 2.3.0       | canary-1b-flash | 	0.8114   |  0.8118   |   0.8165    |   0.8546   |   0.8228   |   0.8475    |
+
+[COVOST-v2](https://github.com/facebookresearch/covost) test set:
 
-BLEU score on [COVOST-v2](https://github.com/facebookresearch/covost) test set:
+BLEU score:
+| **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
+|:-----------:|:---------:|:----------:|:----------:|:----------:|
+| 2.3.0       | canary-1b-flash |   39.33   |    41.86    |   41.43    |
 
+COMET score:
 | **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
 |:-----------:|:---------:|:----------:|:----------:|:----------:|
-| 2.2.0       | canary-1b-flash |   39.33   |    41.86    |   41.43    |
+| 2.3.0       | canary-1b-flash |   0.8553   |    0.8585    |   0.8511    |
 
-BLEU score on [mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso) test set:
+[mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso) test set:
 
+BLEU score:
 | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
 |:-----------:|:---------:|:----------:|:----------:|:----------:|
-| 2.2.0       | canary-1b-flash |   22.91    |   35.69   |   27.85   |
+| 2.3.0       | canary-1b-flash |   22.91    |   35.69   |   27.85   |
+
+COMET score:
+| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
+|:-----------:|:---------:|:----------:|:----------:|:----------:|
+| 2.3.0       | canary-1b-flash |   0.7889    |   0.8211   |   0.7910   |
 
 ### Timestamp Prediction
 F1-score on [Librispeech Test sets](https://www.openslr.org/12) at collar value of 200ms
 | **Version** | **Model** | **test-clean** | **test-other** |
 |:-----------:|:---------:|:----------:|:----------:|
-| 2.2.0       | canary-1b-flash |   95.5    |   93.5   |
+| 2.3.0       | canary-1b-flash |   95.5    |   93.5   |
 
 ### Hallucination Robustness
 Number of characters per minute on [MUSAN](https://www.openslr.org/17) 48 hrs eval set
 | **Version** | **Model** | **# of character per minute** |
 |:-----------:|:---------:|:----------:|
-| 2.2.0       | canary-1b-flash |   60.92   |
+| 2.3.0       | canary-1b-flash |   60.92   |
 
 ### Noise Robustness
 WER on [Librispeech Test Clean](https://www.openslr.org/12) at different SNR (signal to noise ratio) levels of additive white noise
 
 | **Version** | **Model** | **SNR 10** | **SNR 5** | **SNR 0** | **SNR -5** |
 |:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|
-| 2.2.0       | canary-1b-flash |    2.34   |   3.69   |   8.84   |    29.71  |
+| 2.3.0       | canary-1b-flash |    2.34   |   3.69   |   8.84   |    29.71  |
 
 ## Model Fairness Evaluation
 
-As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset" [8], we assessed the canary-1b-flash model for fairness. The model was evaluated on the CausalConversations-v1 dataset, and the results are reported as follows:
+As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset" [9], we assessed the canary-1b-flash model for fairness. The model was evaluated on the CausalConversations-v1 dataset, and the results are reported as follows:
 
 ### Gender Bias:
 
@@ -562,22 +579,24 @@ As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversat
 canary-1b-flash is released under the CC-BY-4.0 license. By using this model, you are agreeing to the [terms and conditions](https://choosealicense.com/licenses/cc-by-4.0/) of the license. <br>
 
 ## References:
-[1] [Less is More: Accurate Speech Recognition & Translation without Web-Scale Data](https://www.isca-archive.org/interspeech_2024/puvvada24_interspeech.pdf) <br>
-[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10389701)
+[1] [Training and Inference Efficiency of Encoder-Decoder Speech Models](https://arxiv.org/abs/2503.05931)
+
+[2] [Less is More: Accurate Speech Recognition & Translation without Web-Scale Data](https://www.isca-archive.org/interspeech_2024/puvvada24_interspeech.pdf)
+
+[3] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10389701)
 
-[3] [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
+[4] [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
 
-[4] [Unified Model for Code-Switching Speech Recognition and Language Identification Based on Concatenated Tokenizer](https://aclanthology.org/2023.calcs-1.7.pdf)
+[5] [Unified Model for Code-Switching Speech Recognition and Language Identification Based on Concatenated Tokenizer](https://aclanthology.org/2023.calcs-1.7.pdf)
 
-[5] [Google Sentencepiece Tokenizer](https://github.com/google/sentencepiece)
+[6] [Google Sentencepiece Tokenizer](https://github.com/google/sentencepiece)
 
-[6] [NVIDIA NeMo Toolkit](https://github.com/NVIDIA/NeMo)
+[7] [NVIDIA NeMo Toolkit](https://github.com/NVIDIA/NeMo)
 
-[7] [EMMeTT: Efficient Multimodal Machine Translation Training](https://arxiv.org/abs/2409.13523)
+[8] [EMMeTT: Efficient Multimodal Machine Translation Training](https://arxiv.org/abs/2409.13523)
 
-[8] [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9634168)
+[9] [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9634168)
 
-[9] [Training and Inference Efficiency of Encoder-Decoder Speech Models](https://arxiv.org/pdf/2503.05931)
 
 ## Ethical Considerations:
 NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.