ankitapasad commited on
Commit
5136985
·
verified ·
1 Parent(s): f54e00b

added COMET scores for AST

Browse files
Files changed (1) hide show
  1. README.md +26 -3
README.md CHANGED
@@ -502,25 +502,48 @@ More details on evaluation can be found at [HuggingFace ASR Leaderboard](https:/
502
 
503
  We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html), and use native annotations with punctuation and capitalization in the datasets.
504
 
505
- BLEU score on [FLEURS](https://huggingface.co/datasets/google/fleurs) test set:
 
 
506
 
507
  | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
508
  |:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
509
  | 2.2.0 | canary-180m-flash | 28.18 | 20.47 | 36.66 | 32.08 | 20.09 | 29.75 |
510
 
 
 
 
 
 
511
 
512
- BLEU score on [COVOST-v2](https://github.com/facebookresearch/covost) test set:
 
 
513
 
514
  | **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
515
  |:-----------:|:---------:|:----------:|:----------:|:----------:|
516
  | 2.2.0 | canary-180m-flash | 35.61 | 39.84 | 38.57 |
517
 
518
- BLEU score on [mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso) test set:
 
 
 
 
 
 
 
 
519
 
520
  | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
521
  |:-----------:|:---------:|:----------:|:----------:|:----------:|
522
  | 2.2.0 | canary-180m-flash | 21.60 | 33.45 | 25.96 |
523
 
 
 
 
 
 
 
524
 
525
  ### Timestamp Prediction
526
  F1-score on [Librispeech Test sets](https://www.openslr.org/12) at collar value of 200ms
 
502
 
503
  We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html), and use native annotations with punctuation and capitalization in the datasets.
504
 
505
+ [FLEURS](https://huggingface.co/datasets/google/fleurs) test set:
506
+
507
+ BLEU score:
508
 
509
  | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
510
  |:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
511
  | 2.2.0 | canary-180m-flash | 28.18 | 20.47 | 36.66 | 32.08 | 20.09 | 29.75 |
512
 
513
+ COMET score:
514
+
515
+ | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
516
+ |:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
517
+ | 2.2.0 | canary-180m-flash | 77.56 | 78.10 | 78.53 | 83.03 | 81.48 | 82.28 |
518
 
519
+ [COVOST-v2](https://github.com/facebookresearch/covost) test set:
520
+
521
+ BLEU score:
522
 
523
  | **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
524
  |:-----------:|:---------:|:----------:|:----------:|:----------:|
525
  | 2.2.0 | canary-180m-flash | 35.61 | 39.84 | 38.57 |
526
 
527
+ COMET score:
528
+
529
+ | **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
530
+ |:-----------:|:---------:|:----------:|:----------:|:----------:|
531
+ | 2.2.0 | canary-180m-flash | 80.94 | 84.54 | 82.50 |
532
+
533
+ [mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso) test set:
534
+
535
+ BLEU score:
536
 
537
  | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
538
  |:-----------:|:---------:|:----------:|:----------:|:----------:|
539
  | 2.2.0 | canary-180m-flash | 21.60 | 33.45 | 25.96 |
540
 
541
+ COMET score:
542
+
543
+ | **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
544
+ |:-----------:|:---------:|:----------:|:----------:|:----------:|
545
+ | 2.2.0 | canary-180m-flash | 77.71 | 80.87 | 77.82 |
546
+
547
 
548
  ### Timestamp Prediction
549
  F1-score on [Librispeech Test sets](https://www.openslr.org/12) at collar value of 200ms