added COMET scores for AST
Browse files
README.md
CHANGED
@@ -502,25 +502,48 @@ More details on evaluation can be found at [HuggingFace ASR Leaderboard](https:/
|
|
502 |
|
503 |
We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html), and use native annotations with punctuation and capitalization in the datasets.
|
504 |
|
505 |
-
|
|
|
|
|
506 |
|
507 |
| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
|
508 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
|
509 |
| 2.2.0 | canary-180m-flash | 28.18 | 20.47 | 36.66 | 32.08 | 20.09 | 29.75 |
|
510 |
|
|
|
|
|
|
|
|
|
|
|
511 |
|
512 |
-
|
|
|
|
|
513 |
|
514 |
| **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
|
515 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
516 |
| 2.2.0 | canary-180m-flash | 35.61 | 39.84 | 38.57 |
|
517 |
|
518 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
519 |
|
520 |
| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
|
521 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
522 |
| 2.2.0 | canary-180m-flash | 21.60 | 33.45 | 25.96 |
|
523 |
|
|
|
|
|
|
|
|
|
|
|
|
|
524 |
|
525 |
### Timestamp Prediction
|
526 |
F1-score on [Librispeech Test sets](https://www.openslr.org/12) at collar value of 200ms
|
|
|
502 |
|
503 |
We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html), and use native annotations with punctuation and capitalization in the datasets.
|
504 |
|
505 |
+
[FLEURS](https://huggingface.co/datasets/google/fleurs) test set:
|
506 |
+
|
507 |
+
BLEU score:
|
508 |
|
509 |
| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
|
510 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
|
511 |
| 2.2.0 | canary-180m-flash | 28.18 | 20.47 | 36.66 | 32.08 | 20.09 | 29.75 |
|
512 |
|
513 |
+
COMET score:
|
514 |
+
|
515 |
+
| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** | **De->En** | **Es->En** | **Fr->En** |
|
516 |
+
|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
|
517 |
+
| 2.2.0 | canary-180m-flash | 77.56 | 78.10 | 78.53 | 83.03 | 81.48 | 82.28 |
|
518 |
|
519 |
+
[COVOST-v2](https://github.com/facebookresearch/covost) test set:
|
520 |
+
|
521 |
+
BLEU score:
|
522 |
|
523 |
| **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
|
524 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
525 |
| 2.2.0 | canary-180m-flash | 35.61 | 39.84 | 38.57 |
|
526 |
|
527 |
+
COMET score:
|
528 |
+
|
529 |
+
| **Version** | **Model** | **De->En** | **Es->En** | **Fr->En** |
|
530 |
+
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
531 |
+
| 2.2.0 | canary-180m-flash | 80.94 | 84.54 | 82.50 |
|
532 |
+
|
533 |
+
[mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso) test set:
|
534 |
+
|
535 |
+
BLEU score:
|
536 |
|
537 |
| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
|
538 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
539 |
| 2.2.0 | canary-180m-flash | 21.60 | 33.45 | 25.96 |
|
540 |
|
541 |
+
COMET score:
|
542 |
+
|
543 |
+
| **Version** | **Model** | **En->De** | **En->Es** | **En->Fr** |
|
544 |
+
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
545 |
+
| 2.2.0 | canary-180m-flash | 77.71 | 80.87 | 77.82 |
|
546 |
+
|
547 |
|
548 |
### Timestamp Prediction
|
549 |
F1-score on [Librispeech Test sets](https://www.openslr.org/12) at collar value of 200ms
|