add model

README.md

@@ -0,0 +1,113 @@
+---
+tags:
+- espnet
+- audio
+- automatic-speech-recognition
+- speech-translation
+- language-identification
+language: multilingual
+datasets:
+- owsm_v4
+license: cc-by-4.0
+---
+
+
+## Open Whisper-style Speech Model (OWSM)
+
+OWSM aims to develop fully open speech foundation models using publicly available data and open-source toolkits including [ESPnet](https://github.com/espnet/espnet).
+
+Inference examples can be found in our [project page](https://www.wavlab.org/activities/2024/owsm/).
+The Gradio demo is [here](https://huggingface.co/spaces/pyf98/OWSM_v3_demo). 
+
+[OWSM v4]() is the latest version in the OWSM series, which significantly outperforms OWSM v3.1 in LID and multilingual ASR.
+
+This repo contains a small-sized model with 370M parameters. It is trained on 320k hours of public speech data. It supports the following speech-to-text tasks:
+- Language identification
+- Speech recognition
+- Speech translation
+- Utterance-level timestamp prediction
+- Long-form transcription
+
+
+### OWSM series
+
+#### Encoder-decoder OWSM
+
+| Name | Size | Hugging Face Repo |
+| :--- | ---: | :---------------- |
+| OWSM v3.1 base | 101M | https://huggingface.co/espnet/owsm_v3.1_ebf_base |
+| OWSM v3.1 small | 367M | https://huggingface.co/espnet/owsm_v3.1_ebf_small |
+| OWSM v3.1 medium | 1.02B | https://huggingface.co/espnet/owsm_v3.1_ebf |
+| OWSM v3.2 small | 367M | https://huggingface.co/espnet/owsm_v3.2 |
+| OWSM v4 base | 102M | https://huggingface.co/espnet/owsm_v4_base_102M |
+| OWSM v4 small | 370M | https://huggingface.co/espnet/owsm_v4_small_370M |
+| OWSM v4 medium | 1.02B | https://huggingface.co/espnet/owsm_v4_medium_1B |
+
+
+#### CTC-based OWSM
+
+| Name | Size | Hugging Face Repo |
+| :--- | ---: | :---------------- |
+| OWSM-CTC v3.1 medium | 1.01B | https://huggingface.co/espnet/owsm_ctc_v3.1_1B |
+| OWSM-CTC v3.2 medium | 1.01B | https://huggingface.co/espnet/owsm_ctc_v3.2_ft_1B |
+| OWSM-CTC v4 medium | 1.01B | https://huggingface.co/espnet/owsm_ctc_v4_1B |
+
+
+
+### Citations
+
+#### OWSM v4
+
+```BibTex
+Coming soon
+```
+
+#### OWSM-CTC
+
+```BibTex
+@inproceedings{owsm-ctc,
+    title = "{OWSM}-{CTC}: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification",
+    author = "Peng, Yifan  and
+      Sudo, Yui  and
+      Shakeel, Muhammad  and
+      Watanabe, Shinji",
+    booktitle = "Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)",
+    year = "2024",
+    month= {8},
+    url = "https://aclanthology.org/2024.acl-long.549",
+}
+```
+
+#### OWSM v3.1 and v3.2
+
+```BibTex
+@inproceedings{owsm-v32,
+  title={On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models},
+  author={Jinchuan Tian and Yifan Peng and William Chen and Kwanghee Choi and Karen Livescu and Shinji Watanabe},
+  booktitle={Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH)},
+  year={2024},
+  month={9},
+  pdf="https://arxiv.org/pdf/2406.09282"
+}
+@inproceedings{owsm-v31,
+  title={{OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer}},
+  author={Yifan Peng and Jinchuan Tian and William Chen and Siddhant Arora and Brian Yan and Yui Sudo and Muhammad Shakeel and Kwanghee Choi and Jiatong Shi and Xuankai Chang and Jee-weon Jung and Shinji Watanabe},
+  booktitle={Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH)},
+  year={2024},
+  month={9},
+  pdf="https://arxiv.org/pdf/2401.16658",
+}
+```
+
+#### Initial OWSM (v1, v2, v3)
+
+```BibTex
+@inproceedings{owsm,
+  title={Reproducing Whisper-Style Training Using An Open-Source Toolkit And Publicly Available Data},
+  author={Yifan Peng and Jinchuan Tian and Brian Yan and Dan Berrebbi and Xuankai Chang and Xinjian Li and Jiatong Shi and Siddhant Arora and William Chen and Roshan Sharma and Wangyou Zhang and Yui Sudo and Muhammad Shakeel and Jee-weon Jung and Soumi Maiti and Shinji Watanabe},
+  booktitle={Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
+  year={2023},
+  month={12},
+  pdf="https://arxiv.org/pdf/2309.13876",
+}
+```

data/token_list/bpe_unigram50000/bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ddb01f03dab493c18ab69391e98744c090f897890d8b529b30cae52a8d9eef4
+size 1044580

exp/s2t_stats_raw_bpe50000/train/feats_stats.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00c22dba27594df8f1d8f74a491b20c6e6e8c17e92159f81dfd634f98c098654
+size 1786

exp/s2t_train_conv2d8_size768_e9_d9_mel128_raw_bpe50000/valid.total_count.ave_5best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae83197c66c2d8f5fbd45ec7fa845fbee3e62a708c5f3612f2b5e42dc719c564
+size 1478780074

meta.yaml

@@ -0,0 +1,8 @@
+espnet: '202412'
+files:
+  s2t_model_file: exp/s2t_train_conv2d8_size768_e9_d9_mel128_raw_bpe50000/valid.total_count.ave_5best.pth
+python: 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:26:55) [GCC 12.3.0]
+timestamp: 1738817987.260615
+torch: 2.5.1
+yaml_files:
+  s2t_train_config: exp/s2t_train_conv2d8_size768_e9_d9_mel128_raw_bpe50000/config.yaml