Add model card, acoustic model checkpoint in safetensors format, ONNX model, tokenizer configs and KenLM model

README.md

@@ -1,3 +1,160 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- ru
+pipeline_tag: automatic-speech-recognition
+tags:
+- conformer
+- streaming
+- asr
+- stt
+- telephony
+- russian
+- speech
+- t-tech
+- t-one
+---
+
+# T-one: Streaming ASR for Russian Telephony
+
+**🚀 T-one is a high-performance streaming ASR pipeline for Russian, specialized for the telephony domain.**
+
+T-one provides a complete low-latency solution for real-time transcription. It features a pretrained streaming Conformer-based acoustic model, a custom phrase boundary detector and a decoder, making it a ready-to-use solution for production environments. It provides not only the pretrained model but also a full suite of tools for inference, fine-tuning, and deployment.
+
+Developed by *T-Software DC*, this project is a practical low-latency, high-throughput ASR solution with modular components.
+
+For more details, see the [**GitHub Repository**](https://github.com/voicekit-team/T-one).
+
+## Table of Contents
+1. [Project Summary](#-project-summary)
+2. [Quality benchmarks](#-quality-benchmarks)
+3. [Inference examples](#-inference-examples)
+4. [Fine-tuning](#-fine-tuning)
+5. [Acoustic model](#-acoustic-model)
+6. [Training details](#-training-details)
+7. [License](#-license)
+
+## 📝 Project Summary
+
+**Key Features:**
+- **Streaming-first Architecture:** Built for low-latency, real-time applications.
+- **Ready-to-Use Pipeline:** Includes a pretrained acoustic model, phrase splitter, and a KenLM-based [**CTC**](https://huggingface.co/learn/audio-course/chapter3/ctc) beam search decoder with examples for offline and streaming speech recognition inference.
+- **Demo** — launch a local speech recognition service instantly via Docker and transcribe audio files or real-time microphone input.
+- **Fine-tuning** T-one on a custom dataset is straightforward using the 🤗 ecosystem.
+- **Easy Deployment:** Includes examples for deploying with Triton Inference Server for high-throughput scenarios.
+- **Fully Open Source architecture:** All model and pipeline code is available.
+
+## 📊 Quality benchmarks
+
+**Word Error Rate ([WER](https://huggingface.co/spaces/evaluate-metric/wer))** is used to evaluate the quality of automatic speech recognition systems, which can be interpreted as the percentage of incorrectly recognized words compared to a reference transcript. A lower value indicates higher accuracy. T-one demonstrates state-of-the-art performance, especially on its target domain of telephony, while remaining competitive on general-purpose benchmarks.
+
+| Category | T-one (70M) | GigaAM-RNNT v2 (243M) | GigaAM-CTC v2 (242M) | Vosk-model-ru 0.54 (65M) | Vosk-model-small-streaming-ru 0.54 (20M) | Whisper large-v3 (1540M) |
+|:--|:--|:--|:--|--:|:--|:--|
+| Call-center | **8.63** | 10.22 | 10.57 | 11.28 | 15.53 | 19.39 |
+| Other telephony | **6.20** | 7.88 | 8.15 | 8.69 | 13.49 | 17.29 |
+| Named entities | **5.83** | 9.55 | 9.81 | 12.12 | 17.65 | 17.87 |
+| CommonVoice 19 (test split) | 5.32 | **2.68** | 3.14 | 6.22 | 11.3 | 5.78 |
+| OpenSTT asr_calls_2_val original | 20.27 | **20.07** | 21.24 | 22.64 | 29.45 | 29.02 |
+| OpenSTT asr_calls_2_val re-labeled | **7.94** | 11.14 | 12.43 | 13.22 | 21.03 | 20.82 |
+
+## 👨‍💻 Inference examples
+
+### Offline Inference (for entire audio files)
+```python
+from tone import StreamingCTCPipeline, read_audio, read_example_audio
+
+
+audio = read_example_audio() # or read_audio("your_audio.flac")
+
+pipeline = StreamingCTCPipeline.from_hugging_face()
+print(pipeline.forward_offline(audio))  # run offline recognition
+```
+Output:
+```
+[TextPhrase(text='привет', start_time=1.79, end_time=2.04), TextPhrase(text='это я', start_time=3.72, end_time=4.26), TextPhrase(text='я подумала не хочешь ли ты встретиться спустя все эти годы', start_time=5.88, end_time=10.59)]
+```
+
+### Streaming Inference (for real-time audio)
+
+```python
+from tone import StreamingCTCPipeline, read_stream_example_audio
+
+
+pipeline = StreamingCTCPipeline.from_hugging_face()
+
+state = None  # Current state of the ASR pipeline (None - initial)
+for audio_chunk in read_stream_example_audio():  # Use any source of audio chunks
+    new_phrases, state = pipeline.forward(audio_chunk, state)
+    print(new_phrases)
+
+# Finalize the pipeline and get the remaining phrases
+new_phrases, _ = pipeline.finalize(state)
+print(new_phrases)
+```
+Output:
+```
+TextPhrase(text='привет', start_time=1.79, end_time=2.04)
+TextPhrase(text='это я', start_time=3.72, end_time=4.26)
+TextPhrase(text='я подумала не хочешь ли ты встретиться спустя все эти годы', start_time=5.88, end_time=10.59)
+```
+
+## 🔧 Fine-tuning
+In order to fine-tune T-one from a pre-trained checkpoint you need to prepare the training dataset, load the tokenizer and feature extractor from `t-tech/T-one` 🤗 repo.
+
+```python
+import torch
+
+from tone.training.model_wrapper import ToneForCTC
+
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model = ToneForCTC.from_pretrained("t-tech/T-one").to(device)
+```
+
+Setup the data collator, evaluation metric, training arguments and 🤗 Trainer.
+
+For a complete guide please refer to the [**fine-tuning example notebook**](https://github.com/voicekit-team/T-one/blob/main/examples/finetune_example.ipynb).
+
+## 🎙 Acoustic model
+
+### Architecture
+T-one is a 70M parameter acoustic model based on the **Conformer** architecture, with several key innovations to improve performance and efficiency:
+- **SwiGLU Activation:** The feed-forward module is replaced with a SwiGLU module for better performance.
+- **Modern Normalization:** SiLU (Swish) activations and RMSNorm are used in place of ReLU and LayerNorm.
+- **RoPE Embeddings:** Relative positional embeddings from Transformer-XL are replaced with faster Rotary Position Embeddings (RoPE).
+- **U-Net Structure:** The temporal dimension is downsampled and then upsampled within the Conformer blocks, improving the model's receptive field.
+- **Attention Score Reuse:** Multi-Head Self-Attention layers are grouped, and attention scores are computed only once per group to reduce computation.
+- **Efficient State Management:** Streaming states are used only in the final two layers of the model.
+
+It processes audio in 300 ms chunks and generates transcriptions using either greedy decoding or a KenLM-based CTC beam search decoder.
+
+The model was trained using CTC-Loss.
+T-one is primarily intended for use with telephone-channel audio. However, since it was trained on heterogeneous data, it is robust across different domains and can be used not only for telephony.
+The model supports streaming inference, which means it can process long audio files out-of-the-box in a real-time manner.
+The primary use case for this model is streaming speech recognition of calls. The user sends small audio chunks to the model, and it processes each segment incrementally, returning the finalized text and word-level timestamps in real time.
+T-one can be easily fine-tuned for specific domains.
+
+For a detailed exploration of our architecture, design choices, and implementation, check out our accompanying article (link will be shared shortly). Also refer to our **technical deep dive** on how to improve quality and training speed of a streaming ASR model on [**YouTube**](https://www.youtube.com/watch?v=OQD9o1MdFRE).
+
+## 📉 Training details
+
+### Training Data
+The acoustic model was trained on over 80,000 hours of Russian speech. A significant portion (up to 64%) was pseudo-labeled using a robust ROVER model ensemble.
+
+| Domain      | Hours   | Source      |
+|:------------|:--------|:------------|
+| Telephony   | 57.9k   | internal    |
+| Far-field   | 2.2K    | internal    |
+| Mix         | 18.4K   | internal    |
+| Mix         | 2.3K    | open-source |
+
+### Training Procedure
+The model was trained from scratch (random initialization) for 7 days on 8 A100 GPUs using the **NVIDIA NeMo** framework. Key training parameters include:
+- **Optimizer:** AdamW
+- **Scheduler:** Cosine annealing with warmup
+- **Precision:** 16-bit mixed precision
+- **Batching:** Semi-sorted batching for efficiency
+
+## 📜 License
+
+This project, including the code and pretrained models, is released under the **Apache 2.0 License**.

added_tokens.json

@@ -0,0 +1,5 @@
+{
+  "</s>": 36,
+  "<s>": 35,
+  "<unk>": 37
+}

config.json

@@ -0,0 +1,117 @@
+{
+  "architectures": [
+    "ToneForCTC"
+  ],
+  "ctc_loss_reduction": "mean",
+  "ctc_zero_infinity": true,
+  "decoder_params": {
+    "feat_in": 384,
+    "vocabulary": [
+      "а",
+      "б",
+      "в",
+      "г",
+      "д",
+      "е",
+      "ё",
+      "ж",
+      "з",
+      "и",
+      "й",
+      "к",
+      "л",
+      "м",
+      "н",
+      "о",
+      "п",
+      "р",
+      "с",
+      "т",
+      "у",
+      "ф",
+      "х",
+      "ц",
+      "ч",
+      "ш",
+      "щ",
+      "ъ",
+      "ы",
+      "ь",
+      "э",
+      "ю",
+      "я",
+      " "
+    ]
+  },
+  "encoder_params": {
+    "chunk_size": 10,
+    "conv_kernel_size": 31,
+    "d_model": 384,
+    "dropout": 0.1,
+    "dropout_att": 0.1,
+    "feat_in": 64,
+    "ff_expansion_factor": 4,
+    "mhsa_state_size": 30,
+    "mhsa_stateless_layers": 14,
+    "n_heads": 8,
+    "n_layers": 16,
+    "reduction_factor": 2,
+    "reduction_kernel_size": 3,
+    "reduction_position": 6,
+    "rope_dim": 32,
+    "should_recompute_att_scores": [
+      true,
+      false,
+      false,
+      false,
+      false,
+      false,
+      false,
+      true,
+      false,
+      false,
+      false,
+      false,
+      false,
+      false,
+      true,
+      true
+    ],
+    "subsampling_conv_channels": [
+      32,
+      64
+    ],
+    "subsampling_kernel_size": [
+      [
+        11,
+        21
+      ],
+      [
+        11,
+        11
+      ]
+    ],
+    "subsampling_strides": [
+      [
+        1,
+        1
+      ],
+      [
+        3,
+        1
+      ]
+    ],
+    "upsample_position": 14
+  },
+  "feature_extraction_params": {
+    "n_fft": 160,
+    "n_mels": 64,
+    "preemphasis_coefficient": 0.97,
+    "sample_rate": 8000,
+    "window_size": 0.02,
+    "window_stride": 0.01
+  },
+  "pad_token_id": 34,
+  "torch_dtype": "float32",
+  "transformers_version": "4.41.2"
+}

kenlm.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c31a489a51a6e9236112dacb6bed12f45e8df734057615fa6bf220a5a769a1d
+size 5463477004

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc9b82db8419044430557e117a2669056b41c80cde907f01c78dab84333acb2f
+size 144199888

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60f42e72314cd132a25a770913a6c55ebd44fb1a665d0e2371267551eab08007
+size 286858388

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "[PAD]",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,47 @@
+{
+  "added_tokens_decoder": {
+    "34": {
+      "content": "[PAD]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "35": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "36": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "37": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "do_lower_case": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "replace_word_delimiter_char": " ",
+  "target_lang": null,
+  "tokenizer_class": "Wav2Vec2CTCTokenizer",
+  "unk_token": "<unk>",
+  "word_delimiter_token": "|"
+}

vocab.json

@@ -0,0 +1,37 @@
+{
+  "[PAD]": 34,
+  "|": 33,
+  "а": 0,
+  "б": 1,
+  "в": 2,
+  "г": 3,
+  "д": 4,
+  "е": 5,
+  "ж": 7,
+  "з": 8,
+  "и": 9,
+  "й": 10,
+  "к": 11,
+  "л": 12,
+  "м": 13,
+  "н": 14,
+  "о": 15,
+  "п": 16,
+  "р": 17,
+  "с": 18,
+  "т": 19,
+  "у": 20,
+  "ф": 21,
+  "х": 22,
+  "ц": 23,
+  "ч": 24,
+  "ш": 25,
+  "щ": 26,
+  "ъ": 27,
+  "ы": 28,
+  "ь": 29,
+  "э": 30,
+  "ю": 31,
+  "я": 32,
+  "ё": 6
+}