Upload 10 files

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 256,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,94 @@
 ---
+language:
+- ru
+
+pipeline_tag: sentence-similarity
+
+tags:
+- russian
+- pretraining
+- embeddings
+- tiny
+- feature-extraction
+- sentence-similarity
+- sentence-transformers
+- transformers
+- mteb
+
+datasets:
+- IlyaGusev/gazeta
+- zloelias/lenta-ru
+- HuggingFaceFW/fineweb-2
+- HuggingFaceFW/fineweb
+
 license: mit
+
+
 ---
+
+
+Быстрая модель BERT для русского языка с размером ембеддинга 256 и длиной контекста 512. Модель получена методом последовательной дистилляции моделей [sergeyzh/rubert-tiny-turbo](https://huggingface.co/sergeyzh/rubert-tiny-turbo) и [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3). Выигрывает по скорости  у [rubert-tiny-turbo](https://huggingface.co/sergeyzh/rubert-tiny-turbo) при аналогичном качестве на CPU в ~x1.4, на GPU в ~x1.2 раза.
+
+
+
+## Использование
+```Python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer('sergeyzh/rubert-tiny-lite')
+
+sentences = ["привет мир", "hello world", "здравствуй вселенная"]
+embeddings = model.encode(sentences)
+
+print(model.similarity(embeddings, embeddings))
+```
+
+## Метрики
+
+Оценки модели на бенчмарке [encodechka](https://github.com/avidale/encodechka):
+
+| model                              | STS      | PI       | NLI      | SA       | TI       |
+|:-----------------------------------|:---------|:---------|:---------|:---------|:---------|
+| BAAI/bge-m3                        |  0.864   |  0.749   |  0.510   |  0.819   |  0.973   |
+| intfloat/multilingual-e5-large     |  0.862   |  0.727   |  0.473   |  0.810   |  0.979   |
+| **sergeyzh/rubert-tiny-lite**      |  0.839   |  0.712   |  0.488   |  0.788   |  0.949   |
+| intfloat/multilingual-e5-base      |  0.835   |  0.704   |  0.459   |  0.796   |  0.964   |
+| [sergeyzh/rubert-tiny-turbo](https://huggingface.co/sergeyzh/rubert-tiny-turbo)     |  0.828   |  0.722   |  0.476   |  0.787   |  0.955   |
+| intfloat/multilingual-e5-small     |  0.822   |  0.714   |  0.457   |  0.758   |  0.957   |
+| cointegrated/rubert-tiny2          |  0.750   |  0.651   |  0.417   |  0.737   |  0.937   |
+
+Оценки модели на бенчмарке [ruMTEB](https://habr.com/ru/companies/sberdevices/articles/831150/):
+
+|Model Name                         | Metric              | rubert-tiny2    | [rubert-tiny-turbo](https://huggingface.co/sergeyzh/rubert-tiny-turbo) | rubert-tiny-lite | multilingual-e5-small | multilingual-e5-base | multilingual-e5-large |
+|:----------------------------------|:--------------------|----------------:|------------------:|------------------:|----------------------:|---------------------:|----------------------:|
+|CEDRClassification                 | Accuracy            |      0.369      |        0.390      |        0.407      |        0.401          |        0.423         |       **0.448**       |
+|GeoreviewClassification            | Accuracy            |      0.396      |        0.414      |        0.423      |        0.447          |        0.461         |       **0.497**       |
+|GeoreviewClusteringP2P             | V-measure           |      0.442      |        0.597      |      **0.611**    |        0.586          |        0.545         |         0.605         |
+|HeadlineClassification             | Accuracy            |      0.742      |        0.686      |        0.652      |        0.732          |        0.757         |       **0.758**       |
+|InappropriatenessClassification    | Accuracy            |      0.586      |        0.591      |        0.588      |        0.592          |        0.588         |       **0.616**       |
+|KinopoiskClassification            | Accuracy            |      0.491      |        0.505      |        0.507      |        0.500          |        0.509         |       **0.566**       |
+|RiaNewsRetrieval                   | NDCG@10             |      0.140      |        0.513      |        0.617      |        0.700          |        0.702         |       **0.807**       |
+|RuBQReranking                      | MAP@10              |      0.461      |        0.622      |        0.631      |        0.715          |        0.720         |       **0.756**       |
+|RuBQRetrieval                      | NDCG@10             |      0.109      |        0.517      |        0.511      |        0.685          |        0.696         |       **0.741**       |
+|RuReviewsClassification            | Accuracy            |      0.570      |        0.607      |        0.615      |        0.612          |        0.630         |       **0.653**       |
+|RuSTSBenchmarkSTS                  | Pearson correlation |      0.694      |        0.787      |        0.799      |        0.781          |        0.796         |       **0.831**       |
+|RuSciBenchGRNTIClassification      | Accuracy            |      0.456      |        0.529      |        0.544      |        0.550          |        0.563         |       **0.582**       |
+|RuSciBenchGRNTIClusteringP2P       | V-measure           |      0.414      |        0.481      |        0.510      |        0.511          |        0.516         |       **0.520**       |
+|RuSciBenchOECDClassification       | Accuracy            |      0.355      |        0.415      |        0.424      |        0.427          |        0.423         |       **0.445**       |
+|RuSciBenchOECDClusteringP2P        | V-measure           |      0.381      |        0.411      |        0.438      |        0.443          |        0.448         |       **0.450**       |
+|SensitiveTopicsClassification      | Accuracy            |      0.220      |        0.244      |      **0.282**    |        0.228          |        0.234         |         0.257         |
+|TERRaClassification                | Average Precision   |      0.519      |        0.563      |        0.574      |        0.551          |        0.550         |       **0.584**       |
+                                                                                                  
+|Model Name                         | Metric              | rubert-tiny2    | [rubert-tiny-turbo](https://huggingface.co/sergeyzh/rubert-tiny-turbo) | rubert-tiny-lite  | multilingual-e5-small | multilingual-e5-base | multilingual-e5-large |
+|:----------------------------------|:--------------------|----------------:|------------------:|------------------:|----------------------:|----------------------:|---------------------:|
+|Classification                     | Accuracy            |      0.514      |        0.535      |        0.536      |        0.551          |        0.561          |      **0.588**       |
+|Clustering                         | V-measure           |      0.412      |        0.496      |        0.520      |        0.513          |        0.503          |      **0.525**       |
+|MultiLabelClassification           | Accuracy            |      0.294      |        0.317      |        0.344      |        0.314          |        0.329          |      **0.353**       |
+|PairClassification                 | Average Precision   |      0.519      |        0.563      |        0.574      |        0.551          |        0.550          |      **0.584**       |
+|Reranking                          | MAP@10              |      0.461      |        0.622      |        0.631      |        0.715          |        0.720          |      **0.756**       |
+|Retrieval                          | NDCG@10             |      0.124      |        0.515      |        0.564      |        0.697          |        0.699          |      **0.774**       |
+|STS                                | Pearson correlation |      0.694      |        0.787      |        0.799      |        0.781          |        0.796          |      **0.831**       |
+|Average                            | Average             |      0.431      |        0.548      |        0.567      |        0.588          |        0.594          |      **0.630**       |
+
+
+

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": ".sergeyzh/rubert-tiny-lite",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "emb_size": 256,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 3,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 83828
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1deef831e2634e8f074e2520eebc18d4308e72ca74adccca1ffc23beb2554cd3
+size 92174712

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,64 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "max_length": 512,
+  "model_max_length": 2048,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}