File size: 28,217 Bytes
627ef7f c2f9933 627ef7f a736a2a 627ef7f 962fd50 627ef7f 962fd50 c2f9933 962fd50 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f 962fd50 627ef7f c2f9933 627ef7f 962fd50 3677c5b 962fd50 3677c5b 962fd50 3677c5b 962fd50 3677c5b 962fd50 3677c5b 962fd50 38a4be7 962fd50 627ef7f 962fd50 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f de3ab2c 962fd50 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f a736a2a c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f 4fdf2a3 627ef7f c2f9933 627ef7f c2f9933 627ef7f a736a2a 627ef7f c2f9933 627ef7f c2f9933 627ef7f c2f9933 627ef7f 4fdf2a3 627ef7f c2f9933 627ef7f a736a2a 627ef7f c2f9933 627ef7f 4fdf2a3 627ef7f a736a2a 627ef7f c2f9933 627ef7f a736a2a 627ef7f a736a2a 627ef7f a736a2a 627ef7f a736a2a 627ef7f 962fd50 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 |
---
tags:
- sentence-transformers
- sparse-encoder
- sparse
- splade
- generated_from_trainer
- dataset_size:1200000
- loss:SpladeLoss
- loss:SparseMarginMSELoss
- loss:FlopsLoss
base_model:
- prajjwal1/bert-tiny
widget:
- text: >-
Most Referenced:report - Return to the USDOJ/OIG Home Page - US Department
of JusticeReturn to the USDOJ/OIG Home Page - US Department of Justice.
Opinion:Roberts: Feds to stop using private prisons.
- text: >-
Paul O'Neill, the founder of the Trans-Siberian Orchestra (pictured) has
died at age 61. Paul O'Neill, the founder of the popular Christmas-themed
rock ensemble Trans-Siberian Orchestra has died. A statement on the group's
Facebook page reads: The entire Trans-Siberian Orchestra family, past and
present, is heartbroken to share the devastating news that Paul O’Neill has
passed away from chronic illness.
- text: meaning for concern
- text: >-
Additional Tips. 1 Do not rub the ink stains as it can spread the stains
further. 2 Make sure you test the cleaning solution on a small, hidden area
to check if it is suitable for the material. 3 In case an ink stain has
become old and dried, the above mentioned home remedies may not be
effective.arpet: For ink stained spots on a carpet, you may apply a paste of
cornstarch and milk. Leave it for a few hours before brushing it off.
Finally, clean the residue with a vacuum cleaner. Leather: Try using a
leather shampoo or a leather ink remover for removing ink stains from
leather items.
- text: >-
See below: 1. Get your marriage license. Before you can change your name,
you'll need the original (or certified) marriage license with the raised
seal and your new last name on it. Call the clerk's office where your
license was filed to get copies if one wasn't automatically sent to you. 2.
Change your Social Security card.
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
- dot_accuracy@1
- dot_accuracy@3
- dot_accuracy@5
- dot_accuracy@10
- dot_precision@1
- dot_precision@3
- dot_precision@5
- dot_precision@10
- dot_recall@1
- dot_recall@3
- dot_recall@5
- dot_recall@10
- dot_ndcg@10
- dot_mrr@10
- dot_map@100
- query_active_dims
- query_sparsity_ratio
- corpus_active_dims
- corpus_sparsity_ratio
model-index:
- name: SPLADE Sparse Encoder
results:
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: Unknown
type: unknown
metrics:
- type: dot_accuracy@1
value: 0.4772
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.793
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.8964
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.96
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.4772
name: Dot Precision@1
- type: dot_precision@3
value: 0.2713333333333333
name: Dot Precision@3
- type: dot_precision@5
value: 0.18644000000000002
name: Dot Precision@5
- type: dot_precision@10
value: 0.10094000000000002
name: Dot Precision@10
- type: dot_recall@1
value: 0.4616666666666666
name: Dot Recall@1
- type: dot_recall@3
value: 0.7798833333333334
name: Dot Recall@3
- type: dot_recall@5
value: 0.8874
name: Dot Recall@5
- type: dot_recall@10
value: 0.95595
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.721747648718731
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.6489996031746051
name: Dot Mrr@10
- type: dot_map@100
value: 0.6446961471449598
name: Dot Map@100
- type: query_active_dims
value: 18.334199905395508
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.9993993119747921
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 121.65303042911474
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9960142510179831
name: Corpus Sparsity Ratio
datasets:
- microsoft/ms_marco
language:
- en
---
# SPLADE Sparse Encoder
This is a SPLADE sparse retrieval model based on BERT-Tiny (4M) that was trained by distilling a Cross-Encoder on the MSMARCO dataset. The cross-encoder used was [ms-marco-MiniLM-L6-v2](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2).
This Tiny SPLADE model beats `BM25` by `65.6%` on the MSMARCO benchmark. While this model is `15x` smaller than Naver's official `splade-v3-distilbert`, is posesses `80%` of it's performance on MSMARCO. This model is small enough to be used without a GPU on a dataset of a few thousand documents.
- `Collection:` https://huggingface.co/collections/rasyosef/splade-tiny-msmarco-687c548c0691d95babf65b70
- `Distillation Dataset:` https://huggingface.co/datasets/yosefw/msmarco-train-distil-v2
- `Code:` https://github.com/rasyosef/splade-tiny-msmarco
## Performance
The splade models were evaluated on 55 thousand queries and 8.84 million documents from the [MSMARCO](https://huggingface.co/datasets/microsoft/ms_marco) dataset.
||Size (# Params)|MRR@10 (MS MARCO dev)|
|:---|:----|:-------------------|
|`BM25`|-|18.0|-|-|
|`rasyosef/splade-tiny`|4.4M|30.9|
|`rasyosef/splade-mini`|11.2M|33.2|
|`naver/splade-v3-distilbert`|67.0M|38.7|
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SparseEncoder
# Download from the 🤗 Hub
model = SparseEncoder("rasyosef/splade-tiny")
# Run inference
queries = [
"what do i need to change my name on my license in ma",
]
documents = [
'Change your name on MA state-issued ID such as driver’s license or MA ID card. All documents you bring to RMV need to be originals or certified copies by the issuing agency. PAPERWORK NEEDED: Proof of legal name change — A court order showing your legal name change. Your Social Security Card with your new legal name change',
"See below: 1. Get your marriage license. Before you can change your name, you'll need the original (or certified) marriage license with the raised seal and your new last name on it. Call the clerk's office where your license was filed to get copies if one wasn't automatically sent to you. 2. Change your Social Security card.",
"You'll keep the same number—just your name will be different. Mail in your application to the local Social Security Administration office. You should get your new card within 10 business days. 3. Change your license at the DMV. Take a trip to the local Department of Motor Vehicles office to get a new license with your new last name. Bring every form of identification you can get your hands on—your old license, your certified marriage certificate and, most importantly, your new Social Security card.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]
# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[16.6297, 13.4552, 10.1923]])
```
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
## Model Details
### Model Description
- **Model Type:** SPLADE Sparse Encoder
- **Base model:** [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny)
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 30522 dimensions
- **Similarity Function:** Dot Product
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder)
### Full Model Architecture
```
SparseEncoder(
(0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
(1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)
```
## More
<details><summary>Click to expand</summary>
## Evaluation
### Metrics
#### Sparse Information Retrieval
* Evaluated with [<code>SparseInformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator)
| Metric | Value |
|:----------------------|:-----------|
| dot_accuracy@1 | 0.4772 |
| dot_accuracy@3 | 0.793 |
| dot_accuracy@5 | 0.8964 |
| dot_accuracy@10 | 0.96 |
| dot_precision@1 | 0.4772 |
| dot_precision@3 | 0.2713 |
| dot_precision@5 | 0.1864 |
| dot_precision@10 | 0.1009 |
| dot_recall@1 | 0.4617 |
| dot_recall@3 | 0.7799 |
| dot_recall@5 | 0.8874 |
| dot_recall@10 | 0.9559 |
| **dot_ndcg@10** | **0.7217** |
| dot_mrr@10 | 0.649 |
| dot_map@100 | 0.6447 |
| query_active_dims | 18.3342 |
| query_sparsity_ratio | 0.9994 |
| corpus_active_dims | 121.653 |
| corpus_sparsity_ratio | 0.996 |
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Dataset
#### Unnamed Dataset
* Size: 1,200,000 training samples
* Columns: <code>query</code>, <code>positive</code>, <code>negative_1</code>, <code>negative_2</code>, and <code>label</code>
* Approximate statistics based on the first 1000 samples:
| | query | positive | negative_1 | negative_2 | label |
|:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-----------------------------------|
| type | string | string | string | string | list |
| details | <ul><li>min: 4 tokens</li><li>mean: 9.08 tokens</li><li>max: 35 tokens</li></ul> | <ul><li>min: 23 tokens</li><li>mean: 79.02 tokens</li><li>max: 192 tokens</li></ul> | <ul><li>min: 18 tokens</li><li>mean: 78.24 tokens</li><li>max: 230 tokens</li></ul> | <ul><li>min: 13 tokens</li><li>mean: 75.26 tokens</li><li>max: 230 tokens</li></ul> | <ul><li>size: 2 elements</li></ul> |
* Samples:
| query | positive | negative_1 | negative_2 | label |
|:----------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------|
| <code>does alzheimer's affect sleep</code> | <code>People with Alzheimer’s disease go through many changes, and sleep problems are often some of the most noticeable. Most adults have changes in their sleep patterns as they age. But the problems are more severe and happen more often for people with Alzheimer’s.</code> | <code>Could the position you SLEEP in affect your risk of Alzheimer's? People who sleep on their side enable their brain to 'detox' better while they rest. While asleep, brain is hard at work removing toxins that build up in the day. If left to build up, these toxins can cause Alzheimer's and Parkinson's.</code> | <code>The Scary Connection Between Snoring and Dementia. For more, visit TIME Health. If you don't snore, you likely know someone who does. Between 19% and 40% of adults snore when they sleep, and that percentage climbs even higher, particularly for men, as we age.</code> | <code>[1.407266616821289, 10.169305801391602]</code> |
| <code>what is fy in steel design</code> | <code>Since the yield strength of the steel is quite clearly defined and controlled, this establishes a very precise reference in structural investigations. An early design decision is that for the yield strength (specified by the Grade of steel used) that is to be used in the design work.Several different grades of steel may be used for large projects, with a minimum grade for ordinary tasks and higher grades for more demanding ones.ost steel used for reinforcement is highly ductile in nature. Its usable strength is its yield strength, as this stress condition initiates such a magnitude of deformation (into the plastic yielding range of the steel), that major cracking will occur in the concrete.</code> | <code>fy is the yield point of the material. E is the symbol for Young's Modulus of the material. E can be measured by dividing the elastic stress by the elastic strain.That is, this measurement must be made before the yield point of the material is reached.y is the yield point of the material. E is the symbol for Young's Modulus of the material. E can be measured by dividing the elastic stress by the elastic strain.</code> | <code>The longest dimension of the cant. WT is 13'. Using ASTM A992 carbon steel, a WT9x35.5 is at full bending stress and deflection limits. (Fy = 50 ksi). The only information I've found about using stainless for structural design is that type 304 is usually used.This yield strength (Fy) is only equal to 39 or 42ksi.sing ASTM A992 carbon steel, a WT9x35.5 is at full bending stress and deflection limits. (Fy = 50 ksi). The only information I've found about using stainless for structural design is that type 304 is usually used.</code> | <code>[0.5, 0.5]</code> |
| <code>most common nutritional deficiencies for teenagers</code> | <code>: Appendix B: Vitamin and Mineral Deficiencies in the U.S. Some American adults get too little vitamin D, vitamin E, magnesium, calcium, vitamin A and vitamin C (Table B1). More than 40 percent of adults have dietary intakes of vitamin A, C, D and E, calcium and magnesium below the average requirement for their age and gender. Inadequate intake of vitamins and minerals is most common among 14-to-18-year-old teenagers. Adolescent girls have lower nutrient intake than boys (Berner 2014; Fulgoni 2011). But nutrient deficiencies are rare among younger American children; the exceptions are dietary vitamin D and E, for which intake is low for all Americans, and calcium.</code> | <code>Common Nutritional Deficiencies. 10 Most Common Nutritional Deficiencies.. Calcium. Calcium is one of the most abundant minerals in your body, yet most people still manage to have a calcium deficiency. Calcium is best know for adding strength to your bones and teeth.</code> | <code>1) Vitamin D–Vitamin D deficiency is common in infants born to mothers with low levels of Vitamin D. Severe deficiency of this nutrient in infancy and early childhood can lead to the development of Rickets, a disease that affects bone formation and causes bow-legs.</code> | <code>[3.182860851287842, 7.834665775299072]</code> |
* Loss: [<code>SpladeLoss</code>](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
```json
{
"loss": "SparseMarginMSELoss",
"document_regularizer_weight": 0.2,
"query_regularizer_weight": 0.3
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: epoch
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 32
- `num_train_epochs`: 5
- `lr_scheduler_type`: cosine
- `warmup_ratio`: 0.05
- `fp16`: True
- `load_best_model_at_end`: True
- `optim`: adamw_torch_fused
- `push_to_hub`: True
#### All Hyperparameters
<details><summary>Click to expand</summary>
- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: epoch
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 32
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 5e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 5
- `max_steps`: -1
- `lr_scheduler_type`: cosine
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.05
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: True
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch_fused
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: True
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: None
- `hub_always_push`: False
- `hub_revision`: None
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`:
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `liger_kernel_config`: None
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: proportional
- `router_mapping`: {}
- `learning_rate_mapping`: {}
</details>
### Training Logs
| Epoch | Step | Training Loss | dot_ndcg@10 |
|:-------:|:----------:|:-------------:|:-----------:|
| 1.0 | 37500 | 11.4095 | 0.7103 |
| 2.0 | 75000 | 10.5305 | 0.7139 |
| 3.0 | 112500 | 9.5368 | 0.7197 |
| **4.0** | **150000** | **8.717** | **0.7216** |
| 5.0 | 187500 | 8.3094 | 0.7217 |
* The bold row denotes the saved checkpoint.
### Framework Versions
- Python: 3.11.13
- Sentence Transformers: 5.0.0
- Transformers: 4.54.0
- PyTorch: 2.6.0+cu124
- Accelerate: 1.8.1
- Datasets: 4.0.0
- Tokenizers: 0.21.2
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### SpladeLoss
```bibtex
@misc{formal2022distillationhardnegativesampling,
title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
year={2022},
eprint={2205.04733},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2205.04733},
}
```
#### SparseMarginMSELoss
```bibtex
@misc{hofstätter2021improving,
title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
year={2021},
eprint={2010.02666},
archivePrefix={arXiv},
primaryClass={cs.IR}
}
```
#### FlopsLoss
```bibtex
@article{paria2020minimizing,
title={Minimizing flops to learn efficient sparse representations},
author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
journal={arXiv preprint arXiv:2004.05665},
year={2020}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->
</details> |