SPLADE-BERT-Mini

This is a SPLADE Sparse Encoder model finetuned from prajjwal1/bert-mini using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: prajjwal1/bert-mini
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Language: en
• License: mit

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("rasyosef/SPLADE-BERT-Mini")
# Run inference
queries = [
    "cantigny gardens cost",
]
documents = [
    'The fee for a ceremony ranges from $400 to $2,500 with reception rental or $3,000 for a ceremony-only wedding. Please inquire about discounted rates for ceremony guest counts under 75. The average wedding cost at Cantigny Park is estimated at between $12,881 and $22,238 for a ceremony & reception for 100 guests.',
    'Nestled in a serene setting, Cantigny Park is a scenic realm where you will create a unique wedding, the memories of which you will always cherish. This expansive estate encompasses 500 acres of beautiful gardens, colorful botanicals and tranquil water features, creating an idyllic background for this ideal day.',
    'Cantigny Park. Cantigny is a 500-acre (2.0 km2) park in Wheaton, Illinois, 30 miles west of Chicago. It is the former estate of Joseph Medill and his grandson Colonel Robert R. McCormick, publishers of the Chicago Tribune, and is open to the public.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[18.8703, 13.8253, 13.4587]])

Evaluation

Metrics

Sparse Information Retrieval

• Evaluated with SparseInformationRetrievalEvaluator

Metric	Value
dot_accuracy@1	0.6303
dot_accuracy@3	0.7972
dot_accuracy@5	0.851
dot_accuracy@10	0.9055
dot_precision@1	0.6303
dot_precision@3	0.2657
dot_precision@5	0.1702
dot_precision@10	0.0905
dot_recall@1	0.6303
dot_recall@3	0.7972
dot_recall@5	0.851
dot_recall@10	0.9055
dot_ndcg@10	0.769
dot_mrr@10	0.7251
dot_map@100	0.7288
query_active_dims	26.4352
query_sparsity_ratio	0.9991
corpus_active_dims	326.676
corpus_sparsity_ratio	0.9893

Training Details

Training Dataset

Unnamed Dataset

• Size: 250,000 training samples
• Columns: query, positive, negative_1, and negative_2
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative_1	negative_2
  type	string	string	string	string
  details	min: 4 tokens mean: 8.87 tokens max: 31 tokens	min: 20 tokens mean: 82.54 tokens max: 218 tokens	min: 20 tokens mean: 79.98 tokens max: 252 tokens	min: 19 tokens mean: 80.55 tokens max: 211 tokens

• Samples:

  query	positive	negative_1	negative_2
  how do automotive technicians get paid	104 months ago. The amount of pay from company to company does not vary too much, but you do have a wide variety of compensation methods. There are various combinations of hourly and commission pay rates, which depending on what type of work you specialize in can vary your bottom line considerably.04 months ago. The amount of pay from company to company does not vary too much, but you do have a wide variety of compensation methods. There are various combinations of hourly and commission pay rates, which depending on what type of work you specialize in can vary your bottom line considerably.	Bureau of Labor Statistics figures indicate that automotive technicians earned an average annual salary of $38,560 and an average hourly wage of $18.54 as of May 2011.Half of auto technicians reported annual salaries of between $26,850 and $47,540 and hourly wages of between $12.91 and $22.86.The 10 percent of automotive techs who earned the lowest made $20,620 or less per year, and the top 10 percent of earners made $59,600 or more per year.ver one-third of all automotive technicians employed as of May 2011 worked in the automotive repair and maintenance industry, where they earned an average of $35,090 per year.	It really depends on what automaker your working for, how much experience you have, and how long you've been in the industry. Obviously if you're working for a highend company(BMW,Mercedes,Ferrari) you can expect to be paid more per hour. And automotive technicians don't get paid by the hour.We get paid per FLAT RATE hour. Which basically means that we get paid by the job. Which could range from 0.2 of an hour for replacing a headlight bulb to 10hours for a transmission overhaul. Then there's a difference between warranty jobs and cash jobs.ut I won't get into too much detail. Automotive technicians get paid around $12-$15/hr at entry level. But can make around $18-$26/hr with much more experience. Which means you can expect to make 30,000 to 60,000/year. Though most technicians don't see past 45,000 a year.
  how far is steamboat springs from golden?	The distance between Steamboat Springs and Golden in a straight line is 100 miles or 160.9 Kilometers. Driving Directions & Drive Times from Steamboat Springs to Golden can be found further down the page.	Steamboat Springs Vacation Rentals Steamboat Springs Vacations Steamboat Springs Restaurants Things to Do in Steamboat Springs Steamboat Springs Travel Forum Steamboat Springs Photos Steamboat Springs Map Steamboat Springs Travel Guide All Steamboat Springs Hotels; Steamboat Springs Hotel Deals; Last Minute Hotels in Steamboat Springs; By Hotel Type Steamboat Springs Family Hotels	There are 98.92 miles from Golden to Steamboat Springs in northwest direction and 143 miles (230.14 kilometers) by car, following the US-40 route. Golden and Steamboat Springs are 3 hours 20 mins far apart, if you drive non-stop. This is the fastest route from Golden, CO to Steamboat Springs, CO. The halfway point is Heeney, CO. Golden, CO and Steamboat Springs, CO are in the same time zone (MDT). Current time in both locations is 1:26 pm.
  incoming wire routing number for california bank and trust	Please call California Bank And Trust representative at (888) 315-2271 for more information. 1 Routing Number: 122003396. 2 250 EAST FIRST STREET # 700. LOS ANGELES, CA 90012-0000. 3 Phone Number: (888) 315-2271.	When asked to provide a routing number for incoming wire transfers to Union Bank accounts, the routing number to use is: 122000496. back to top What options do I have to send wires?	Business Contracting Officers (BCO) have access to Online Banking wires. Simply sign on to Online Banking, click “Send Wires”, and then complete the required information. This particular service is limited to sending wires to U.S. banks only.

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
      "document_regularizer_weight": 0.001,
      "query_regularizer_weight": 0.002
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 6e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.025
• fp16: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 6e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.025
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dot_ndcg@10
1.0	3907	23.8846	0.7509
2.0	7814	0.785	0.7670
3.0	11721	0.6873	0.7685
4.0	15628	0.6283	0.7690
-1	-1	-	0.7690

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.0.0
• Transformers: 4.53.1
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}