wilsonmarciliojr/matryoshka-embed-b16

SentenceTransformer based on distilbert/distilroberta-base

This is a sentence-transformers model finetuned from distilbert/distilroberta-base on the all-nli-hard-negatives dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilroberta-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • all-nli-hard-negatives

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("wilsonmarciliojr/matryoshka-embed-b16")
# Run inference
sentences = [
    'A man reading the paper at a cafe.',
    'A man starring at a piece of paper.',
    'A man is sitting.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4274, 0.2306],
#         [0.4274, 1.0000, 0.3276],
#         [0.2306, 0.3276, 1.0000]])

Evaluation

Metrics

Semantic Similarity

• Datasets: sts-dev-768 and sts-test-768
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	sts-dev-768	sts-test-768
pearson_cosine	0.8448	0.7969
spearman_cosine	0.8425	0.7943

Semantic Similarity

• Datasets: sts-dev-512 and sts-test-512
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	sts-dev-512	sts-test-512
pearson_cosine	0.8443	0.7963
spearman_cosine	0.8424	0.7951

Semantic Similarity

• Datasets: sts-dev-256 and sts-test-256
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	sts-dev-256	sts-test-256
pearson_cosine	0.8399	0.7917
spearman_cosine	0.8394	0.7937

Semantic Similarity

• Datasets: sts-dev-64 and sts-test-64
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	sts-dev-64	sts-test-64
pearson_cosine	0.8186	0.7759
spearman_cosine	0.8249	0.783

Semantic Similarity

• Datasets: sts-dev-2 and sts-test-2
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 2
  }
  

Metric	sts-dev-2	sts-test-2
pearson_cosine	0.3108	0.3026
spearman_cosine	0.438	0.3979

Training Details

Training Dataset

all-nli-hard-negatives

• Dataset: all-nli-hard-negatives at 9e4fbfd
• Size: 200,266 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 7 tokens mean: 15.96 tokens max: 50 tokens	min: 4 tokens mean: 10.29 tokens max: 22 tokens	min: 4 tokens mean: 9.31 tokens max: 24 tokens	min: 4 tokens mean: 9.21 tokens max: 24 tokens	min: 4 tokens mean: 9.16 tokens max: 24 tokens	min: 4 tokens mean: 9.14 tokens max: 22 tokens	min: 4 tokens mean: 9.34 tokens max: 27 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  An older man is drinking orange juice at a restaurant.	A man is drinking juice.	A man seated at a restaurant.	The older man is making food.	the guy in the orange shirt has food in front of him	An elderly person is being served food	A man wears an orange shirt.
  A man with blond-hair, and a brown shirt drinking out of a public water fountain.	A blond man drinking water from a fountain.	Man having a drink.	A man is playing in the fountain.	A man is drinking something.	The water fountain is wet.	This man is wet
  Two women, holding food carryout containers, hug.	Two women hug each other.	The two woman are holding their arms	Both women have things in their hands.	Two woman standing near each other while one of them holds an item.	Two women carry bags	Two people give each other a hug.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          64,
          2
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

all-nli-hard-negatives

• Dataset: all-nli-hard-negatives at 9e4fbfd
• Size: 6,494 evaluation samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 6 tokens mean: 17.84 tokens max: 66 tokens	min: 4 tokens mean: 9.91 tokens max: 29 tokens	min: 4 tokens mean: 9.22 tokens max: 27 tokens	min: 5 tokens mean: 9.38 tokens max: 25 tokens	min: 4 tokens mean: 9.28 tokens max: 29 tokens	min: 4 tokens mean: 9.35 tokens max: 41 tokens	min: 4 tokens mean: 9.54 tokens max: 41 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  Two women are embracing while holding to go packages.	Two woman are holding packages.	A group of women with flowers.	There are women relaxing.	Women are holding a flag	A woman is holding one young children with another standing next to her	An old woman is carrying two pails.
  Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink.	Two kids in numbered jerseys wash their hands.	Children are walking.	THe children are playing.	Two kids are playing outside.	The people have clothes on.	Children are playing a game
  A man selling donuts to a customer during a world exhibition event held in the city of Angeles	A man selling donuts to a customer.	A man is giving a presentation.	Goods and Services are sold by undercover agents..	It is called Service Merchandise here.	A street vendor is outside.	I'm happy that I don't work in a store.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          64,
          2
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	sts-dev-768_spearman_cosine	sts-dev-512_spearman_cosine	sts-dev-256_spearman_cosine	sts-dev-64_spearman_cosine	sts-dev-2_spearman_cosine	sts-test-768_spearman_cosine	sts-test-512_spearman_cosine	sts-test-256_spearman_cosine	sts-test-64_spearman_cosine	sts-test-2_spearman_cosine
0.0399	500	9.7558	5.4971	0.8366	0.8394	0.8364	0.8116	0.3516	-	-	-	-	-
0.0799	1000	5.8126	5.3466	0.8265	0.8262	0.8221	0.7994	0.3604	-	-	-	-	-
0.1198	1500	5.6179	5.3155	0.8365	0.8342	0.8297	0.8074	0.3820	-	-	-	-	-
0.1598	2000	5.4337	5.0697	0.8392	0.8383	0.8347	0.8125	0.4098	-	-	-	-	-
0.1997	2500	5.2708	5.1591	0.8320	0.8296	0.8259	0.8051	0.3725	-	-	-	-	-
0.2397	3000	5.2173	5.1042	0.8300	0.8291	0.8255	0.8044	0.3829	-	-	-	-	-
0.2796	3500	5.0789	4.9034	0.8299	0.8277	0.8231	0.7996	0.3710	-	-	-	-	-
0.3196	4000	5.1056	4.7930	0.8419	0.8415	0.8372	0.8162	0.4059	-	-	-	-	-
0.3595	4500	4.9763	4.8291	0.8422	0.8406	0.8355	0.8159	0.3812	-	-	-	-	-
0.3995	5000	4.9729	4.9692	0.8452	0.8435	0.8405	0.8238	0.4114	-	-	-	-	-
0.4394	5500	4.7542	4.9973	0.8394	0.8387	0.8357	0.8190	0.3756	-	-	-	-	-
0.4793	6000	4.7886	4.7710	0.8475	0.8475	0.8453	0.8285	0.4099	-	-	-	-	-
0.5193	6500	4.7406	4.6588	0.8417	0.8415	0.8385	0.8232	0.4347	-	-	-	-	-
0.5592	7000	4.7258	4.7451	0.8465	0.8460	0.8436	0.8249	0.4287	-	-	-	-	-
0.5992	7500	4.5991	4.7196	0.8396	0.8386	0.8357	0.8207	0.4000	-	-	-	-	-
0.6391	8000	4.5976	4.6225	0.8416	0.8409	0.8369	0.8183	0.4153	-	-	-	-	-
0.6791	8500	4.5607	4.6760	0.8407	0.8412	0.8372	0.8208	0.4317	-	-	-	-	-
0.7190	9000	4.5406	4.6796	0.8399	0.8395	0.8359	0.8197	0.4321	-	-	-	-	-
0.7590	9500	4.5464	4.5882	0.8395	0.8395	0.8366	0.8205	0.4305	-	-	-	-	-
0.7989	10000	4.4328	4.5758	0.8427	0.8426	0.8400	0.8245	0.4210	-	-	-	-	-
0.8389	10500	4.495	4.5487	0.8402	0.8402	0.8367	0.8214	0.4146	-	-	-	-	-
0.8788	11000	4.392	4.5094	0.8420	0.8419	0.8390	0.8244	0.4408	-	-	-	-	-
0.9188	11500	4.4206	4.4939	0.8431	0.8437	0.8403	0.8266	0.4411	-	-	-	-	-
0.9587	12000	4.3311	4.4776	0.8415	0.8413	0.8384	0.8236	0.4378	-	-	-	-	-
0.9986	12500	4.3707	4.4892	0.8425	0.8424	0.8394	0.8249	0.4380	-	-	-	-	-
-1	-1	-	-	-	-	-	-	-	0.7943	0.7951	0.7937	0.7830	0.3979

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.0.0
• Transformers: 4.53.1
• PyTorch: 2.7.1+cu126
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}