wilsonmarciliojr/matryoshka-embed-b512

SentenceTransformer based on distilbert/distilroberta-base

This is a sentence-transformers model finetuned from distilbert/distilroberta-base on the all-nli-hard-negatives dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilroberta-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • all-nli-hard-negatives

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("wilsonmarciliojr/matryoshka-embed-b512")
# Run inference
sentences = [
    'A man reading the paper at a cafe.',
    'A man starring at a piece of paper.',
    'A man is sitting.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4452, 0.2729],
#         [0.4452, 1.0000, 0.2601],
#         [0.2729, 0.2601, 1.0000]])

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-test-768
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
pearson_cosine	0.8117
spearman_cosine	0.8046

Semantic Similarity

• Dataset: sts-test-512
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
pearson_cosine	0.8104
spearman_cosine	0.804

Semantic Similarity

• Dataset: sts-test-256
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
pearson_cosine	0.8071
spearman_cosine	0.8041

Semantic Similarity

• Dataset: sts-test-64
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
pearson_cosine	0.7907
spearman_cosine	0.7932

Semantic Similarity

• Dataset: sts-test-2
• Evaluated with EmbeddingSimilarityEvaluator with these parameters:

  {
      "truncate_dim": 2
  }
  

Metric	Value
pearson_cosine	0.3114
spearman_cosine	0.423

Training Details

Training Dataset

all-nli-hard-negatives

• Dataset: all-nli-hard-negatives at 9e4fbfd
• Size: 200,266 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 7 tokens mean: 15.96 tokens max: 50 tokens	min: 4 tokens mean: 10.29 tokens max: 22 tokens	min: 4 tokens mean: 9.31 tokens max: 24 tokens	min: 4 tokens mean: 9.21 tokens max: 24 tokens	min: 4 tokens mean: 9.16 tokens max: 24 tokens	min: 4 tokens mean: 9.14 tokens max: 22 tokens	min: 4 tokens mean: 9.34 tokens max: 27 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  An older man is drinking orange juice at a restaurant.	A man is drinking juice.	A man seated at a restaurant.	The older man is making food.	the guy in the orange shirt has food in front of him	An elderly person is being served food	A man wears an orange shirt.
  A man with blond-hair, and a brown shirt drinking out of a public water fountain.	A blond man drinking water from a fountain.	Man having a drink.	A man is playing in the fountain.	A man is drinking something.	The water fountain is wet.	This man is wet
  Two women, holding food carryout containers, hug.	Two women hug each other.	The two woman are holding their arms	Both women have things in their hands.	Two woman standing near each other while one of them holds an item.	Two women carry bags	Two people give each other a hug.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          64,
          2
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

all-nli-hard-negatives

• Dataset: all-nli-hard-negatives at 9e4fbfd
• Size: 6,494 evaluation samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 6 tokens mean: 17.84 tokens max: 66 tokens	min: 4 tokens mean: 9.91 tokens max: 29 tokens	min: 4 tokens mean: 9.22 tokens max: 27 tokens	min: 5 tokens mean: 9.38 tokens max: 25 tokens	min: 4 tokens mean: 9.28 tokens max: 29 tokens	min: 4 tokens mean: 9.35 tokens max: 41 tokens	min: 4 tokens mean: 9.54 tokens max: 41 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  Two women are embracing while holding to go packages.	Two woman are holding packages.	A group of women with flowers.	There are women relaxing.	Women are holding a flag	A woman is holding one young children with another standing next to her	An old woman is carrying two pails.
  Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink.	Two kids in numbered jerseys wash their hands.	Children are walking.	THe children are playing.	Two kids are playing outside.	The people have clothes on.	Children are playing a game
  A man selling donuts to a customer during a world exhibition event held in the city of Angeles	A man selling donuts to a customer.	A man is giving a presentation.	Goods and Services are sold by undercover agents..	It is called Service Merchandise here.	A street vendor is outside.	I'm happy that I don't work in a store.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          64,
          2
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	sts-test-768_spearman_cosine	sts-test-512_spearman_cosine	sts-test-256_spearman_cosine	sts-test-64_spearman_cosine	sts-test-2_spearman_cosine
-1	-1	0.8046	0.8040	0.8041	0.7932	0.4230

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.0.0
• Transformers: 4.53.1
• PyTorch: 2.7.1+cu126
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}