rjomega/modernbert-embed-base-legal-matryoshka-2

ModernBERT Embed base Legal Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("rjomega/modernbert-embed-base-legal-matryoshka-2")
# Run inference
sentences = [
    'filed.  See id.  In light of this factual predicate, the CIA does not explain how requiring NSC to \nfile a new FOIA request would have resulted in no delay.  The CIA would have first needed to \nprocess the new request, and although “the legwork for the request ha[d] [already] been \ncompleted,” Fifth Lutz Decl. ¶ 12, the CIA would have admittedly had to assess NSC’s fee status',
    'What request would result in delay according to the CIA?',
    'What do the solicitations include for WOSB or SDVOSB offerors?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.524
cosine_accuracy@3	0.5626
cosine_accuracy@5	0.6584
cosine_accuracy@10	0.7141
cosine_precision@1	0.524
cosine_precision@3	0.4967
cosine_precision@5	0.3765
cosine_precision@10	0.2195
cosine_recall@1	0.1868
cosine_recall@3	0.4918
cosine_recall@5	0.6077
cosine_recall@10	0.7035
cosine_ndcg@10	0.6187
cosine_mrr@10	0.5671
cosine_map@100	0.6087

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.5379
cosine_accuracy@3	0.5734
cosine_accuracy@5	0.6584
cosine_accuracy@10	0.7094
cosine_precision@1	0.5379
cosine_precision@3	0.5064
cosine_precision@5	0.3808
cosine_precision@10	0.2195
cosine_recall@1	0.1929
cosine_recall@3	0.5023
cosine_recall@5	0.6136
cosine_recall@10	0.7002
cosine_ndcg@10	0.6246
cosine_mrr@10	0.5774
cosine_map@100	0.6168

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.4977
cosine_accuracy@3	0.5348
cosine_accuracy@5	0.6136
cosine_accuracy@10	0.6832
cosine_precision@1	0.4977
cosine_precision@3	0.4719
cosine_precision@5	0.3521
cosine_precision@10	0.2099
cosine_recall@1	0.1784
cosine_recall@3	0.4674
cosine_recall@5	0.5657
cosine_recall@10	0.6686
cosine_ndcg@10	0.5872
cosine_mrr@10	0.5376
cosine_map@100	0.5791

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.4266
cosine_accuracy@3	0.459
cosine_accuracy@5	0.5611
cosine_accuracy@10	0.6291
cosine_precision@1	0.4266
cosine_precision@3	0.4013
cosine_precision@5	0.3128
cosine_precision@10	0.1915
cosine_recall@1	0.1556
cosine_recall@3	0.3991
cosine_recall@5	0.5063
cosine_recall@10	0.615
cosine_ndcg@10	0.525
cosine_mrr@10	0.4702
cosine_map@100	0.5147

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.3029
cosine_accuracy@3	0.3385
cosine_accuracy@5	0.4266
cosine_accuracy@10	0.4915
cosine_precision@1	0.3029
cosine_precision@3	0.2875
cosine_precision@5	0.2328
cosine_precision@10	0.1516
cosine_recall@1	0.1118
cosine_recall@3	0.289
cosine_recall@5	0.377
cosine_recall@10	0.4825
cosine_ndcg@10	0.3982
cosine_mrr@10	0.3446
cosine_map@100	0.3907

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 28 tokens mean: 96.33 tokens max: 160 tokens	min: 8 tokens mean: 16.49 tokens max: 46 tokens

• Samples:

  positive	anchor
  DIA’s withholding of responsive information under FOIA Exemption 5, see supra Parts III.J.1(b), III.J.2(b), and the Court grants summary judgment to the DIA in all other respects. • The Court denies summary judgment to the ODNI on Count Six in No. 11-445. See supra Parts III.H.3, III.J.1(b), III.J.2(b). • The Court grants in part and denies in part summary judgment to the CIA on Count	In the document number mentioned, what is the count related to ODNI?
  requirements, EPIC’s entitlement to mandamus relief is straightforward. The party seeking mandamus has the burden of showing “(1) a clear and indisputable right to relief, (2) that the government agency or official is violating a clear duty to act, and (3) that no adequate alternative remedy exists.” Am. Hosp. Ass’n v. Burwell, 812 F.3d 183, 189 (D.C. Cir. 2016).12	Which Circuit is mentioned in the case citation?
  3 responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT”). I. Background ¶ 5 Star Hearthstone rented an apartment to Al-Hamim and his cotenants in April 2020. Al-Hamim alleged in his complaint that IRT Living managed the apartment complex for a portion of the time	In what month and year did Star Hearthstone rent an apartment to Al-Hamim?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8791	10	5.6524	-	-	-	-	-
1.0	12	-	0.5976	0.5885	0.5490	0.4790	0.3413
1.7033	20	2.4407	-	-	-	-	-
2.0	24	-	0.6174	0.6173	0.5789	0.5182	0.3828
2.5275	30	1.7536	-	-	-	-	-
3.0	36	-	0.6187	0.6254	0.5859	0.5244	0.3978
3.3516	40	1.5993	-	-	-	-	-
4.0	48	-	0.6187	0.6246	0.5872	0.525	0.3982

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.53.0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}