IoannisKat1/modernbert-embed-base-legal-matryoshka-2

ModernBERT Embed base Legal Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("IoannisKat1/modernbert-embed-base-legal-matryoshka-2")
# Run inference
sentences = [
    'When did the plaintiff contact the bank after realizing the transfer?',
    "Court (Civil/Criminal): Civil  \nProvisions:  \nTime of commission: July 21, 2020  \nOutcome (innocent, guilty): Partially accepts the lawsuit against the Bank.  \nReasoning: Alongside the bank's responsibility for the ineffective protection of its systems against the continually changing and evolving methods of deception that affect them, leading to financial damage for its clients who entrusted it, which constitutes a breach of its contractual obligations and generates contractual liability, there is also the personal responsibility of the plaintiff, who did not safeguard, as required, his sensitive banking information and disclosed it when he should not have, even though the message that misled him came from the context of an SMS conversation with the defendant, even if he was directed through the link to a website resembling that of the defendant, because he overlooked the relevant instructions and strict recommendations of the bank, violating his duty to safeguard them. Therefore, in light of the above, the court must partially accept the lawsuit regarding its substance, and with the defendant's objection regarding the plaintiff's fault being accepted at a rate of 1/6 of the damage suffered, it must be acknowledged – following the admissible and lawful modification of the lawsuit's request from a dismissive to a declaratory nature as proposed – the defendant's obligation to pay him the amount of €2,500, with legal interest, as specified in the operative part. Regarding the request to recognize the defendant's obligation to pay him €1,000 as compensation for the moral damage he suffered from the fraud committed against him, this is deemed dismissible as unfounded, because the defendant's liability is contractual and not tortious, and there are no special conditions justifying the treatment of this specific non-contractual behavior as tortious, making it possible to attribute moral damage to the plaintiff from the defendant.  \nFacts: The plaintiff maintained a joint bank account with his wife and had access to the defendant bank's e-banking. On July 21, 2020, following successful identification via the agreed-upon credentials (userID, password, OTP), a request was made to change the mobile phone number for receiving OTPs, which was completed through the bank's platform. However, this change was not made by the plaintiff himself, but by a third party who had previously deceived him using the phishing method, obtaining his banking details through a misleading SMS. Following the phone number change, on July 31, 2020, a transfer of €3,000 was made to a third party's account. The transaction was conducted using the necessary credentials and OTP sent to the new mobile phone, resulting in the bank considering the order valid and executing it. Once the plaintiff realized the transfer, he immediately contacted the bank and informed them that he had not given such an order himself. Despite the initial assurance that the funds could be recovered, they were ultimately not returned, as they had been transferred to a third party's account, and the bank denied any responsibility for the loss.",
    'Everyone who obtains access to data recorded in a computer or in the external memory of a computer or transmitted by telecommunication systems shall be punished with imprisonment for up to six months or by a fine from 29 to 15,000 Euro, under the condition that these acts have been committed without right, especially in violation of prohibitions or of security measures taken by the legal holder. If the act concerns the international relations or the security of the State, he shall be punished according to Article 148.\nIf the offender is in the service of the legal holder of the data, the act of the preceding paragraph shall be punished only if it has been explicitly prohibited by internal regulations or by a written decision of the holder or of a competent employee of his.\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.5278
cosine_accuracy@3	0.5909
cosine_accuracy@5	0.6364
cosine_accuracy@10	0.6768
cosine_precision@1	0.5278
cosine_precision@3	0.5143
cosine_precision@5	0.4884
cosine_precision@10	0.4293
cosine_recall@1	0.108
cosine_recall@3	0.2761
cosine_recall@5	0.3695
cosine_recall@10	0.5013
cosine_ndcg@10	0.6047
cosine_mrr@10	0.5664
cosine_map@100	0.6541

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.5152
cosine_accuracy@3	0.5758
cosine_accuracy@5	0.6288
cosine_accuracy@10	0.6566
cosine_precision@1	0.5152
cosine_precision@3	0.5017
cosine_precision@5	0.4818
cosine_precision@10	0.4187
cosine_recall@1	0.1043
cosine_recall@3	0.2673
cosine_recall@5	0.3656
cosine_recall@10	0.4959
cosine_ndcg@10	0.5907
cosine_mrr@10	0.5524
cosine_map@100	0.642

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.5
cosine_accuracy@3	0.5556
cosine_accuracy@5	0.5884
cosine_accuracy@10	0.6212
cosine_precision@1	0.5
cosine_precision@3	0.4848
cosine_precision@5	0.4571
cosine_precision@10	0.3927
cosine_recall@1	0.1034
cosine_recall@3	0.2653
cosine_recall@5	0.3552
cosine_recall@10	0.4776
cosine_ndcg@10	0.5638
cosine_mrr@10	0.5318
cosine_map@100	0.6164

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.4646
cosine_accuracy@3	0.5025
cosine_accuracy@5	0.5278
cosine_accuracy@10	0.5808
cosine_precision@1	0.4646
cosine_precision@3	0.4411
cosine_precision@5	0.4086
cosine_precision@10	0.3561
cosine_recall@1	0.0979
cosine_recall@3	0.2449
cosine_recall@5	0.3223
cosine_recall@10	0.436
cosine_ndcg@10	0.5189
cosine_mrr@10	0.491
cosine_map@100	0.5721

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.3965
cosine_accuracy@3	0.4268
cosine_accuracy@5	0.4646
cosine_accuracy@10	0.5303
cosine_precision@1	0.3965
cosine_precision@3	0.3771
cosine_precision@5	0.351
cosine_precision@10	0.3134
cosine_recall@1	0.0831
cosine_recall@3	0.2116
cosine_recall@5	0.2794
cosine_recall@10	0.3901
cosine_ndcg@10	0.4539
cosine_mrr@10	0.4244
cosine_map@100	0.508

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,580 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 15.23 tokens max: 31 tokens	min: 25 tokens mean: 664.41 tokens max: 2429 tokens

• Samples:

  anchor	positive
  When was the obligation for the controller to respond to requests from the data subject set to take effect?	Modalities should be provided for facilitating the exercise of the data subject's rights under this Regulation, including mechanisms to request and, if applicable, obtain, free of charge, in particular, access to and rectification or erasure of personal data and the exercise of the right to object. The controller should also provide means for requests to be made electronically, especially where personal data are processed by electronic means. The controller should be obliged to respond to requests from the data subject without undue delay and at the latest within one month and to give reasons where the controller does not intend to comply with any such requests. 4.5.2016 L 119/11 Official Journal of the European Union EN
  Where can disputes be mediated or resolved?	The term technological measures means any technology, device or component designed to prevent or restrict unauthorized acts in respect of works or other subject-matter. Effective measures include access control or protection processes like encryption or copy control mechanisms. It is prohibited to circumvent effective technological measures without permission from the rightholder, when done knowingly or with reasonable grounds to know. It is prohibited to manufacture, import, distribute, sell, rent, advertise, or possess for commercial purposes devices or services intended to circumvent technological measures. Violation of the provisions is punishable by imprisonment of at least one year and a fine of 2,900 to 15,000 Euro. It also entails civil sanctions under article 65 of Law 2121/1993. Rightholders must ensure exceptions for reproduction for private use, teaching, libraries, judicial purposes, or accessibility for people with disabilities. Disputes may be mediated or resolved by the...
  How much was the defendant ordered to pay?	Court (Civil/Criminal): Provisions: Time of commission of the act: Outcome (not guilty, guilty): ORDERS the defendant to pay the plaintiff the amount of two thousand four hundred thirty-four euros and eighty-three cents (€2,434.83) with legal interest from the service of the lawsuit. Reasoning: Law 4537/2018 introduces mandatory provisions in favor of users, as according to Article 103, payment service providers are prohibited from deviating from the provisions to the detriment of payment service users, unless the possibility of deviation is expressly provided, and they can decide to offer only more favorable terms to payment service users. Under this law and its provisions, providers are only liable when there are unusual and unforeseen circumstances beyond the control of the party invoking them, and whose consequences could not have been avoided despite efforts to the contrary. However, operational risks and security risks of the system do not constitute unusual and unforeseen circu...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• num_train_epochs: 15
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 15
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.2020	10	20.6078	-	-	-	-	-
0.4040	20	18.0629	-	-	-	-	-
0.6061	30	15.9938	-	-	-	-	-
0.8081	40	14.5545	-	-	-	-	-
1.0	50	13.721	0.4888	0.4758	0.4745	0.4213	0.3679
1.2020	60	10.4947	-	-	-	-	-
1.4040	70	10.9842	-	-	-	-	-
1.6061	80	8.6381	-	-	-	-	-
1.8081	90	8.8047	-	-	-	-	-
2.0	100	9.37	0.5576	0.5402	0.5027	0.4600	0.3758
2.2020	110	6.4456	-	-	-	-	-
2.4040	120	5.5425	-	-	-	-	-
2.6061	130	6.4737	-	-	-	-	-
2.8081	140	7.6558	-	-	-	-	-
3.0	150	5.8459	0.5611	0.5559	0.5354	0.4581	0.3917
3.2020	160	3.7962	-	-	-	-	-
3.4040	170	4.4323	-	-	-	-	-
3.6061	180	3.8344	-	-	-	-	-
3.8081	190	3.9649	-	-	-	-	-
4.0	200	3.0804	0.5781	0.5710	0.5283	0.4579	0.4072
4.2020	210	2.9219	-	-	-	-	-
4.4040	220	3.2647	-	-	-	-	-
4.6061	230	2.9356	-	-	-	-	-
4.8081	240	3.6304	-	-	-	-	-
5.0	250	2.3513	0.5810	0.5763	0.5436	0.4821	0.4373
5.2020	260	2.13	-	-	-	-	-
5.4040	270	2.5041	-	-	-	-	-
5.6061	280	2.0452	-	-	-	-	-
5.8081	290	1.9489	-	-	-	-	-
6.0	300	1.6624	0.5752	0.5690	0.5353	0.4911	0.4196
6.2020	310	2.2382	-	-	-	-	-
6.4040	320	1.4628	-	-	-	-	-
6.6061	330	1.9498	-	-	-	-	-
6.8081	340	1.9256	-	-	-	-	-
7.0	350	1.9855	0.5806	0.5766	0.5635	0.4943	0.4399
7.2020	360	1.7137	-	-	-	-	-
7.4040	370	1.8472	-	-	-	-	-
7.6061	380	1.3159	-	-	-	-	-
7.8081	390	1.7977	-	-	-	-	-
8.0	400	1.4496	0.5842	0.5857	0.5411	0.4937	0.4451
8.2020	410	1.3037	-	-	-	-	-
8.4040	420	1.4886	-	-	-	-	-
8.6061	430	1.0403	-	-	-	-	-
8.8081	440	0.8165	-	-	-	-	-
9.0	450	1.4516	0.5870	0.6003	0.5632	0.4997	0.4415
9.2020	460	0.8564	-	-	-	-	-
9.4040	470	0.8697	-	-	-	-	-
9.6061	480	0.9484	-	-	-	-	-
9.8081	490	0.6718	-	-	-	-	-
10.0	500	0.7662	0.5873	0.5851	0.5501	0.4940	0.4555
10.2020	510	0.7644	-	-	-	-	-
10.4040	520	1.0033	-	-	-	-	-
10.6061	530	0.7571	-	-	-	-	-
10.8081	540	0.6782	-	-	-	-	-
11.0	550	0.9404	0.5954	0.5903	0.5594	0.5066	0.4628
11.2020	560	0.5852	-	-	-	-	-
11.4040	570	0.8594	-	-	-	-	-
11.6061	580	0.9976	-	-	-	-	-
11.8081	590	1.0234	-	-	-	-	-
12.0	600	0.7846	0.5996	0.5888	0.5684	0.5145	0.4575
12.2020	610	0.6015	-	-	-	-	-
12.4040	620	0.6266	-	-	-	-	-
12.6061	630	0.6573	-	-	-	-	-
12.8081	640	0.6517	-	-	-	-	-
13.0	650	0.4972	0.6047	0.5907	0.5638	0.5189	0.4539
13.2020	660	0.6062	-	-	-	-	-
13.4040	670	0.691	-	-	-	-	-
13.6061	680	0.5038	-	-	-	-	-
13.8081	690	0.9533	-	-	-	-	-
14.0	700	0.5026	0.5965	0.5934	0.5673	0.5186	0.4632
14.2020	710	0.6599	-	-	-	-	-
14.4040	720	0.756	-	-	-	-	-
14.6061	730	0.8242	-	-	-	-	-
14.7071	735	-	0.5989	0.5955	0.5664	0.5152	0.4648
-1	-1	-	0.6047	0.5907	0.5638	0.5189	0.4539

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 4.0.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}