modernbert-embed-quickb / README.md

Mdean77

Add new SentenceTransformer model

f62ab3b verified 7 months ago

preview code

raw

history blame

31.4 kB

metadata

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:1567
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: nomic-ai/modernbert-embed-base
widget:
  - source_sentence: How many authors are listed for the trial?
    sentences:
      - >-
        chemotherapy and bone marrow transplantation for certain malignancies
        and has a long track

        record of safe use in adults and children. The incidence of adverse
        events such as fever, chills,

        bone pain, dyspnea, tachycardia, and hemodynamic instability was no
        different between GM-

        CSF and placebo-treated groups in controlled adult BMT studies. Rapid IV
        administration of
      - >-
        clinical ICU staff in accordance with institutional practice and
        judgment.

        Child Assent Subjects who are eligible for this study will be critically
        ill, and child assent is

        typically not possible at the time of study enrollment. However, during
        follow up after discharge

        from the ICU, issues about assent become applicable. Children who are
        capable of giving assent
      - >-
        Controlled Phase 2 Trial. Stroke, 49(5):1210–1216, 2018.

        [76] M. K. R. Somagutta, M. K. Lourdes Pormento, P. Hamid, A. Hamdan, M.
        A. Khan,

        R. Desir, R. Vijayan, S. Shirke, R. Jeyakumar, Z. Dogar, S. S. Makkar,
        P. Guntipalli,

        N. N. Ngardig, M. S. Nagineni, T. Paul, E. Luvsannyam, C. Riddick, and
        M. A. Sanchez-
  - source_sentence: What type of event can lead to the suspension of enrollment in the study?
    sentences:
      - >-
        and data generated by this study must be available for inspection upon
        request by representatives

        (when applicable) of the Food and Drug Administration (FDA), NIH, other
        Federal funders or

        study sponsors, and the Institutional Review Board (IRB) for each study
        site.

        9 Protection of Human Subjects

        9.1 Risks to Human Subjects

        9.1.1 Human Subjects Involvement and Characteristics
      - >-
        two consecutive days while receiving study drug, the drug will be
        discontinued.

        Adverse events will be monitored as described in Section 10.2.6 on page
        61. The medical

        monitor has the authority to suspend enrollment in the event of an
        unexpected, study-related

        serious adverse event that is judged to change the risk/benefit of
        subject participation.
      - >-
        innate immune system is common and measurable in pediatric sepsis.
        Innate immune cells such

        as monocytes and neutrophils serve critical functions including
        migration to sites of infection,

        phagocytosis of pathogens, promotion of microbial killing, antigen
        presentation, and production

        of immunomodulatory cytokines. We have repeatedly shown that severe
        reduction in the ability
  - source_sentence: When will the reviews start?
    sentences:
      - >-
        mg/kg/day given for three days by continuous infusion was used.23, 63
        Despite its apparent safety

        in adults, this dose is substantially higher than what has been used in
        children with HLH/MAS

        or adults with COVID-19.

        In the largest (to date) published study of anakinra in hospitalized,
        hyper-inflamed adults

        with COVID-19 (N=392), a dose of 10 mg/kg/day IV divided every 12 hours
        (infused over 1
      - >-
        data are required for Federal reporting purposes to delineate subject
        accrual by race, ethnicity,

        and gender.

        For purposes of the DCC handling potential protected health information
        (PHI) and pro-

        ducing the de–identified research data sets that will be used for
        analyses, all study sites have

        been offered a Business Associate Agreement with the University of Utah.
        Copies of executed
      - >-
        empirically whether these patients differ from those remaining in the
        study for the scheduled

        treatment and follow-up time. Missingness for primary, secondary,
        exploratory, and safety

        outcomes will be reviewed in aggregate and by site. Reviews will start
        as soon as enrollment

        opens and will be regulatory monitored so missing data problems can be
        addressed early in the

        study.
  - source_sentence: >-
      What type of results will be communicated to the Data Coordinating Center
      and clinical site investigator?
    sentences:
      - >-
        ing of a medical condition that was present at the time of randomization
        will be considered a

        new adverse event and reported.

        After patient randomization all adverse events (including serious
        adverse events) will be

        recorded according to relatedness, severity, and expectedness, as well
        as their duration and
      - >-
        12.2 Health Insurance Portability and Accountability Act

        Data elements collected include the date of birth and date of admission.
        Prior to statistical

        analyses, dates will be used to calculate patient age at the time of the
        study events.

        Data elements for race, ethnicity, and gender are also being collected.
        These demographic
      - >-
        The Collaborative Pediatric Critical Care Research NetworkPage 34 of 76
        Protocol 90 (Hall, Zuppa and Mourani)

        4.5 Randomization

        Upon determination of a subject’s immunophenotype, Dr. Hall or his
        designee will notify the

        Data Coordinating Center and the clinical site investigator of the
        laboratory results. Subjects
  - source_sentence: What age groups will be enrolled in the study?
    sentences:
      - >-
        have mild to moderate inflammation (i.e. a serum ferritin level <2,000
        ng/ml) from the TRIPS

        trial. Those subjects will be instead entered into a completely distinct
        clinical trial of immune

        stimulation with GM-CSF (GRACE-2) that is covered by a separate IND
        (#112277).

        PRECISE Protocol Version 1.07

        Protocol Version Date: June 16, 2023
      - >-
        Subject Population to be Studied Participating sites will enroll
        infants, children and adoles-

        cent patients who are admitted to a Pediatric or Cardiac Intensive Care
        Unit with sepsis-induced

        multiple organ dysfunction syndrome (MODS). The goal is to determine if
        personalized im-

        munomodulation is an effective strategy to reduce mortality and
        morbidity from sepsis-induced
      - >-
        Loosdregt, N. M. Wulffraat, S. de Roock, and S. J. Vastert. Treatment to
        target using

        recombinant interleukin-1 receptor antagonist as first-line monotherapy
        in new-onset

        systemic juvenile idiopathic arthritis: Results from a five-year
        follow-up study. Arthritis

        Rheumatol, 71(7):1163–1173, 2019.

        [78] R. K. Thakkar, R. Devine, J. Popelka, J. Hensley, R. Fabia, J. A.
        Muszynski, and M. W.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: Fine-tuned with [QuicKB](https://github.com/ALucek/QuicKB)
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.5714285714285714
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.7828571428571428
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8114285714285714
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8742857142857143
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.5714285714285714
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2609523809523809
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16228571428571423
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08742857142857141
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.5714285714285714
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.7828571428571428
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8114285714285714
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8742857142857143
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7304617900805063
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.6836485260770975
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6898282619821292
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.5485714285714286
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.7885714285714286
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8285714285714286
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8685714285714285
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.5485714285714286
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2628571428571428
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16571428571428568
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08685714285714283
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.5485714285714286
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.7885714285714286
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8285714285714286
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8685714285714285
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7172419802927883
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.6675759637188208
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6741729815259775
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.5485714285714286
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.76
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.84
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9085714285714286
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.5485714285714286
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2533333333333333
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16799999999999995
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09085714285714283
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.5485714285714286
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.76
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.84
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9085714285714286
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7268936400245406
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.6687596371882085
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6719911574054431
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.49142857142857144
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.7028571428571428
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.7885714285714286
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8685714285714285
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.49142857142857144
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.23428571428571424
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.15771428571428567
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08685714285714284
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.49142857142857144
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.7028571428571428
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.7885714285714286
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8685714285714285
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.6778419592624233
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.6168730158730158
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6219971103464577
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.38285714285714284
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.5714285714285714
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.6571428571428571
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.7885714285714286
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.38285714285714284
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.19047619047619044
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1314285714285714
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.07885714285714283
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.38285714285714284
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.5714285714285714
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.6571428571428571
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.7885714285714286
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.5697625172066919
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.5015079365079367
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.5090522718083348
            name: Cosine Map@100

Fine-tuned with QuicKB

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: nomic-ai/modernbert-embed-base
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Mdean77/modernbert-embed-quickb")
# Run inference
sentences = [
    'What age groups will be enrolled in the study?',
    'Subject Population to be Studied Participating sites will enroll infants, children and adoles-\ncent patients who are admitted to a Pediatric or Cardiac Intensive Care Unit with sepsis-induced\nmultiple organ dysfunction syndrome (MODS). The goal is to determine if personalized im-\nmunomodulation is an effective strategy to reduce mortality and morbidity from sepsis-induced',
    'have mild to moderate inflammation (i.e. a serum ferritin level <2,000 ng/ml) from the TRIPS\ntrial. Those subjects will be instead entered into a completely distinct clinical trial of immune\nstimulation with GM-CSF (GRACE-2) that is covered by a separate IND (#112277).\nPRECISE Protocol Version 1.07\nProtocol Version Date: June 16, 2023',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.5714	0.5486	0.5486	0.4914	0.3829
cosine_accuracy@3	0.7829	0.7886	0.76	0.7029	0.5714
cosine_accuracy@5	0.8114	0.8286	0.84	0.7886	0.6571
cosine_accuracy@10	0.8743	0.8686	0.9086	0.8686	0.7886
cosine_precision@1	0.5714	0.5486	0.5486	0.4914	0.3829
cosine_precision@3	0.261	0.2629	0.2533	0.2343	0.1905
cosine_precision@5	0.1623	0.1657	0.168	0.1577	0.1314
cosine_precision@10	0.0874	0.0869	0.0909	0.0869	0.0789
cosine_recall@1	0.5714	0.5486	0.5486	0.4914	0.3829
cosine_recall@3	0.7829	0.7886	0.76	0.7029	0.5714
cosine_recall@5	0.8114	0.8286	0.84	0.7886	0.6571
cosine_recall@10	0.8743	0.8686	0.9086	0.8686	0.7886
cosine_ndcg@10	0.7305	0.7172	0.7269	0.6778	0.5698
cosine_mrr@10	0.6836	0.6676	0.6688	0.6169	0.5015
cosine_map@100	0.6898	0.6742	0.672	0.622	0.5091

Training Details

Training Dataset

Unnamed Dataset

Size: 1,567 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 8 tokens
mean: 15.03 tokens
max: 30 tokens

min: 57 tokens
mean: 90.85 tokens
max: 185 tokens

	anchor	positive
type	string	string
details	min: 8 tokens mean: 15.03 tokens max: 30 tokens	min: 57 tokens mean: 90.85 tokens max: 185 tokens

Samples:

anchor	positive
`How many terabytes of data are referenced?`	`over 125 terabytes of data. Information systems are available 24/7/365 unless a scheduled maintenance period or mitigation of an unexpected event is required. Critical systems availability has exceeded 99.9% for the past 5 years. 7.2.3 Security, Support, Encryption, and Confidentiality The data center coordinates the network infrastructure and security with University Information`
`What regulation allows single parent permission for the study?`	`for their child in the study. Single parent permission is permitted under 45 CFR §46.405. The parent or legal guardian will be informed about the objectives of the study and the potential risks and benefits of their child’s participation. If the parent or legal guardian refuses permission for their child to participate, then all clinical management will continue to be provided by the`
`What is included in the follow-up plan for non-compliant sites?`	`planned site visits, criteria for focused visits, additional visits or remote monitoring, a plan for chart review and a follow up plan for non-compliant sites. The monitoring plan also describes the type of monitoring that will take place (e.g., sample of all subjects within a site; key data or all data), the schedule of visits, how they are reported and a time frame to resolve any issues found.`

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 16
gradient_accumulation_steps: 16
learning_rate: 2e-05
num_train_epochs: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
tf32: False
load_best_model_at_end: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 16
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: False
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
1.0	7	-	0.6698	0.6606	0.6458	0.6146	0.5049
1.4898	10	55.7211	-	-	-	-	-
2.0	14	-	0.7210	0.7080	0.7183	0.6653	0.5621
2.9796	20	26.9161	-	-	-	-	-
3.0	21	-	0.7309	0.7172	0.7262	0.6762	0.5694
3.4898	24	-	0.7305	0.7172	0.7269	0.6778	0.5698

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.3
Sentence Transformers: 3.4.1
Transformers: 4.49.0
PyTorch: 2.6.0
Accelerate: 1.4.0
Datasets: 3.3.2
Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}