SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: Snowflake/snowflake-arctic-embed-l
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 1024 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AneetaXavier/reformer-pilates-embed-ft-49fc1835-9968-433d-9c45-1538ea91dcc9")
# Run inference
sentences = [
    'What modifications are suggested if the exercise feels too intense on the arms or wrists?',
    "spine relax the shoulders lift the head\nand again\nhead goes down Pike it up inhale\nand exhale roll through\n[Applause]\ngood keep going here if this is too\nintense on the arms or wrists especially\nyou're going to do the same thing here\nPike it up\non your knees and roll through my knees\nare just kind of facing over to the left\nside Pike it up\ninhale and exhale roll\ngood two more you guys you're doing so\ngood it's intense I know\nroll through\nand lift\nlast one\nand finishing that Pike good you guys\ntake those feet\nonto the carriage catch your breath if\nyou want lean it back if you can lift\nyour foot bar to find that click to kind\nof lean back stretch through your\nshoulders kind of depending on your\nreformer if yours is able to pull back",
    "towards the spine but keep the spine in\na neutral position fully straighten the\nlegs when you straighten them and now\ninto VMO knock-knees okay so your toes\nare exactly where they are you push out\nkeeping the knees together go all the\nway back into the stopper and then\nwithin that range you're going to do 20\nof these so the knees are together\nthroughout the whole of the exercise the\ntoes are on the bar as they were in the\nV position but then the heels are out\nwider so it's like a knocked knee this\nreally gets into the muscles on the\ninside of the knees and the inside of\nthe legs in through the nose out through\nthe mouth\nexpanding the ribs and then contracting\nthe abdominals keeping the muscles in\nthe legs engaged throughout prehensile",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.7333
cosine_accuracy@3 0.9667
cosine_accuracy@5 1.0
cosine_accuracy@10 1.0
cosine_precision@1 0.7333
cosine_precision@3 0.3222
cosine_precision@5 0.2
cosine_precision@10 0.1
cosine_recall@1 0.7333
cosine_recall@3 0.9667
cosine_recall@5 1.0
cosine_recall@10 1.0
cosine_ndcg@10 0.876
cosine_mrr@10 0.8344
cosine_map@100 0.8344

Training Details

Training Dataset

Unnamed Dataset

  • Size: 120 training samples
  • Columns: sentence_0 and sentence_1
  • Approximate statistics based on the first 120 samples:
    sentence_0 sentence_1
    type string string
    details
    • min: 12 tokens
    • mean: 18.46 tokens
    • max: 29 tokens
    • min: 85 tokens
    • mean: 158.07 tokens
    • max: 173 tokens
  • Samples:
    sentence_0 sentence_1
    What equipment and spring settings does Dez recommend for starting the Pilates reformer workout? [Music]
    foreign
    [Music]
    hey guys welcome back to my channel I'm
    Dez and today I'm taking you through
    another full body Pilates reformer
    workout this workout includes some fun
    and challenging series and will give you
    a full class experience you won't need
    any additional props today just you and
    your reformer so let's get started
    okay you guys we're going to start today
    on two heavy Springs with hip rolls so
    if you need additional assistance for
    your low back add on also a light to
    medium tension spring I'm going to be
    going to two heavy Springs or two Reds
    on this machine
    and we're going to light on on our box
    head rest will be down flat
    to protect the neck
    good we're going to place our heels on
    the bar
    find your neutral spine
    How does Dez suggest protecting the neck during the hip rolls exercise? [Music]
    foreign
    [Music]
    hey guys welcome back to my channel I'm
    Dez and today I'm taking you through
    another full body Pilates reformer
    workout this workout includes some fun
    and challenging series and will give you
    a full class experience you won't need
    any additional props today just you and
    your reformer so let's get started
    okay you guys we're going to start today
    on two heavy Springs with hip rolls so
    if you need additional assistance for
    your low back add on also a light to
    medium tension spring I'm going to be
    going to two heavy Springs or two Reds
    on this machine
    and we're going to light on on our box
    head rest will be down flat
    to protect the neck
    good we're going to place our heels on
    the bar
    find your neutral spine
    What is the correct breathing technique to use while rocking between imprint and neutral spine positions? heels on the bar hip distance there we
    go inhale
    exhale we're just going to tuck into our
    imprint
    pressing that low back down activating
    the core and inhale Rock back
    and exhale press that low back down
    going into your imprinted spine and then
    rocking back to your neutral
    good keep that breathing going we're
    thinking just ribs towards your hips as
    you rock into that imprint and then Rock
    back
    one more time
    and rock it back this time we're going
    to roll all the way up press that low
    back down and then scoop the hips use
    the hamstrings and glutes to roll up we
    want to keep the carriage into the
    stopper that's the challenging part
    inhale and then exhale soften from the
    ribs and roll back down one vertebrae at
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "MultipleNegativesRankingLoss",
        "matryoshka_dims": [
            768,
            512,
            256,
            128,
            64
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 10
  • per_device_eval_batch_size: 10
  • num_train_epochs: 30
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 10
  • per_device_eval_batch_size: 10
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 30
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • tp_size: 0
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Epoch Step cosine_ndcg@10
1.0 12 0.8455
2.0 24 0.8970
3.0 36 0.9064
4.0 48 0.9237
4.1667 50 0.9360
5.0 60 0.8633
6.0 72 0.9016
7.0 84 0.8814
8.0 96 0.8676
8.3333 100 0.8599
9.0 108 0.8633
10.0 120 0.8903
11.0 132 0.8760
12.0 144 0.8793
12.5 150 0.8960
13.0 156 0.8970
14.0 168 0.8970
15.0 180 0.9026
16.0 192 0.8903
16.6667 200 0.8804
17.0 204 0.8927
18.0 216 0.9093
19.0 228 0.8960
20.0 240 0.8916
20.8333 250 0.8916
21.0 252 0.8916
22.0 264 0.8927
23.0 276 0.8916
24.0 288 0.8916
25.0 300 0.8750
26.0 312 0.8750
27.0 324 0.8627
28.0 336 0.8637
29.0 348 0.8760
29.1667 350 0.8760
30.0 360 0.8760

Framework Versions

  • Python: 3.11.12
  • Sentence Transformers: 4.1.0
  • Transformers: 4.51.3
  • PyTorch: 2.6.0+cu124
  • Accelerate: 1.6.0
  • Datasets: 2.14.4
  • Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
2
Safetensors
Model size
335M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for dwb2023/artic-embed-sw-ft-12979a9b-5e10-426a-a140-66385a68406c

Finetuned
(167)
this model

Evaluation results