Hugging Face's logo

richie-ghost
/
sbert_facebook_large_mnli_openVino2

Sentence Similarity
sentence-transformers
OpenVINO
roberta
feature-extraction
Generated from Trainer
dataset_size:72338
loss:CosineSimilarityLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
sbert_facebook_large_mnli_openVino2 / README.md
richie-ghost's picture
richie-ghost
Updating model with the latest version
d706920 verified
preview code
|
raw
history blame contribute delete
16.5 kB
metadata
base_model: FacebookAI/roberta-large-mnli
library_name: sentence-transformers
metrics:
  - pearson_cosine
  - spearman_cosine
  - pearson_manhattan
  - spearman_manhattan
  - pearson_euclidean
  - spearman_euclidean
  - pearson_dot
  - spearman_dot
  - pearson_max
  - spearman_max
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:72338
  - loss:CosineSimilarityLoss
widget:
  - source_sentence: Do I need to know HTML & CSS to learn javascript?
    sentences:
      - What Would the Piano Chords to "Winter, You Tease" by Layla be?
      - Men playing a sport outside.
      - How do I learn web development as quickly as possible?
  - source_sentence: >-
      Isn't it inconsistent to prefer both a well-informed electorate and an
      ignorant jury?
    sentences:
      - Some people like when the electorate is stupid.
      - Two people working on computer
      - How is 0+0+0+0+0+0+0…= undefined?
  - source_sentence: >-
      A fluffy white and brown puppy is playing with a white, curly-haired
      puppy.
    sentences:
      - Why is H2O liquid and H2S solid at room temperature?
      - The bird is sitting in a nest.
      - The puppies are playing together.
  - source_sentence: A woman in a blue shirt and sunglasses dancing.
    sentences:
      - The woman is dancing.
      - Is Qatar part of UAE?
      - Two lovers walk together in Paris.
  - source_sentence: A motorbike rider is barreling across a grass lawn.
    sentences:
      - The girl is wearing a shirt.
      - Why doesn't Java have pointers?
      - The rider is outdoors on a motorbike.
model-index:
  - name: SentenceTransformer based on FacebookAI/roberta-large-mnli
    results:
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: eval
          type: eval
        metrics:
          - type: pearson_cosine
            value: 0.8457307745816387
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.810079801718123
            name: Spearman Cosine
          - type: pearson_manhattan
            value: 0.8108388961642436
            name: Pearson Manhattan
          - type: spearman_manhattan
            value: 0.7916598710432559
            name: Spearman Manhattan
          - type: pearson_euclidean
            value: 0.8106363007947738
            name: Pearson Euclidean
          - type: spearman_euclidean
            value: 0.7916399795577503
            name: Spearman Euclidean
          - type: pearson_dot
            value: 0.8566895266416593
            name: Pearson Dot
          - type: spearman_dot
            value: 0.8163029561419852
            name: Spearman Dot
          - type: pearson_max
            value: 0.8566895266416593
            name: Pearson Max
          - type: spearman_max
            value: 0.8163029561419852
            name: Spearman Max

SentenceTransformer based on FacebookAI/roberta-large-mnli

This is a sentence-transformers model finetuned from FacebookAI/roberta-large-mnli. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: FacebookAI/roberta-large-mnli
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 1024 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("richie-ghost/sbert_facebook_large_mnli_openVino2")
# Run inference
sentences = [
    'A motorbike rider is barreling across a grass lawn.',
    'The rider is outdoors on a motorbike.',
    'The girl is wearing a shirt.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

Metric Value
pearson_cosine 0.8457
spearman_cosine 0.8101
pearson_manhattan 0.8108
spearman_manhattan 0.7917
pearson_euclidean 0.8106
spearman_euclidean 0.7916
pearson_dot 0.8567
spearman_dot 0.8163
pearson_max 0.8567
spearman_max 0.8163

Training Details

Training Dataset

Unnamed Dataset

  • Size: 72,338 training samples
  • Columns: sentence_0, sentence_1, and label
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1 label
    type string string int
    details
    • min: 5 tokens
    • mean: 18.11 tokens
    • max: 82 tokens
    • min: 5 tokens
    • mean: 12.82 tokens
    • max: 65 tokens
    • 0: ~50.70%
    • 1: ~49.30%
  • Samples:
    sentence_0 sentence_1 label
    Hows would you create strategies and tactics in various combat situations? I have girlfriend and their parents accepted for my marriage, I m working in Nagpur but her parents wanted me to shift Bangalore? Is it valid wish? 0
    Man from the army speaking with civilian women. The man is a sergeant 0
    An old man with a white shirt and black pants sits on a chair in the opening of a stone tunnel. Someone has black pants. 1
  • Loss: CosineSimilarityLoss with these parameters:
    {
    "loss_fct": "torch.nn.modules.loss.MSELoss"
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • num_train_epochs: 4
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Epoch Step Training Loss eval_spearman_max
0.1106 500 0.1845 0.6681
0.2211 1000 0.0942 0.7711
0.3317 1500 0.0821 0.6355
0.4423 2000 0.0794 0.7283
0.5529 2500 0.0788 0.7129
0.6634 3000 0.0737 0.7853
0.7740 3500 0.07 0.7013
0.8846 4000 0.0686 0.7809
0.9951 4500 0.0683 0.7578
1.0 4522 - 0.7976
1.1057 5000 0.07 0.7749
1.2163 5500 0.0656 0.7826
1.3268 6000 0.0587 0.8032
1.4374 6500 0.0584 0.7666
1.5480 7000 0.0582 0.7917
1.6586 7500 0.0546 0.7945
1.7691 8000 0.0528 0.7786
1.8797 8500 0.051 0.7732
1.9903 9000 0.0527 0.7996
2.0 9044 - 0.7898
2.1008 9500 0.0509 0.7957
2.2114 10000 0.0492 0.7988
2.3220 10500 0.0451 0.8044
2.4326 11000 0.0443 0.7961
2.5431 11500 0.0445 0.7975
2.6537 12000 0.0433 0.8054
2.7643 12500 0.0394 0.7890
2.8748 13000 0.0387 0.8020
2.9854 13500 0.0401 0.8096
3.0 13566 - 0.8087
3.0960 14000 0.0399 0.8098
3.2065 14500 0.039 0.8077
3.3171 15000 0.0346 0.8021
3.4277 15500 0.0339 0.8082
3.5383 16000 0.0347 0.8150
3.6488 16500 0.0352 0.8144
3.7594 17000 0.032 0.8141
3.8700 17500 0.0326 0.8151
3.9805 18000 0.0318 0.8162
4.0 18088 - 0.8163

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.2.1
  • Transformers: 4.44.2
  • PyTorch: 2.4.1+cu121
  • Accelerate: 1.0.1
  • Datasets: 3.0.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}