abdulmunimjemal/amharic-xlmr-finetuned

SentenceTransformer based on sentence-transformers/paraphrase-xlm-r-multilingual-v1

This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-xlm-r-multilingual-v1 on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/paraphrase-xlm-r-multilingual-v1
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • csv

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("abdulmunimjemal/amharic-xlmr-finetuned")
# Run inference
sentences = [
    'ሰዎች ተቀምጠዋል',
    'ሁለት ሰዎች አንድ ልብ በምድር ውስጥ ከተቀመጠባቸው ጥቂት እግሮች ጥቂት መሬት ተቀምጠዋል.',
    'ሰዎች እየተራመዱ ሲሆን አንድ ወንድ ጭንቅላቱን ወደ ግራ ዞሯል.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Datasets: amharic-xlmr-nli-dev and amharic-xlmr-finetuned-dev
• Evaluated with TripletEvaluator

Metric	amharic-xlmr-nli-dev	amharic-xlmr-finetuned-dev
cosine_accuracy	0.815	0.8575

Training Details

Training Dataset

csv

• Dataset: csv
• Size: 40,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 6.5 tokens max: 7 tokens	min: 10 tokens mean: 24.81 tokens max: 63 tokens	min: 18 tokens mean: 26.34 tokens max: 38 tokens

• Samples:

  anchor	positive	negative
  ሰውየው ውጭ ነው.	በቤቶች በተከበበችው የኪሳ ፓርክ ውስጥ ረዥም ፀጉር ያለው አንድ ሰው የመንሸራተቻ ሰሌዳ ነው.	በወንድ ወንበር ላይ በወንድ ወንበር ላይ በመዝገቢያ ወረቀት ላይ በመዝጋት ላይ ይተኛል.
  ሰውየው ውጭ ነው።	ረጅም ፀጉር ያለው ሰው በቤቶች በተከበበ የበረዶ መንሸራተቻ ፓርክ ውስጥ በስኬትቦርዲንግ ላይ ነው።	በቀን ውስጥ አንድ ሰው በአንድ ክፍል ውስጥ የእጅ መቆንጠጫ ይሠራል.
  ሰውየው ውጭ ነው.	በቤቶች በተከበበችው የኪሳ ፓርክ ውስጥ ረዥም ፀጉር ያለው አንድ ሰው የመንሸራተቻ ሰሌዳ ነው.	ባለሙያ የለበሰ ሰው ከኮንሶል ፊት ለፊት ተቀምጧል።

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 5
  }
  

Evaluation Dataset

csv

• Dataset: csv
• Size: 40,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 5 tokens mean: 13.91 tokens max: 56 tokens	min: 5 tokens mean: 20.28 tokens max: 67 tokens	min: 5 tokens mean: 21.07 tokens max: 55 tokens

• Samples:

  anchor	positive	negative
  ቡናማ ውሻ እየሮጠ እና እየተመለከተ ነው.	ቡናማ ውሻ ወደ ሰማይ እያየ ይሮጣል.	ከዛፉ ስር አንድ ቡናማ ውሻ ይዞ ይገኛል.
  በወርቃማ ቀለም ያለው ቀለም ያለው ቆዳዎች በሣር ውስጥ.	ውሻ በሳሩ ውስጥ ነው.	ውሻ ውጭ እየተጣደፈ ነው.
  ቡናማ ውሻ ከቤት ውጭ እየተጫወተ ነው.	አንድ እንስሳ ውጭ ነው.	ቡናማ ውሻ በኩሽና ውስጥ እየበላ ነው.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 5
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• gradient_accumulation_steps: 2
• learning_rate: 1e-05
• weight_decay: 0.01
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	amharic-xlmr-nli-dev_cosine_accuracy	amharic-xlmr-finetuned-dev_cosine_accuracy
0	0	-	-	0.6432	-
0.05	100	4.6673	4.3076	0.8492	-
0.1	200	4.1006	3.6344	0.821	-
0.15	300	3.843	4.1666	0.7652	-
0.2	400	4.0508	3.8094	0.815	-
0.25	500	3.9858	-	-	0.8237
0.2	100	4.15	-	-	-
0.4	200	4.1811	-	-	-
0.6	300	4.3359	-	-	-
0.8	400	4.382	-	-	-
1.0	500	3.6309	3.5175	-	0.858
1.198	600	4.1283	-	-	-
1.3980	700	4.0372	-	-	-
1.5980	800	4.2113	-	-	-
1.798	900	4.059	-	-	-
1.998	1000	3.4594	3.5366	-	0.8565
2.196	1100	4.0407	-	-	-
2.396	1200	3.9531	-	-	-
2.596	1300	4.1321	-	-	-
2.7960	1400	3.9537	-	-	-
2.996	1500	3.4291	3.5476	-	0.8575

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}