--- language: - tr license: apache-2.0 tags: - sentence-transformers - cross-encoder - generated_from_trainer - dataset_size:89964 - loss:CachedMultipleNegativesRankingLoss base_model: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 datasets: - seroe/vodex-turkish-reranker-triplets pipeline_tag: text-ranking library_name: sentence-transformers metrics: - map - mrr@10 - ndcg@10 model-index: - name: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 results: - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: val hard type: val-hard metrics: - type: map value: 0.6093 name: Map - type: mrr@10 value: 0.6085 name: Mrr@10 - type: ndcg@10 value: 0.6994 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: test hard type: test-hard metrics: - type: map value: 0.6085 name: Map - type: mrr@10 value: 0.6077 name: Mrr@10 - type: ndcg@10 value: 0.6987 name: Ndcg@10 --- # cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [cross-encoder/mmarco-mMiniLMv2-L12-H384-v1](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1) on the [vodex-turkish-reranker-triplets](https://huggingface.co/datasets/seroe/vodex-turkish-reranker-triplets) dataset using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search. ## Model Details ## ⚠️ Domain-Specific Warning This model was fine-tuned on Turkish data specifically sourced from the **telecommunications domain**. While it performs well on telecom-related tasks such as mobile services, billing, campaigns, and subscription details, it may not generalize well to other domains. Please assess its performance carefully before applying it outside of telecommunications use cases. ### Model Description - **Model Type:** Cross Encoder - **Base model:** [cross-encoder/mmarco-mMiniLMv2-L12-H384-v1](https://huggingface.co/cross-encoder/mmarco-mMiniLMv2-L12-H384-v1) - **Maximum Sequence Length:** 512 tokens - **Number of Output Labels:** 1 label - **Training Dataset:** - [vodex-turkish-reranker-triplets](https://huggingface.co/datasets/seroe/vodex-turkish-reranker-triplets) - **Language:** tr - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the 🤗 Hub model = CrossEncoder("seroe/mmarco-mMiniLMv2-L12-H384-v1-turkish-reranker-triplet") # Get scores for pairs of texts pairs = [ ['Faturasız tarifelerde yurtdışı mesaj ücretleri ne kadardır?', 'Yurtdışına gönderilen mesajlar için ücret 75 kuruş olarak belirlenmiştir.'], ['Kampanya süresince internet hızı nasıl değişebilir?', 'Kampanya süresince, limit ve altyapının desteklediği azami internet hızına kadar internet hızı yükseltilebilir.'], ["Vodafone'un tarifelerinde KDV ve ÖİV dahil midir?", "Vodafone'un tarifelerinde belirtilen ücretlere KDV ve ÖİV dahildir."], ['Taahhüt süresi dolmadan internet hizmeti iptal edilirse ne olur?', 'Eğer taahhüt süresi bitmeden internet hizmeti iptal edilirse, aboneye sunulan D-Smart hizmeti de iptal edilecektir.'], ['Aylık 15 GB ek paketini nereden satın alabilirim?', 'Bu ek paketi almak için hangi kanalları kullanabilirim?'], ] scores = model.predict(pairs) print(scores.shape) # (5,) # Or rank different texts based on similarity to a single text ranks = model.rank( 'Faturasız tarifelerde yurtdışı mesaj ücretleri ne kadardır?', [ 'Yurtdışına gönderilen mesajlar için ücret 75 kuruş olarak belirlenmiştir.', 'Kampanya süresince, limit ve altyapının desteklediği azami internet hızına kadar internet hızı yükseltilebilir.', "Vodafone'un tarifelerinde belirtilen ücretlere KDV ve ÖİV dahildir.", 'Eğer taahhüt süresi bitmeden internet hizmeti iptal edilirse, aboneye sunulan D-Smart hizmeti de iptal edilecektir.', 'Bu ek paketi almak için hangi kanalları kullanabilirim?', ] ) # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] ``` ## Evaluation ### Metrics #### Cross Encoder Reranking * Datasets: `val-hard` and `test-hard` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": true } ``` | Metric | val-hard | test-hard | |:------------|:---------------------|:---------------------| | map | 0.6093 (-0.0246) | 0.6085 (-0.0178) | | mrr@10 | 0.6085 (-0.0254) | 0.6077 (-0.0186) | | **ndcg@10** | **0.6994 (+0.0641)** | **0.6987 (+0.0705)** | ## Training Details ### Training Dataset #### vodex-turkish-reranker-triplets * Dataset: [vodex-turkish-reranker-triplets](https://huggingface.co/datasets/seroe/vodex-turkish-reranker-triplets) at [ca7d206](https://huggingface.co/datasets/seroe/vodex-turkish-reranker-triplets/tree/ca7d2063ad4fec15fbf739835ab6926e051950c0) * Size: 89,964 training samples * Columns: query, positive, and negative * Approximate statistics based on the first 1000 samples: | | query | positive | negative | |:--------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | query | positive | negative | |:-------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------| | Faturasız tarifelerde yurtdışı mesaj ücretleri ne kadardır? | Yurtdışına gönderilen mesajlar için ücret 75 kuruş olarak belirlenmiştir. | Faturasız tarifelerde yurtdışı mesaj ücretleri 10 kuruş olarak uygulanmaktadır. | | Kampanya süresince internet hızı nasıl değişebilir? | Kampanya süresince, limit ve altyapının desteklediği azami internet hızına kadar internet hızı yükseltilebilir. | Kampanya süresince internet hızı sabit kalır ve değişiklik yapılamaz. | | Vodafone'un tarifelerinde KDV ve ÖİV dahil midir? | Vodafone'un tarifelerinde belirtilen ücretlere KDV ve ÖİV dahildir. | Vodafone tarifelerinde KDV ve ÖİV, abonelerin talep etmesi durumunda eklenmektedir. | * Loss: [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#cachedmultiplenegativesrankingloss) with these parameters: ```json { "scale": 10.0, "num_negatives": 4, "activation_fn": "torch.nn.modules.activation.Sigmoid", "mini_batch_size": 32 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 1024 - `per_device_eval_batch_size`: 1024 - `learning_rate`: 5e-07 - `weight_decay`: 0.1 - `max_grad_norm`: 0.8 - `warmup_ratio`: 0.25 - `bf16`: True - `dataloader_num_workers`: 8 - `load_best_model_at_end`: True - `group_by_length`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 1024 - `per_device_eval_batch_size`: 1024 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 5e-07 - `weight_decay`: 0.1 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 0.8 - `num_train_epochs`: 3 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.25 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 8 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: True - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | val-hard_ndcg@10 | test-hard_ndcg@10 | |:-----:|:----:|:-------------:|:----------------:|:-----------------:| | 1.125 | 100 | 1.3041 | 0.7093 (+0.0740) | 0.7065 (+0.0783) | | 2.25 | 200 | 0.9232 | 0.6994 (+0.0641) | 0.6987 (+0.0705) | ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 4.2.0.dev0 - Transformers: 4.46.3 - PyTorch: 2.5.1+cu124 - Accelerate: 1.6.0 - Datasets: 3.6.0 - Tokenizers: 0.20.3 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```