--- language: - en license: apache-2.0 tags: - cross-encoder - sentence-transformers - text-classification - sentence-pair-classification - semantic-similarity - semantic-search - retrieval - reranking - generated_from_trainer - dataset_size:1047690 - loss:BinaryCrossEntropyLoss base_model: Alibaba-NLP/gte-reranker-modernbert-base datasets: - aditeyabaral-redis/langcache-sentencepairs-v1 pipeline_tag: text-ranking library_name: sentence-transformers metrics: - accuracy - accuracy_threshold - f1 - f1_threshold - precision - recall - average_precision model-index: - name: Redis fine-tuned CrossEncoder model for semantic caching on LangCache results: - task: type: cross-encoder-classification name: Cross Encoder Classification dataset: name: val type: val metrics: - type: accuracy value: 0.773111243307555 name: Accuracy - type: accuracy_threshold value: 0.7637044787406921 name: Accuracy Threshold - type: f1 value: 0.6950724637681159 name: F1 - type: f1_threshold value: 0.04638597369194031 name: F1 Threshold - type: precision value: 0.6454912516823688 name: Precision - type: recall value: 0.7529042386185243 name: Recall - type: average_precision value: 0.7833280130154174 name: Average Precision - task: type: cross-encoder-classification name: Cross Encoder Classification dataset: name: test type: test metrics: - type: accuracy value: 0.7230292965285952 name: Accuracy - type: accuracy_threshold value: 0.9352303147315979 name: Accuracy Threshold - type: f1 value: 0.7144263194410831 name: F1 - type: f1_threshold value: 0.9142870903015137 name: F1 Threshold - type: precision value: 0.6302559284880577 name: Precision - type: recall value: 0.8245437616387337 name: Recall - type: average_precision value: 0.6906882331078481 name: Average Precision --- # Redis fine-tuned CrossEncoder model for semantic caching on LangCache This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [Alibaba-NLP/gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) on the [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) dataset using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for sentence pair classification. ## Model Details ### Model Description - **Model Type:** Cross Encoder - **Base model:** [Alibaba-NLP/gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) - **Maximum Sequence Length:** 8192 tokens - **Number of Output Labels:** 1 label - **Training Dataset:** - [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the 🤗 Hub model = CrossEncoder("aditeyabaral-redis/langcache-reranker-v1-wdwr") # Get scores for pairs of texts pairs = [ ["He said the foodservice pie business doesn 't fit the company 's long-term growth strategy .", '" The foodservice pie business does not fit our long-term growth strategy .'], ['Magnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war .', 'His wife said he was " 100 percent behind George Bush " and looked forward to using his years of training in the war .'], ['The dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat .', 'The dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .'], ['The AFL-CIO is waiting until October to decide if it will endorse a candidate .', 'The AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .'], ['No dates have been set for the civil or the criminal trial .', 'No dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .'], ] scores = model.predict(pairs) print(scores.shape) # (5,) # Or rank different texts based on similarity to a single text ranks = model.rank( "He said the foodservice pie business doesn 't fit the company 's long-term growth strategy .", [ '" The foodservice pie business does not fit our long-term growth strategy .', 'His wife said he was " 100 percent behind George Bush " and looked forward to using his years of training in the war .', 'The dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .', 'The AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .', 'No dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .', ] ) # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] ``` ## Evaluation ### Metrics #### Cross Encoder Classification * Datasets: `val` and `test` * Evaluated with [CrossEncoderClassificationEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderClassificationEvaluator) | Metric | val | test | |:----------------------|:-----------|:-----------| | accuracy | 0.7731 | 0.723 | | accuracy_threshold | 0.7637 | 0.9352 | | f1 | 0.6951 | 0.7144 | | f1_threshold | 0.0464 | 0.9143 | | precision | 0.6455 | 0.6303 | | recall | 0.7529 | 0.8245 | | **average_precision** | **0.7833** | **0.6907** | ## Training Details ### Training Dataset #### LangCache Sentence Pairs (all) * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) * Size: 8,405 training samples * Columns: sentence1, sentence2, and label * Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:-------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | sentence1 | sentence2 | label | |:--------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | He said the foodservice pie business doesn 't fit the company 's long-term growth strategy . | " The foodservice pie business does not fit our long-term growth strategy . | 1 | | Magnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war . | His wife said he was " 100 percent behind George Bush " and looked forward to using his years of training in the war . | 0 | | The dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat . | The dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent . | 0 | * Loss: [BinaryCrossEntropyLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: ```json { "activation_fn": "torch.nn.modules.linear.Identity", "pos_weight": null } ``` ### Evaluation Dataset #### LangCache Sentence Pairs (all) * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) * Size: 8,405 evaluation samples * Columns: sentence1, sentence2, and label * Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:-------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | sentence1 | sentence2 | label | |:--------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | He said the foodservice pie business doesn 't fit the company 's long-term growth strategy . | " The foodservice pie business does not fit our long-term growth strategy . | 1 | | Magnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war . | His wife said he was " 100 percent behind George Bush " and looked forward to using his years of training in the war . | 0 | | The dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat . | The dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent . | 0 | * Loss: [BinaryCrossEntropyLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: ```json { "activation_fn": "torch.nn.modules.linear.Identity", "pos_weight": null } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 48 - `per_device_eval_batch_size`: 48 - `learning_rate`: 0.0002 - `weight_decay`: 0.01 - `num_train_epochs`: 20 - `warmup_ratio`: 0.1 - `load_best_model_at_end`: True - `optim`: adamw_torch - `push_to_hub`: True - `hub_model_id`: aditeyabaral-redis/langcache-reranker-v1-wdwr #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 48 - `per_device_eval_batch_size`: 48 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 0.0002 - `weight_decay`: 0.01 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 20 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: True - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: True - `resume_from_checkpoint`: None - `hub_model_id`: aditeyabaral-redis/langcache-reranker-v1-wdwr - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `hub_revision`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `liger_kernel_config`: None - `eval_use_gather_object`: False - `average_tokens_across_devices`: True - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs
Click to expand | Epoch | Step | Training Loss | Validation Loss | val_average_precision | test_average_precision | |:----------:|:--------:|:-------------:|:---------------:|:---------------------:|:----------------------:| | -1 | -1 | - | - | 0.7676 | 0.6907 | | 0.1833 | 1000 | 0.3563 | 0.4805 | 0.7831 | - | | **0.3666** | **2000** | **0.2065** | **0.5394** | **0.8221** | **-** | | 0.5499 | 3000 | 0.1983 | 0.5019 | 0.8178 | - | | 0.7331 | 4000 | 0.1923 | 0.5109 | 0.7960 | - | | 0.9164 | 5000 | 0.1886 | 0.4726 | 0.8058 | - | | 1.0997 | 6000 | 0.183 | 0.5062 | 0.8032 | - | | 1.2830 | 7000 | 0.1838 | 0.5152 | 0.8021 | - | | 1.4663 | 8000 | 0.1858 | 0.5105 | 0.7926 | - | | 1.6496 | 9000 | 0.1905 | 0.5052 | 0.7859 | - | | 1.8328 | 10000 | 0.1926 | 0.5316 | 0.7895 | - | | 2.0161 | 11000 | 0.1951 | 0.5340 | 0.7681 | - | | 2.1994 | 12000 | 0.1853 | 0.5573 | 0.7577 | - | | 2.3827 | 13000 | 0.1848 | 0.5530 | 0.7946 | - | | 2.5660 | 14000 | 0.1813 | 0.5754 | 0.7655 | - | | 2.7493 | 15000 | 0.1793 | 0.5316 | 0.7514 | - | | 2.9326 | 16000 | 0.1778 | 0.5230 | 0.7868 | - | | 3.1158 | 17000 | 0.1681 | 0.5246 | 0.7816 | - | | 3.2991 | 18000 | 0.1662 | 0.4946 | 0.7732 | - | | 3.4824 | 19000 | 0.1648 | 0.5262 | 0.7853 | - | | 3.6657 | 20000 | 0.1649 | 0.5007 | 0.7871 | - | | 3.8490 | 21000 | 0.1633 | 0.5368 | 0.7807 | - | | 4.0323 | 22000 | 0.1602 | 0.5559 | 0.7769 | - | | 4.2155 | 23000 | 0.149 | 0.5796 | 0.7697 | - | | 4.3988 | 24000 | 0.1486 | 0.5322 | 0.7608 | - | | 4.5821 | 25000 | 0.1495 | 0.5142 | 0.7713 | - | | 4.7654 | 26000 | 0.1493 | 0.5203 | 0.7866 | - | | 4.9487 | 27000 | 0.1498 | 0.5433 | 0.7738 | - | | 5.1320 | 28000 | 0.1391 | 0.5589 | 0.7803 | - | | 5.3152 | 29000 | 0.1346 | 0.5267 | 0.7713 | - | | 5.4985 | 30000 | 0.1367 | 0.5657 | 0.7803 | - | | 5.6818 | 31000 | 0.1358 | 0.5631 | 0.7646 | - | | 5.8651 | 32000 | 0.136 | 0.5444 | 0.7753 | - | | 6.0484 | 33000 | 0.1346 | 0.5605 | 0.7703 | - | | 6.2317 | 34000 | 0.1222 | 0.5399 | 0.7776 | - | | 6.4150 | 35000 | 0.1241 | 0.5272 | 0.7899 | - | | 6.5982 | 36000 | 0.1243 | 0.6096 | 0.7723 | - | | 6.7815 | 37000 | 0.1266 | 0.5661 | 0.7609 | - | | 6.9648 | 38000 | 0.1246 | 0.5341 | 0.7889 | - | | 7.1481 | 39000 | 0.1128 | 0.6223 | 0.7884 | - | | 7.3314 | 40000 | 0.1124 | 0.5485 | 0.7743 | - | | 7.5147 | 41000 | 0.1127 | 0.5375 | 0.7842 | - | | 7.6979 | 42000 | 0.1122 | 0.5231 | 0.7939 | - | | 7.8812 | 43000 | 0.1141 | 0.5608 | 0.7705 | - | | 8.0645 | 44000 | 0.1088 | 0.6511 | 0.7813 | - | | 8.2478 | 45000 | 0.0998 | 0.6217 | 0.7648 | - | | 8.4311 | 46000 | 0.1017 | 0.6000 | 0.7822 | - | | 8.6144 | 47000 | 0.1031 | 0.5469 | 0.7866 | - | | 8.7977 | 48000 | 0.1012 | 0.5862 | 0.7790 | - | | 8.9809 | 49000 | 0.1031 | 0.5527 | 0.7876 | - | | 9.1642 | 50000 | 0.0921 | 0.5460 | 0.7788 | - | | 9.3475 | 51000 | 0.0909 | 0.5820 | 0.7815 | - | | 9.5308 | 52000 | 0.0919 | 0.5589 | 0.7841 | - | | 9.7141 | 53000 | 0.0939 | 0.5521 | 0.7821 | - | | 9.8974 | 54000 | 0.0925 | 0.6942 | 0.7797 | - | | 10.0806 | 55000 | 0.0863 | 0.6208 | 0.7729 | - | | 10.2639 | 56000 | 0.0803 | 0.6632 | 0.7911 | - | | 10.4472 | 57000 | 0.0797 | 0.6583 | 0.7833 | - | | 10.6305 | 58000 | 0.0824 | 0.6194 | 0.7862 | - | | 10.8138 | 59000 | 0.0829 | 0.6136 | 0.7783 | - | | 10.9971 | 60000 | 0.0819 | 0.5833 | 0.7727 | - | | 11.1804 | 61000 | 0.0693 | 0.6491 | 0.7881 | - | | 11.3636 | 62000 | 0.0709 | 0.6449 | 0.7784 | - | | 11.5469 | 63000 | 0.0721 | 0.6158 | 0.7838 | - | | 11.7302 | 64000 | 0.0721 | 0.6649 | 0.7841 | - | | 11.9135 | 65000 | 0.0732 | 0.6403 | 0.7702 | - | | 12.0968 | 66000 | 0.0679 | 0.6079 | 0.7817 | - | | 12.2801 | 67000 | 0.0615 | 0.6862 | 0.7787 | - | | 12.4633 | 68000 | 0.0629 | 0.7239 | 0.7824 | - | | 12.6466 | 69000 | 0.0643 | 0.6419 | 0.7897 | - | | 12.8299 | 70000 | 0.0635 | 0.6743 | 0.7762 | - | | 13.0132 | 71000 | 0.064 | 0.7135 | 0.7741 | - | | 13.1965 | 72000 | 0.0545 | 0.6643 | 0.7723 | - | | 13.3798 | 73000 | 0.0548 | 0.6508 | 0.7758 | - | | 13.5630 | 74000 | 0.0547 | 0.7003 | 0.7785 | - | | 13.7463 | 75000 | 0.0548 | 0.7170 | 0.7846 | - | | 13.9296 | 76000 | 0.0553 | 0.6917 | 0.7722 | - | | 14.1129 | 77000 | 0.0508 | 0.7000 | 0.7767 | - | | 14.2962 | 78000 | 0.0474 | 0.7336 | 0.7730 | - | | 14.4795 | 79000 | 0.0465 | 0.7122 | 0.7795 | - | | 14.6628 | 80000 | 0.0478 | 0.7321 | 0.7779 | - | | 14.8460 | 81000 | 0.0468 | 0.7112 | 0.7796 | - | | 15.0293 | 82000 | 0.0465 | 0.7534 | 0.7788 | - | | 15.2126 | 83000 | 0.0395 | 0.7238 | 0.7808 | - | | 15.3959 | 84000 | 0.0401 | 0.7686 | 0.7905 | - | | 15.5792 | 85000 | 0.0408 | 0.7296 | 0.7900 | - | | 15.7625 | 86000 | 0.0414 | 0.7533 | 0.7822 | - | | 15.9457 | 87000 | 0.0402 | 0.7748 | 0.7867 | - | | 16.1290 | 88000 | 0.0352 | 0.8267 | 0.7844 | - | | 16.3123 | 89000 | 0.0354 | 0.7488 | 0.7912 | - | | 16.4956 | 90000 | 0.0337 | 0.7850 | 0.7857 | - | | 16.6789 | 91000 | 0.0333 | 0.7812 | 0.7815 | - | | 16.8622 | 92000 | 0.0341 | 0.8184 | 0.7786 | - | | 17.0455 | 93000 | 0.0333 | 0.8166 | 0.7781 | - | | 17.2287 | 94000 | 0.0288 | 0.7980 | 0.7803 | - | | 17.4120 | 95000 | 0.0282 | 0.8195 | 0.7774 | - | | 17.5953 | 96000 | 0.0285 | 0.7864 | 0.7829 | - | | 17.7786 | 97000 | 0.0284 | 0.8000 | 0.7838 | - | | 17.9619 | 98000 | 0.0279 | 0.8118 | 0.7873 | - | | 18.1452 | 99000 | 0.0245 | 0.8727 | 0.7866 | - | | 18.3284 | 100000 | 0.0235 | 0.8695 | 0.7836 | - | | 18.5117 | 101000 | 0.0236 | 0.8246 | 0.7820 | - | | 18.6950 | 102000 | 0.0232 | 0.8543 | 0.7828 | - | | 18.8783 | 103000 | 0.0234 | 0.8840 | 0.7793 | - | | 19.0616 | 104000 | 0.0219 | 0.8804 | 0.7783 | - | | 19.2449 | 105000 | 0.0201 | 0.8885 | 0.7812 | - | | 19.4282 | 106000 | 0.0194 | 0.8901 | 0.7821 | - | | 19.6114 | 107000 | 0.0197 | 0.8850 | 0.7824 | - | | 19.7947 | 108000 | 0.0196 | 0.8835 | 0.7830 | - | | 19.9780 | 109000 | 0.0197 | 0.8803 | 0.7833 | - | * The bold row denotes the saved checkpoint.
### Framework Versions - Python: 3.12.3 - Sentence Transformers: 5.1.0 - Transformers: 4.55.0 - PyTorch: 2.8.0+cu128 - Accelerate: 1.10.0 - Datasets: 4.0.0 - Tokenizers: 0.21.4 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```