--- language: - en license: apache-2.0 tags: - sentence-transformers - cross-encoder - generated_from_trainer - dataset_size:578402 - loss:BinaryCrossEntropyLoss base_model: answerdotai/ModernBERT-base pipeline_tag: text-ranking library_name: sentence-transformers metrics: - map - mrr@10 - ndcg@10 model-index: - name: ModernBERT-base trained on GooAQ results: - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: gooaq dev type: gooaq-dev metrics: - type: map value: 0.7283 name: Map - type: mrr@10 value: 0.7272 name: Mrr@10 - type: ndcg@10 value: 0.7716 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoMSMARCO R100 type: NanoMSMARCO_R100 metrics: - type: map value: 0.4489 name: Map - type: mrr@10 value: 0.4376 name: Mrr@10 - type: ndcg@10 value: 0.5096 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNFCorpus R100 type: NanoNFCorpus_R100 metrics: - type: map value: 0.3159 name: Map - type: mrr@10 value: 0.4737 name: Mrr@10 - type: ndcg@10 value: 0.3176 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNQ R100 type: NanoNQ_R100 metrics: - type: map value: 0.4904 name: Map - type: mrr@10 value: 0.5075 name: Mrr@10 - type: ndcg@10 value: 0.5388 name: Ndcg@10 - task: type: cross-encoder-nano-beir name: Cross Encoder Nano BEIR dataset: name: NanoBEIR R100 mean type: NanoBEIR_R100_mean metrics: - type: map value: 0.4184 name: Map - type: mrr@10 value: 0.4729 name: Mrr@10 - type: ndcg@10 value: 0.4553 name: Ndcg@10 --- # ModernBERT-base trained on GooAQ This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search. ## Model Details ### Model Description - **Model Type:** Cross Encoder - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) - **Maximum Sequence Length:** 8192 tokens - **Number of Output Labels:** 1 label - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the 🤗 Hub model = CrossEncoder("Oysiyl/reranker-ModernBERT-base-gooaq-bce") # Get scores for pairs of texts pairs = [ ['what do you do with a degree in criminal justice?', "['Police Patrol Officer. Job Description. ... ', 'Criminal Investigators & Special Agents. Job Description. ... ', 'Private Detective or Investigator. ... ', 'First-Line Police Supervisor. ... ', 'Correctional Officer. ... ', 'Probation and Parole Officers. ... ', 'Postsecondary Criminal Justice & Law Enforcement Teachers.']"], ['what do you do with a degree in criminal justice?', "['Prison officer. ... ', 'Police officer. ... ', 'Detective. ... ', 'Criminologist. ... ', 'Probation officer. ... ', 'Forensic scientist. ... ', 'Crime Scene investigator. ... ', 'Court reporter.']"], ['what do you do with a degree in criminal justice?', "A high school diploma is required to work as a detective. In some cases a bachelor's degree in criminal justice or law enforcement may be needed. Experience in law enforcement is usually required, but the amount varies by employer."], ['what do you do with a degree in criminal justice?', "['Court Reporter. ... ', 'Criminal Intelligence Analyst. ... ', 'Forensic Accountant. ... ', 'Police Officer and Police Support Roles. ... ', 'Immigration, Customs and Border Roles. ... ', 'Prison Officer. ... ', 'Probation Officer. ... ', 'Scene of Crime Officer.']"], ['what do you do with a degree in criminal justice?', "Crime Scene Investigator Education and Training Although a high school diploma or equivalent is a minimum requirement for some positions, many police departments and law enforcement agencies prefer a minimum of an associate's (two-year) or a bachelor's (four-year) degree in criminal justice or a natural science."], ] scores = model.predict(pairs) print(scores.shape) # (5,) # Or rank different texts based on similarity to a single text ranks = model.rank( 'what do you do with a degree in criminal justice?', [ "['Police Patrol Officer. Job Description. ... ', 'Criminal Investigators & Special Agents. Job Description. ... ', 'Private Detective or Investigator. ... ', 'First-Line Police Supervisor. ... ', 'Correctional Officer. ... ', 'Probation and Parole Officers. ... ', 'Postsecondary Criminal Justice & Law Enforcement Teachers.']", "['Prison officer. ... ', 'Police officer. ... ', 'Detective. ... ', 'Criminologist. ... ', 'Probation officer. ... ', 'Forensic scientist. ... ', 'Crime Scene investigator. ... ', 'Court reporter.']", "A high school diploma is required to work as a detective. In some cases a bachelor's degree in criminal justice or law enforcement may be needed. Experience in law enforcement is usually required, but the amount varies by employer.", "['Court Reporter. ... ', 'Criminal Intelligence Analyst. ... ', 'Forensic Accountant. ... ', 'Police Officer and Police Support Roles. ... ', 'Immigration, Customs and Border Roles. ... ', 'Prison Officer. ... ', 'Probation Officer. ... ', 'Scene of Crime Officer.']", "Crime Scene Investigator Education and Training Although a high school diploma or equivalent is a minimum requirement for some positions, many police departments and law enforcement agencies prefer a minimum of an associate's (two-year) or a bachelor's (four-year) degree in criminal justice or a natural science.", ] ) # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] ``` ## Evaluation ### Metrics #### Cross Encoder Reranking * Dataset: `gooaq-dev` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": false } ``` | Metric | Value | |:------------|:---------------------| | map | 0.7283 (+0.1972) | | mrr@10 | 0.7272 (+0.2033) | | **ndcg@10** | **0.7716 (+0.1804)** | #### Cross Encoder Reranking * Datasets: `NanoMSMARCO_R100`, `NanoNFCorpus_R100` and `NanoNQ_R100` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": true } ``` | Metric | NanoMSMARCO_R100 | NanoNFCorpus_R100 | NanoNQ_R100 | |:------------|:---------------------|:---------------------|:---------------------| | map | 0.4489 (-0.0407) | 0.3159 (+0.0549) | 0.4904 (+0.0708) | | mrr@10 | 0.4376 (-0.0399) | 0.4737 (-0.0262) | 0.5075 (+0.0808) | | **ndcg@10** | **0.5096 (-0.0309)** | **0.3176 (-0.0074)** | **0.5388 (+0.0382)** | #### Cross Encoder Nano BEIR * Dataset: `NanoBEIR_R100_mean` * Evaluated with [CrossEncoderNanoBEIREvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "msmarco", "nfcorpus", "nq" ], "rerank_k": 100, "at_k": 10, "always_rerank_positives": true } ``` | Metric | Value | |:------------|:---------------------| | map | 0.4184 (+0.0284) | | mrr@10 | 0.4729 (+0.0049) | | **ndcg@10** | **0.4553 (-0.0000)** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 578,402 training samples * Columns: question, answer, and label * Approximate statistics based on the first 1000 samples: | | question | answer | label | |:--------|:-----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | question | answer | label | |:---------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | what do you do with a degree in criminal justice? | ['Police Patrol Officer. Job Description. ... ', 'Criminal Investigators & Special Agents. Job Description. ... ', 'Private Detective or Investigator. ... ', 'First-Line Police Supervisor. ... ', 'Correctional Officer. ... ', 'Probation and Parole Officers. ... ', 'Postsecondary Criminal Justice & Law Enforcement Teachers.'] | 1 | | what do you do with a degree in criminal justice? | ['Prison officer. ... ', 'Police officer. ... ', 'Detective. ... ', 'Criminologist. ... ', 'Probation officer. ... ', 'Forensic scientist. ... ', 'Crime Scene investigator. ... ', 'Court reporter.'] | 0 | | what do you do with a degree in criminal justice? | A high school diploma is required to work as a detective. In some cases a bachelor's degree in criminal justice or law enforcement may be needed. Experience in law enforcement is usually required, but the amount varies by employer. | 0 | * Loss: [BinaryCrossEntropyLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: ```json { "activation_fn": "torch.nn.modules.linear.Identity", "pos_weight": 5 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 1 - `warmup_ratio`: 0.1 - `seed`: 12 - `bf16`: True - `dataloader_num_workers`: 4 - `load_best_model_at_end`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 12 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 4 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `tp_size`: 0 - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | gooaq-dev_ndcg@10 | NanoMSMARCO_R100_ndcg@10 | NanoNFCorpus_R100_ndcg@10 | NanoNQ_R100_ndcg@10 | NanoBEIR_R100_mean_ndcg@10 | |:----------:|:---------:|:-------------:|:--------------------:|:------------------------:|:-------------------------:|:--------------------:|:--------------------------:| | -1 | -1 | - | 0.1374 (-0.4539) | 0.0392 (-0.5012) | 0.2824 (-0.0427) | 0.0371 (-0.4635) | 0.1196 (-0.3358) | | 0.0000 | 1 | 1.3234 | - | - | - | - | - | | 0.0277 | 1000 | 1.1925 | - | - | - | - | - | | 0.0553 | 2000 | 0.9574 | - | - | - | - | - | | 0.0830 | 3000 | 0.7563 | - | - | - | - | - | | 0.1106 | 4000 | 0.7247 | 0.7171 (+0.1259) | 0.5045 (-0.0359) | 0.3093 (-0.0158) | 0.5701 (+0.0694) | 0.4613 (+0.0059) | | 0.1383 | 5000 | 0.6744 | - | - | - | - | - | | 0.1660 | 6000 | 0.6876 | - | - | - | - | - | | 0.1936 | 7000 | 0.6446 | - | - | - | - | - | | 0.2213 | 8000 | 0.6519 | 0.7319 (+0.1407) | 0.5106 (-0.0298) | 0.3346 (+0.0095) | 0.6054 (+0.1048) | 0.4835 (+0.0282) | | 0.2490 | 9000 | 0.6339 | - | - | - | - | - | | 0.2766 | 10000 | 0.6168 | - | - | - | - | - | | 0.3043 | 11000 | 0.6005 | - | - | - | - | - | | 0.3319 | 12000 | 0.6363 | 0.7463 (+0.1550) | 0.5087 (-0.0317) | 0.3075 (-0.0176) | 0.5757 (+0.0751) | 0.4640 (+0.0086) | | 0.3596 | 13000 | 0.5882 | - | - | - | - | - | | 0.3873 | 14000 | 0.5888 | - | - | - | - | - | | 0.4149 | 15000 | 0.5824 | - | - | - | - | - | | 0.4426 | 16000 | 0.5882 | 0.7487 (+0.1574) | 0.5277 (-0.0127) | 0.3215 (-0.0036) | 0.5609 (+0.0603) | 0.4701 (+0.0147) | | 0.4702 | 17000 | 0.5661 | - | - | - | - | - | | 0.4979 | 18000 | 0.5758 | - | - | - | - | - | | 0.5256 | 19000 | 0.556 | - | - | - | - | - | | 0.5532 | 20000 | 0.5524 | 0.7556 (+0.1644) | 0.5419 (+0.0014) | 0.3191 (-0.0059) | 0.6132 (+0.1125) | 0.4914 (+0.0360) | | 0.5809 | 21000 | 0.5546 | - | - | - | - | - | | 0.6086 | 22000 | 0.563 | - | - | - | - | - | | 0.6362 | 23000 | 0.5369 | - | - | - | - | - | | 0.6639 | 24000 | 0.5492 | 0.7570 (+0.1658) | 0.4779 (-0.0626) | 0.2881 (-0.0369) | 0.5079 (+0.0073) | 0.4246 (-0.0307) | | 0.6915 | 25000 | 0.5443 | - | - | - | - | - | | 0.7192 | 26000 | 0.5522 | - | - | - | - | - | | 0.7469 | 27000 | 0.5323 | - | - | - | - | - | | 0.7745 | 28000 | 0.5167 | 0.7695 (+0.1782) | 0.4941 (-0.0463) | 0.3167 (-0.0083) | 0.5194 (+0.0188) | 0.4434 (-0.0119) | | 0.8022 | 29000 | 0.4998 | - | - | - | - | - | | 0.8299 | 30000 | 0.5326 | - | - | - | - | - | | 0.8575 | 31000 | 0.5262 | - | - | - | - | - | | 0.8852 | 32000 | 0.5136 | 0.7682 (+0.1769) | 0.5004 (-0.0400) | 0.3273 (+0.0023) | 0.5318 (+0.0311) | 0.4531 (-0.0022) | | 0.9128 | 33000 | 0.5145 | - | - | - | - | - | | 0.9405 | 34000 | 0.5038 | - | - | - | - | - | | 0.9682 | 35000 | 0.5236 | - | - | - | - | - | | **0.9958** | **36000** | **0.5347** | **0.7716 (+0.1804)** | **0.5096 (-0.0309)** | **0.3176 (-0.0074)** | **0.5388 (+0.0382)** | **0.4553 (-0.0000)** | | -1 | -1 | - | 0.7716 (+0.1804) | 0.5096 (-0.0309) | 0.3176 (-0.0074) | 0.5388 (+0.0382) | 0.4553 (-0.0000) | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.10 - Sentence Transformers: 4.1.0 - Transformers: 4.51.3 - PyTorch: 2.7.0+cu128 - Accelerate: 1.7.0 - Datasets: 3.6.0 - Tokenizers: 0.21.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```