Prosho/sentinel-src-25

📊 Estimating Machine Translation Difficulty

       

This repository contains the SENTINEL_SRC metric model used for Difficulty Sampling at the WMT25 General Machine Translation Shared Task, and analyzed in our paper Estimating Machine Translation Difficulty.

Usage

To run this model, install the following git repository:

pip install git+https://github.com/prosho-97/guardians-mt-eval

After that, you can use this model within Python in the following way:

from sentinel_metric import download_model, load_from_checkpoint

model_path = download_model("Prosho/sentinel-src-25")
model = load_from_checkpoint(model_path)

data = [
    {"src": "Please sign the form."},
    {"src": "He spilled the beans, then backpedaled—talk about mixed signals!"}
]

output = model.predict(data, batch_size=8, gpus=1)

Output:

# Segment scores
>>> output.scores
[0.5604351758956909, -0.08413456380367279]

# System score
>>> output.system_score
0.23815030604600906

Where the higher the output score, the easier it is to translate the input source text.

Cite this work

This work has been accepted at EMNLP 2025. If you use any part, please consider citing our paper as follows:

@misc{proietti2025estimatingmachinetranslationdifficulty,
      title={Estimating Machine Translation Difficulty}, 
      author={Lorenzo Proietti and Stefano Perrella and Vilém Zouhar and Roberto Navigli and Tom Kocmi},
      year={2025},
      eprint={2508.10175},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2508.10175}, 
}