mdeberta-v3-base-subjectivity-bulgarian

This model is a fine-tuned version of microsoft/mdeberta-v3-base for Subjectivity Detection in News Articles. It was presented by AI Wizards in the paper AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles as part of the CLEF 2025 CheckThat! Lab Task 1.

The official code and materials for this project can be found on the GitHub repository.

It achieves the following results on the evaluation set:

• Loss: 0.5111
• Macro F1: 0.7869
• Macro P: 0.7949
• Macro R: 0.7839
• Subj F1: 0.7510
• Subj P: 0.8033
• Subj R: 0.7050
• Accuracy: 0.7930

Model description

This model identifies whether a sentence is subjective (e.g., opinion-laden) or objective. This task is a key component in combating misinformation, improving fact-checking pipelines, and supporting journalists. This specific checkpoint is fine-tuned for the Bulgarian language.

The primary strategy behind this model involves enhancing transformer-based classifiers (specifically mDeBERTaV3-base) by integrating sentiment scores, derived from an auxiliary model, with sentence representations. This aims to improve upon standard fine-tuning, particularly boosting subjective F1 score. To address class imbalance prevalent across languages, decision threshold calibration optimized on the development set was employed. The research achieved high rankings in the CLEF 2025 CheckThat! Lab Task 1, notably ranking 1st for Greek (zero-shot, Macro F1 = 0.51) and securing 1st–4th place in most monolingual settings.

Intended uses & limitations

Intended Uses: This model is intended for research and practical applications focused on subjectivity detection in news articles, particularly for distinguishing subjective (opinion-laden) from objective content. It can be particularly useful in:

• Combating misinformation by identifying opinionated content.
• Improving fact-checking pipelines.
• Supporting journalists in content analysis and understanding bias.

Limitations:

• While the overarching research explored multilingual and zero-shot settings, this specific model checkpoint is fine-tuned for Bulgarian. Its performance might vary when applied to other languages or domains not represented in the training data without further fine-tuning.
• The paper notes that an initial submission quirk led to skewed class distribution and under-calibrated thresholds; the reported results reflect the corrected evaluation. Users should be aware of potential nuances when applying the model to data with significantly different class distributions.

Training and evaluation data

This model was trained and evaluated as part of the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles. Training and development datasets were provided for Arabic, German, English, Italian, and Bulgarian. The final evaluation included additional unseen languages such as Greek, Romanian, Polish, and Ukrainian to assess generalization capabilities.

To address class imbalance, a common issue across these languages, a decision threshold calibration optimized on the development set was employed. More details on the datasets and experimental setup can be found in the paper and the GitHub repository.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Macro F1	Macro P	Macro R	Subj F1	Subj P	Subj R	Accuracy
No log	1.0	46	0.6658	0.6925	0.7258	0.7124	0.7160	0.6080	0.8705	0.6943
No log	2.0	92	0.5538	0.7673	0.7748	0.7646	0.7280	0.7787	0.6835	0.7739
No log	3.0	138	0.5300	0.7353	0.7690	0.7327	0.6695	0.8144	0.5683	0.7516
No log	4.0	184	0.5090	0.7680	0.7833	0.7645	0.7222	0.8053	0.6547	0.7771
No log	5.0	230	0.5065	0.7728	0.7835	0.7696	0.7315	0.7966	0.6763	0.7803
No log	6.0	276	0.5111	0.7869	0.7949	0.7839	0.7510	0.8033	0.7050	0.7930

Framework versions

• Transformers 4.50.0
• Pytorch 2.5.1+cu121
• Datasets 3.3.1
• Tokenizers 0.21.0

How to use

You can use this model directly with the Hugging Face transformers library for text classification:

from transformers import pipeline

# Load the text classification pipeline
classifier = pipeline(
    "text-classification",
    model="MatteoFasulo/mdeberta-v3-base-subjectivity-bulgarian",
    tokenizer="microsoft/mdeberta-v3-base",
)

# Example usage:
result1 = classifier("По принцип никой не иска войни, но за нещастие те се случват.")
print(f"Classification: {result1}")
# Expected output: [{'label': 'SUBJ', 'score': ...}]

result2 = classifier("В един момент започнал сам да търси изход за своето спасение и здраве")
print(f"Classification: {result2}")
# Expected output: [{'label': 'OBJ', 'score': ...}]

Citation

If you find our work helpful or inspiring, please feel free to cite it:

@misc{fasulo2025aiwizardscheckthat2025,
      title={AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles}, 
      author={Matteo Fasulo and Luca Babboni and Luca Tedeschini},
      year={2025},
      eprint={2507.11764},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11764}, 
}