Llama-3.2-1B-subjectivity-english

This model is a fine-tuned version of meta-llama/Llama-3.2-1B for Subjectivity Detection in News Articles. It was presented by AI Wizards in the paper AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles.

This model achieves the following results on the evaluation set:

Loss: 0.6766
Macro F1: 0.7718
Macro P: 0.7731
Macro R: 0.7715
Subj F1: 0.7862
Subj P: 0.7689
Subj R: 0.8042
Accuracy: 0.7727

Model description

This model, Llama-3.2-1B-subjectivity-english, is part of AI Wizards' participation in the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles. Its core purpose is to classify sentences as subjective (opinion-laden) or objective.

The model is based on the Llama 3.2-1B architecture and is fine-tuned using Parameter-Efficient Fine-Tuning (PEFT), specifically LoRA. A key innovation of this work is the enhancement of transformer-based classifiers by integrating sentiment scores, derived from an auxiliary model, with the sentence representations. This sentiment-augmented architecture aims to improve upon standard fine-tuning approaches, particularly boosting performance on the subjective F1 score. Additionally, decision threshold calibration was employed to address class imbalance prevalent across the datasets.

Intended uses & limitations

Intended Uses: This model is intended for classifying sentences in news articles as either subjective or objective. Potential applications include:

Combating misinformation by identifying opinionated content.
Improving fact-checking pipelines.
Supporting journalists in analyzing and curating news articles.

While this specific model is fine-tuned for English, the underlying framework was also explored for multilingual and zero-shot settings as part of the broader project.

Limitations:

In the original research, BERT-like models (e.g., mDeBERTaV3-base, ModernBERT-base) consistently outperformed LLM baselines like Llama-3.2-1B in most scenarios. This Llama-3.2-1B model served as a zero-shot baseline with a classifier added on top of its embeddings.
The model's performance may vary on highly nuanced or ambiguous texts, or on text domains significantly different from news articles.
Generalization to unseen languages in zero-shot settings, while explored in the paper, should be re-evaluated for specific applications.

Training and evaluation data

This model was fine-tuned on the English split of the datasets provided for the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles.

The overall project utilized training and development datasets for Arabic, German, English, Italian, and Bulgarian. The final evaluation included additional unseen languages such as Greek, Romanian, Polish, and Ukrainian to assess generalization capabilities. Decision threshold calibration was applied during training to mitigate class imbalance issues across languages.

How to use

You can use this model for text classification with the pipeline function from the transformers library.

from transformers import pipeline

# Initialize the text classification pipeline
classifier = pipeline(
    "text-classification",
    model="MatteoFasulo/Llama-3.2-1B-subjectivity-english",
    tokenizer="meta-llama/Llama-3.2-1B", # Use the tokenizer of the base model
    device=0 # Specify device for GPU usage, or -1 for CPU
)

# Example 1: Subjective sentence
text_subjective = "This film is an absolute masterpiece, truly breathtaking!"
result_subjective = classifier(text_subjective)
print(f"'{text_subjective}' -> {result_subjective}")
# Expected output: [{'label': 'LABEL_1', 'score': X.XXX}] (Assuming LABEL_1 is subjective)

# Example 2: Objective sentence
text_objective = "The company's quarterly report showed a 5% increase in net profit."
result_objective = classifier(text_objective)
print(f"'{text_objective}' -> {result_objective}")
# Expected output: [{'label': 'LABEL_0', 'score': X.XXX}] (Assuming LABEL_0 is objective)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Macro F1	Macro P	Macro R	Subj F1	Subj P	Subj R	Accuracy
No log	1.0	52	0.6133	0.6564	0.6716	0.6595	0.7092	0.6451	0.7875	0.6645
No log	2.0	104	0.5806	0.7458	0.7469	0.7454	0.7617	0.7450	0.7792	0.7468
No log	3.0	156	0.5663	0.7464	0.7523	0.7462	0.7717	0.7313	0.8167	0.7489
No log	4.0	208	0.5916	0.7568	0.7650	0.7566	0.7836	0.7363	0.8375	0.7597
No log	5.0	260	0.6430	0.7693	0.7716	0.7689	0.7863	0.7617	0.8125	0.7706
No log	6.0	312	0.6766	0.7718	0.7731	0.7715	0.7862	0.7689	0.8042	0.7727

Framework versions

PEFT 0.14.0
Transformers 4.49.0
Pytorch 2.5.1+cu121
Datasets 3.3.1
Tokenizers 0.21.0

Code

The official code and materials for AI Wizards' submission to CLEF 2025 CheckThat! Lab are available on GitHub: https://github.com/MatteoFasulo/clef2025-checkthat

Project Page

You can explore a collection of models and results related to this work on the Hugging Face Hub: AI Wizards @ CLEF 2025 - CheckThat! Lab - Task 1 Subjectivity

Citation

If you find our work helpful or inspiring, please feel free to cite it:

@misc{fasulo2025aiwizardscheckthat2025,
      title={AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles}, 
      author={Matteo Fasulo and Luca Babboni and Luca Tedeschini},
      year={2025},
      eprint={2507.11764},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11764}, 
}

License

This research work, including the paper and the code, is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Please note that the underlying base model, Llama 3.2-1B, has its own specific usage license, which should also be consulted.

MatteoFasulo
/

Llama-3.2-1B-subjectivity-english