File size: 7,834 Bytes
b9a3629 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 |
---
base_model: EuroBERT/EuroBERT-610m
language:
- en
license: apache-2.0
tags:
- text
- token-classification
- named-entity-recognition
- encoder-only
- euro-bert
- fine-tuned
- domain-specific
metrics:
- seqeval
model-index:
- name: EuroBERT-610m-group-mention-detector-uk-manifestos
results:
- task:
type: token-classification
name: Token classification
dataset:
type: custom
name: custom human-labeled sequence annotation dataset (see model card details)
metrics:
- type: seqeval
name: social group (seqeval)
value: 0.637704918032787
- type: seqeval
name: political group (seqeval)
value: 0.9045226130653266
- type: seqeval
name: political institution (seqeval)
value: 0.6445264452644527
- type: seqeval
name: organization, public institution, or collective actor (seqeval)
value: 0.5463258785942493
- type: seqeval
name: implicit social group reference (seqeval)
value: 0.6131805157593122
---
# EuroBERT-610m-group-mention-detector-uk-manifestos
<!-- Provide a quick summary of what the model is/does. -->
[EuroBERT/EuroBERT-610m](https://huggingface.co/EuroBERT/EuroBERT-610m) model finetuned for social group mention detectin in political texts
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
Token classification model for (social) group mention detection based on [Licht & Sczepanski (2025)](https://doi.org/10.31219/osf.io/ufb96)
This token classification has been finetuned on human sequence annotations of sentences of British parties' election manifestos for the following entity types:
- social group
- implicit social group reference
- political group
- political institution
- organization, public institution, or collective actor
Please refer to [Licht & Sczepanski (2025)](https://doi.org/10.31219/osf.io/ufb96) for details.
- **Developed by:** Hauke Licht
- **Model type:** eurobert
- **Language(s) (NLP):** ['en']
- **License:** apache-2.0
- **Finetuned from model:** EuroBERT/EuroBERT-610m
- **Funded by:** *Center for Comparative and International Studies* of the ETH Zurich and the University of Zurich and the *Deutsche Forschungsgemeinschaft* (DFG, German Research Foundation) under Germany's Excellence Strategy – EXC 2126/1 – 390838866
### Model Sources
<!-- Provide the basic links for the model. -->
- **Repository:** https://github.com/haukelicht/group_mention_detection/release/
- **Paper:** https://doi.org/10.31219/osf.io/ufb96
- **Demo:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
- Evaluation of the classifier in held-out data shows that it makes mistakes (see section *Results*).
- The model has been finetuned only on human-annotated labeled sentences sampled from British parties party manifestos. Applying the classifier in other domains can lead to higher error rates than those reported in section *Results* below.
- The data used to finetune the model come from human annotators. Human annotators can be biased and factors like gender and social background can impact their annotations judgments. This may lead to bias in the detection of specific social groups.
#### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
- Users who want to apply the model outside its training data domain (British parties' election programs) should evaluate its performance in the target data.
- Users who want to apply the model outside its training data domain (British parties' election programs) should contuninue to finetune this model on labeled data.
### How to Get Started with the Model
Use the code below to get started with the model.
```pyhton
from transformers import pipeline
model_id = "haukelicht/roberta-base-group-mention-detector-uk-manifestos"
classifier = pipeline(task="ner", model=model_id, aggregation_strategy="simple")
text = "Our party fights for the deprived and the vulnerable in our country."
annotations = classifier(text)
print(annotations)
# get annotations' character start and end indexes
locations = [(anno['start'], anno['end']) for anno in annotations]
locations
# index the source text using first annotation as an example
loc = locations[0]
text[slice(*loc)]
```
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
The train, dev, and test splits used for model finetuning and evaluation are available on Github: https://github.com/haukelicht/group_mention_detection/release/splits
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Training Hyperparameters
- epochs: 5
- learning rate: 1e-05
- batch size: 32
- weight decay: 0.01
- warmup ratio: 0.1
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
The train, dev, and test splits used for model finetuning and evaluation are available on Github: https://github.com/haukelicht/group_mention_detection/release/splits
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
- seq-eval F1: strict seqeuence labeling evaluation metric per CoNLL-2000 shared task based on https://github.com/chakki-works/seqeval
- "soft" seq-eval F1: a more lenient seqeuence labeling evaluation metric that reports span level average performance suzmmarized across examples per https://github.com/haukelicht/soft-seqeval
- sentence-level F1: binary measure of detection performance considering a sentence a positive example/prediction if it contains at least one enttiy to of the given type
### Results
| type | seq-eval F1 | soft seq-eval F1 | sentence level F1 |
|-------------------------------------------------------|---------------|---------------------|----------------------|
| social group | 0.638 | 0.739 | 0.928 |
| political group | 0.905 | 0.920 | 0.990 |
| political institution | 0.645 | 0.698 | 0.954 |
| organization, public institution, or collective actor | 0.546 | 0.552 | 0.928 |
| implicit social group reference | 0.613 | 0.537 | 0.943 |
## Citation
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
Licht, H., & Sczepanski, R. (2025). Detecting Group Mentions in Political Rhetoric: A Supervised Learning Approach. forthcoming in *British Journal of Political Science*. Preprint available at [OSF](https://doi.org/10.31219/osf.io/ufb96)
## More Information
https://github.com/haukelicht/group_mention_detection/release
## Model Card Contact
[email protected]
|