|
|
--- |
|
|
library_name: transformers |
|
|
language: |
|
|
- fr |
|
|
- de |
|
|
- en |
|
|
- it |
|
|
- lb |
|
|
license: agpl-3.0 |
|
|
tags: |
|
|
- language-identification |
|
|
- multilingual |
|
|
- historical |
|
|
- impresso |
|
|
--- |
|
|
|
|
|
# Model Card for impresso-project/language-identifier |
|
|
|
|
|
## Overview |
|
|
|
|
|
`impresso-project/language-identifier` is a multilingual language identification model fine-tuned for use on historical newspaper content. It supports **German (de), French (fr), Italian (it), English (en), and Luxembourgish (lb)** — the core languages of the [Impresso Project](https://impresso-project.ch), which focuses on analyzing historical media across national and linguistic borders. |
|
|
|
|
|
This model has been adapted for short, OCR-noisy and fragmentary inputs typical of historical digitized texts. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model type:** Language identification |
|
|
- **Interface:** Hugging Face `transformers` pipeline |
|
|
- **Languages supported:** fr, de, en, it, lb |
|
|
- **License:** AGPL-3.0 |
|
|
- **Developed by:** UZH, Switzerland |
|
|
- **Training data:** Historical newspapers from the impresso corpus and related sources |
|
|
|
|
|
## How to Use |
|
|
|
|
|
```python |
|
|
from transformers import pipeline |
|
|
|
|
|
MODEL_NAME = "impresso-project/language-identifier" |
|
|
|
|
|
lang_pipeline = pipeline( |
|
|
"langident", |
|
|
model=MODEL_NAME, |
|
|
trust_remote_code=True, |
|
|
device="cpu", |
|
|
) |
|
|
|
|
|
text = """En l'an 1348, au plus fort des ravages de la peste noire à travers |
|
|
l'Europe, le Royaume de France se trouvait à la fois au bord du désespoir et |
|
|
face à une opportunité.""" |
|
|
|
|
|
langs = lang_pipeline(text) |
|
|
print(langs) |
|
|
``` |
|
|
|
|
|
## Output Format |
|
|
|
|
|
The output is a single dictionary with the predicted language and confidence score: |
|
|
|
|
|
```python |
|
|
{ |
|
|
"language": "fr", |
|
|
"score": 1.0 |
|
|
} |
|
|
``` |
|
|
|
|
|
|
|
|
## Use Cases |
|
|
|
|
|
- Preprocessing for OCR and NLP tasks on historical corpora |
|
|
- Document and segment-level language tagging |
|
|
- Filtering and sorting multilingual newspaper archives |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Works best on **sentence- or paragraph-length** texts |
|
|
- May struggle with code-switching or OCR-degraded text that mixes languages |
|
|
- Primarily optimized for **Impresso-like sources** (19th–20th century newspapers) |
|
|
|
|
|
## Installation |
|
|
|
|
|
```bash |
|
|
pip install transformers floret |
|
|
``` |
|
|
|
|
|
## Contact |
|
|
|
|
|
- Website: [https://impresso-project.ch](https://impresso-project.ch) |
|
|
|
|
|
<p align="center"> |
|
|
<img src="https://github.com/impresso/impresso.github.io/blob/master/assets/images/3x1--Yellow-Impresso-Black-on-White--transparent.png?raw=true" width="300" alt="Impresso Logo"/> |
|
|
</p> |
|
|
|
|
|
|
