README.md

---
language:
- nl
license: mit
tags:
- digital-humanities
- token-classification
base_model:
- FacebookAI/xlm-roberta-base
---

# Model Card for NER-base

[Globalise](https://globalise.huygens.knaw.nl/) NER token-classification model, development version.

## Model Details

### Model Description

This is the first version of a NER model developed for the Globalise project.

- **Developed by:** Sophie Arnoult
- **Shared by:** Globalise Team
- **Funded by:** NWO
- **Model type:** token classification

## Uses

Named-Entity tagging of historical (17th-18th century), VOC-related Dutch documents.


## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->
The texts the model was fine-tuned on are heavily biased, representing colonial standpoints. While care has been taken in designing the labelset and annotating the data, biases may remain when applying the model on similar data; the model has not been tested on other data.

This is a development version. The training and development data consist of [VOC missives](https://research.vu.nl/en/datasets/voc-gm-ner-corpus) data enriched with new annotations. Most entity types used in Globalise are not present in the VOC missives data, while the new annotations are limited in number. Performance on these may therefore not be representative.


## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The training and development data consist of 
- GM NER corpus ([datasplit-all-standard](https://data.yoda.vu.nl:9443/vault-fgw-llc-vocmissives/voc_gm_ner%5B1670857835%5D/original/datasplit_all_standard/), train/dev data), where labels are mapped to their Globalise equivalents
- Globalise annotated data (first set of annotations, to be extended and published at a later date)

The data are pretokenized with [Spacy](https://spacy.io/models/nl#nl_core_news_lg). Sequences are split at 240 word tokens.

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

#### Training Hyperparameters

- **Training regime:** fp32
- **Optimizer:** Adam, learning rate 3e-5
- **max-sequence-length:** 512
- **batch size:** 32
- **max-epochs**: 20


## Evaluation

Model selected based on validation weighted multiclass F1 score, using a single seed. 
<!-- This section describes the evaluation protocols and provides the results. -->


### Results
label |  precision |   recall | f1-score  | support |
| --- | --- | --- |--- | --- |
   CMTY_NAME|       0.72|      0.80|      0.76|       109
   CMTY_QUAL|       1.00|      0.67|      0.80|         9
  CMTY_QUANT|       0.76|      0.85|      0.80|        66
DATE|       0.48|      0.53|      0.51|        43
DOC|       0.61|      0.55|      0.58|        20
ETH_REL|       0.78|      0.81|      0.79|        31
LOC_ADJ|       0.91|      0.96|      0.94|       464
LOC_NAME|       0.91|      0.94|      0.92|      1324
ORG|       0.92|      0.87|      0.89|       265
PER_ATTR|       0.69|      0.82|      0.75|        44
PER_NAME|       0.80|      0.87|      0.83|       613
PRF|       0.70|      0.76|      0.73|        97
SHIP|       0.89|      0.86|      0.87|       519
SHIP_TYPE|       0.79|      0.82|      0.81|        33
STATUS|       0.96|      0.96|      0.96|        27
micro avg |       0.86 |      0.89 |      0.88 |      3664 
macro avg |       0.79 |      0.80 |      0.80 |      3664 
weighted avg |       0.86 |      0.89 |      0.88 |      3664 


## Technical Specifications


### Compute Infrastructure

SURF Snellius

#### Hardware

A100