---
library_name: transformers
tags: []
---

# Tetun BERT model

A fine-tune of [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) trained on Tetun data with a masked language modelling objective.

Tetun data used: [MADLAD](https://huggingface.co/datasets/allenai/MADLAD-400) tet clean split (~40k documents).

Trained for 10 epochs with hyper params from the [MasakhaNER paper](https://aclanthology.org/2021.tacl-1.66.pdf) (lr 5e-5 etc).