distilbert-base-future

Table of Contents

• Model description
• Intended uses & limitations
• Training and evaluation data
• Training procedure

This model is a fine-tuned version of distilbert-base-uncased on the future-statements dataset. It achieves the following results on the evaluation set:

• Train Loss: 0.1142
• Train Sparse Categorical Accuracy: 0.9613
• Validation Loss: 0.1272
• Validation Sparse Categorical Accuracy: 0.9625
• Epoch: 1

Model description

• The model was created by graduate students D. Baradari, F. Bartels, A. Dewald, J. Peters as part of a data science module of the University of Leipzig.
• Model was created on 11/08/22.
• This is version 1.0
• The model is a text classification model which is a fine-tuned version of distilbert-base-uncased
• Questions and comments can be send via the community tab

Intended uses & limitations

• The primary intended use is the classification of input into a future or non-future sentence/statement.
• The model is primarily intended to be used by researchers to filter or label a large number of sentences according to the grammatical tense of the input.

Training and evaluation data

• Distilbert-base-future model was trained and evaluated on the future-statements dataset.
• future-statements is a dataset collected manually and automatically by graduate students D. Baradari, F. Bartels, A. Dewald, J. Peters of the University of Leipzig.
• We collected 2500 statements, 50% of which relate to future events and 50% of which relate to non-future events.
• The sole purpose of the dataset was the fine-tuning process of this model.
• Additional information on the dataset can be found on Huggingface: future-statements dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• optimizer: {'name': 'Adam', 'learning_rate': 5e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
• training_precision: float32

Training results

Train Loss	Train Sparse Categorical Accuracy	Validation Loss	Validation Sparse Categorical Accuracy	Epoch
0.3816	0.8594	0.1547	0.9475	0
0.1142	0.9613	0.1272	0.9625	1

Framework versions

• Transformers 4.18.0
• TensorFlow 2.8.0
• Datasets 2.4.0
• Tokenizers 0.12.1