license: apache-2.0
base_model: bert-base-multilingual-cased
tags:
- generated_from_keras_callback
model-index:
- name: >-
bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract
results: []
bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract
This model is a fine-tuned version of bert-base-multilingual-cased on a labeled dataset provided by CWTS: [CWTS Labeled Data].
This is NOT the full model being used to tag OpenAlex works with a topic. For that, check out the following github repo: OpenAlex Topic Classification
Model description
The input data is expected to be in the following format:
"<TITLE> {insert-processed-title-here}\n<ABSTRACT> {insert-processed-abstract-here}"
Since this was train
Intended uses & limitations
The model is intended to be used as part of a larger model that also incorporates journal information and citation features. However, this model is good if you want to use it for quickly generating a topic based only on a title/abstract.
Since this model was fine-tuned on a BERT model, all of the biases seen in that model will most likely show up in this model as well.
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': {'module': 'transformers.optimization_tf', 'class_name': 'WarmUp', 'config': {'initial_learning_rate': 6e-05, 'decay_schedule_fn': {'module': 'keras.optimizers.schedules', 'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 6e-05, 'decay_steps': 335420, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, 'registered_name': None}, 'warmup_steps': 500, 'power': 1.0, 'name': None}, 'registered_name': 'WarmUp'}, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
- training_precision: float32
Training results
| Train Loss | Validation Loss | Train Accuracy | Epoch |
|---|---|---|---|
| 4.8075 | 3.6686 | 0.3839 | 0 |
| 3.4867 | 3.3360 | 0.4337 | 1 |
| 3.1865 | 3.2005 | 0.4556 | 2 |
| 2.9969 | 3.1379 | 0.4675 | 3 |
| 2.8489 | 3.0900 | 0.4746 | 4 |
| 2.7212 | 3.0744 | 0.4799 | 5 |
| 2.6035 | 3.0660 | 0.4831 | 6 |
| 2.4942 | 3.0737 | 0.4846 | 7 |
Framework versions
- Transformers 4.35.2
- TensorFlow 2.13.0
- Datasets 2.15.0
- Tokenizers 0.15.0