Upload folder using huggingface_hub

.ipynb_checkpoints/README-checkpoint.md

@@ -0,0 +1,67 @@
+---
+task: token-classification
+tags:
+- biomedical
+- bionlp
+license: mit
+base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
+---
+
+# bioner_gnormplus
+
+This is a named entity recognition model fine-tuned from the [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) model. It predicts spans with 3 possible labels. The labels are **DomainMotif, FamilyName and Gene**.
+
+The code used for training this model can be found at https://github.com/Glasgow-AI4BioMed/bioner along with links to other biomedical NER models trained on well-known biomedical corpora. The source dataset information is below.
+
+## Example Usage
+
+The code below will load up the model and apply it to the provided text. It uses a simple aggregation strategy to post-process the individual tokens into larger multi-token entities where needed.
+
+```python
+from transformers import pipeline
+
+# Load the model as part of an NER pipeline
+ner_pipeline = pipeline("token-classification", 
+                        model="Glasgow-AI4BioMed/bioner_gnormplus",
+                        aggregation_strategy="max")
+
+# Apply it to some text
+ner_pipeline("EGFR T790M mutations have been known to affect treatment outcomes for NSCLC patients receiving erlotinib.")
+
+# Output:
+# [ {"entity_group": "FamilyName", "score": 0.44405, "word": "egfr", "start": 0, "end": 4},
+```
+
+## Dataset Info
+
+**Source:** The GNormPlus dataset was downloaded from: https://www.ncbi.nlm.nih.gov/research/bionlp/Tools/gnormplus/
+
+The dataset should be cited with: Wei, Chih-Hsuan, Hung-Yu Kao, and Zhiyong Lu. "GNormPlus: an integrative approach for tagging genes, gene families, and protein domains." BioMed research international 2015.1 (2015): 918710. DOI: [10.1155/2015/918710](https://doi.org/10.1155/2015/918710)
+
+**Preprocessing:** The training set was split 75/25 to create a training and validation set. No changes were made to the annotations. The preprocessing script for this dataset is [prepare_bc5cdr.py](https://github.com/Glasgow-AI4BioMed/bioner/blob/main/prepare_bc5cdr.py).
+
+## Performance
+
+The span-level performance on the test split for the different labels are shown in the tables below. The full performance results are available in the model repo in Markdown format for viewing and JSON format for easier loading. These include the performance at token level (with individual B- and I- labels as the token classifier uses IOB2 token labelling).
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| DomainMotif | 0.602 | 0.670 | 0.634 | 361 |
+| FamilyName | 0.497 | 0.569 | 0.530 | 1250 |
+| Gene | 0.856 | 0.923 | 0.888 | 3225 |
+| macro avg | 0.651 | 0.721 | 0.684 | 4836 |
+| weighted avg | 0.744 | 0.812 | 0.777 | 4836 |
+
+
+## Hyperparameters
+
+Hyperparameter tuning was done with [optuna](https://optuna.org/) and the [hyperparameter_search](https://huggingface.co/docs/transformers/en/hpo_train) functionality. 100 trials were run. Early stopping was applied during training. The best performing model was selected using the macro F1 performance on the validation set. The selected hyperparameters are in the table below.
+
+| Hyperparameter | Value |
+|----------------|-------|
+| epochs | 13.0 |
+| learning_rate | 4.312024782506724e-05 |
+| per_device_train_batch_size | 16 |
+| weight_decay | 0.004637010897989902 |
+| warmup_ratio | 0.00046632724074153857 |
+

README.md

@@ -0,0 +1,67 @@
+---
+task: token-classification
+tags:
+- biomedical
+- bionlp
+license: mit
+base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
+---
+
+# bioner_gnormplus
+
+This is a named entity recognition model fine-tuned from the [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) model. It predicts spans with 3 possible labels. The labels are **DomainMotif, FamilyName and Gene**.
+
+The code used for training this model can be found at https://github.com/Glasgow-AI4BioMed/bioner along with links to other biomedical NER models trained on well-known biomedical corpora. The source dataset information is below.
+
+## Example Usage
+
+The code below will load up the model and apply it to the provided text. It uses a simple aggregation strategy to post-process the individual tokens into larger multi-token entities where needed.
+
+```python
+from transformers import pipeline
+
+# Load the model as part of an NER pipeline
+ner_pipeline = pipeline("token-classification", 
+                        model="Glasgow-AI4BioMed/bioner_gnormplus",
+                        aggregation_strategy="max")
+
+# Apply it to some text
+ner_pipeline("EGFR T790M mutations have been known to affect treatment outcomes for NSCLC patients receiving erlotinib.")
+
+# Output:
+# [ {"entity_group": "FamilyName", "score": 0.44405, "word": "egfr", "start": 0, "end": 4},
+```
+
+## Dataset Info
+
+**Source:** The GNormPlus dataset was downloaded from: https://www.ncbi.nlm.nih.gov/research/bionlp/Tools/gnormplus/
+
+The dataset should be cited with: Wei, Chih-Hsuan, Hung-Yu Kao, and Zhiyong Lu. "GNormPlus: an integrative approach for tagging genes, gene families, and protein domains." BioMed research international 2015.1 (2015): 918710. DOI: [10.1155/2015/918710](https://doi.org/10.1155/2015/918710)
+
+**Preprocessing:** The training set was split 75/25 to create a training and validation set. No changes were made to the annotations. The preprocessing script for this dataset is [prepare_bc5cdr.py](https://github.com/Glasgow-AI4BioMed/bioner/blob/main/prepare_bc5cdr.py).
+
+## Performance
+
+The span-level performance on the test split for the different labels are shown in the tables below. The full performance results are available in the model repo in Markdown format for viewing and JSON format for easier loading. These include the performance at token level (with individual B- and I- labels as the token classifier uses IOB2 token labelling).
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| DomainMotif | 0.602 | 0.670 | 0.634 | 361 |
+| FamilyName | 0.497 | 0.569 | 0.530 | 1250 |
+| Gene | 0.856 | 0.923 | 0.888 | 3225 |
+| macro avg | 0.651 | 0.721 | 0.684 | 4836 |
+| weighted avg | 0.744 | 0.812 | 0.777 | 4836 |
+
+
+## Hyperparameters
+
+Hyperparameter tuning was done with [optuna](https://optuna.org/) and the [hyperparameter_search](https://huggingface.co/docs/transformers/en/hpo_train) functionality. 100 trials were run. Early stopping was applied during training. The best performing model was selected using the macro F1 performance on the validation set. The selected hyperparameters are in the table below.
+
+| Hyperparameter | Value |
+|----------------|-------|
+| epochs | 13.0 |
+| learning_rate | 4.312024782506724e-05 |
+| per_device_train_batch_size | 16 |
+| weight_decay | 0.004637010897989902 |
+| warmup_ratio | 0.00046632724074153857 |
+

best_hyperparameters.json

@@ -0,0 +1,7 @@
+{
+  "epochs": 13.0,
+  "learning_rate": 4.312024782506724e-05,
+  "per_device_train_batch_size": 16,
+  "weight_decay": 0.004637010897989902,
+  "warmup_ratio": 0.00046632724074153857
+}

config.json

@@ -0,0 +1,34 @@
+{
+  "_name_or_path": "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext",
+  "architectures": [
+    "BertForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "O",
+    "1": "B-DomainMotif",
+    "2": "I-DomainMotif",
+    "3": "B-FamilyName",
+    "4": "I-FamilyName",
+    "5": "B-Gene",
+    "6": "I-Gene"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc802b8a0ad2e9ab907de5f825b001e5d91b8dffb6d9345c4c4020952b4171a8
+size 435611468

performance_report.json

@@ -0,0 +1,275 @@
+{
+  "train": {
+    "token_level": {
+      "O": {
+        "precision": 0.9992820624388121,
+        "recall": 0.9996517802733524,
+        "f1-score": 0.9994668871650365,
+        "support": 45948.0
+      },
+      "B-DomainMotif": {
+        "precision": 0.9736842105263158,
+        "recall": 0.9487179487179487,
+        "f1-score": 0.961038961038961,
+        "support": 195.0
+      },
+      "I-DomainMotif": {
+        "precision": 0.9962406015037594,
+        "recall": 0.9888059701492538,
+        "f1-score": 0.9925093632958801,
+        "support": 536.0
+      },
+      "B-FamilyName": {
+        "precision": 0.9926289926289926,
+        "recall": 0.9901960784313726,
+        "f1-score": 0.9914110429447853,
+        "support": 816.0
+      },
+      "I-FamilyName": {
+        "precision": 0.9940054495912807,
+        "recall": 0.9972662657189721,
+        "f1-score": 0.9956331877729258,
+        "support": 1829.0
+      },
+      "B-Gene": {
+        "precision": 0.9959641255605381,
+        "recall": 0.991960696739616,
+        "f1-score": 0.993958379950772,
+        "support": 2239.0
+      },
+      "I-Gene": {
+        "precision": 0.9986388384754991,
+        "recall": 0.9979596463386987,
+        "f1-score": 0.9982991268851343,
+        "support": 4411.0
+      },
+      "accuracy": 0.9987136884982313,
+      "macro avg": {
+        "precision": 0.9929206115321711,
+        "recall": 0.987794055195602,
+        "f1-score": 0.9903309927219279,
+        "support": 55974.0
+      },
+      "weighted avg": {
+        "precision": 0.9987109444979878,
+        "recall": 0.9987136884982313,
+        "f1-score": 0.9987113109741671,
+        "support": 55974.0
+      }
+    },
+    "span_level": {
+      "DomainMotif": {
+        "precision": 0.9067357512953368,
+        "recall": 0.8883248730964467,
+        "f1-score": 0.8974358974358975,
+        "support": 197
+      },
+      "FamilyName": {
+        "precision": 0.9755799755799756,
+        "recall": 0.9803680981595092,
+        "f1-score": 0.97796817625459,
+        "support": 815
+      },
+      "Gene": {
+        "precision": 0.9882671480144405,
+        "recall": 0.9763709317877842,
+        "f1-score": 0.9822830230993497,
+        "support": 2243
+      },
+      "macro avg": {
+        "precision": 0.9568609582965842,
+        "recall": 0.9483546343479133,
+        "f1-score": 0.9525623655966124,
+        "support": 3255
+      },
+      "weighted avg": {
+        "precision": 0.9801560172347932,
+        "recall": 0.9720430107526882,
+        "f1-score": 0.9760675134421517,
+        "support": 3255
+      }
+    }
+  },
+  "val": {
+    "token_level": {
+      "O": {
+        "precision": 0.982154530003922,
+        "recall": 0.982796964939822,
+        "f1-score": 0.9824756424507944,
+        "support": 15288.0
+      },
+      "B-DomainMotif": {
+        "precision": 0.7536231884057971,
+        "recall": 0.6419753086419753,
+        "f1-score": 0.6933333333333334,
+        "support": 81.0
+      },
+      "I-DomainMotif": {
+        "precision": 0.8457142857142858,
+        "recall": 0.714975845410628,
+        "f1-score": 0.774869109947644,
+        "support": 207.0
+      },
+      "B-FamilyName": {
+        "precision": 0.6816479400749064,
+        "recall": 0.6086956521739131,
+        "f1-score": 0.6431095406360424,
+        "support": 299.0
+      },
+      "I-FamilyName": {
+        "precision": 0.7571428571428571,
+        "recall": 0.6656200941915228,
+        "f1-score": 0.70843776106934,
+        "support": 637.0
+      },
+      "B-Gene": {
+        "precision": 0.8721071863580999,
+        "recall": 0.9384010484927916,
+        "f1-score": 0.9040404040404041,
+        "support": 763.0
+      },
+      "I-Gene": {
+        "precision": 0.8863487916394513,
+        "recall": 0.9384508990318119,
+        "f1-score": 0.9116560295599597,
+        "support": 1446.0
+      },
+      "accuracy": 0.9563591688478179,
+      "macro avg": {
+        "precision": 0.8255341113341885,
+        "recall": 0.784416544697495,
+        "f1-score": 0.8025602601482168,
+        "support": 18721.0
+      },
+      "weighted avg": {
+        "precision": 0.9553162576832412,
+        "recall": 0.9563591688478179,
+        "f1-score": 0.9555177384234166,
+        "support": 18721.0
+      }
+    },
+    "span_level": {
+      "DomainMotif": {
+        "precision": 0.6133333333333333,
+        "recall": 0.5679012345679012,
+        "f1-score": 0.5897435897435898,
+        "support": 81
+      },
+      "FamilyName": {
+        "precision": 0.5387323943661971,
+        "recall": 0.5134228187919463,
+        "f1-score": 0.5257731958762886,
+        "support": 298
+      },
+      "Gene": {
+        "precision": 0.8486682808716707,
+        "recall": 0.9127604166666666,
+        "f1-score": 0.8795483061480552,
+        "support": 768
+      },
+      "macro avg": {
+        "precision": 0.6669113361904003,
+        "recall": 0.6646948233421713,
+        "f1-score": 0.6650216972559778,
+        "support": 1147
+      },
+      "weighted avg": {
+        "precision": 0.7515252774460068,
+        "recall": 0.7846556233653008,
+        "f1-score": 0.7671689121726863,
+        "support": 1147
+      }
+    }
+  },
+  "test": {
+    "token_level": {
+      "O": {
+        "precision": 0.9869390488948426,
+        "recall": 0.9731650937657508,
+        "f1-score": 0.9800036754732172,
+        "support": 57537.0
+      },
+      "B-DomainMotif": {
+        "precision": 0.7365591397849462,
+        "recall": 0.7611111111111111,
+        "f1-score": 0.7486338797814208,
+        "support": 360.0
+      },
+      "I-DomainMotif": {
+        "precision": 0.8471337579617835,
+        "recall": 0.81511746680286,
+        "f1-score": 0.8308172826652785,
+        "support": 979.0
+      },
+      "B-FamilyName": {
+        "precision": 0.6186627479794269,
+        "recall": 0.6736,
+        "f1-score": 0.6449636154729989,
+        "support": 1250.0
+      },
+      "I-FamilyName": {
+        "precision": 0.6251338807568726,
+        "recall": 0.705763804917372,
+        "f1-score": 0.6630064369556986,
+        "support": 2481.0
+      },
+      "B-Gene": {
+        "precision": 0.8868312757201646,
+        "recall": 0.9337666357164964,
+        "f1-score": 0.9096939544700738,
+        "support": 3231.0
+      },
+      "I-Gene": {
+        "precision": 0.8947873361048929,
+        "recall": 0.9283344392833444,
+        "f1-score": 0.9112522390490149,
+        "support": 6028.0
+      },
+      "accuracy": 0.9499763448640526,
+      "macro avg": {
+        "precision": 0.7994353124575614,
+        "recall": 0.8272655073709906,
+        "f1-score": 0.8126244405525289,
+        "support": 71866.0
+      },
+      "weighted avg": {
+        "precision": 0.9526540061037759,
+        "recall": 0.9499763448640526,
+        "f1-score": 0.9511135021493018,
+        "support": 71866.0
+      }
+    },
+    "span_level": {
+      "DomainMotif": {
+        "precision": 0.6019900497512438,
+        "recall": 0.6703601108033241,
+        "f1-score": 0.6343381389252949,
+        "support": 361
+      },
+      "FamilyName": {
+        "precision": 0.4965083798882682,
+        "recall": 0.5688,
+        "f1-score": 0.5302013422818792,
+        "support": 1250
+      },
+      "Gene": {
+        "precision": 0.8559102674719585,
+        "recall": 0.9227906976744186,
+        "f1-score": 0.8880931065353624,
+        "support": 3225
+      },
+      "macro avg": {
+        "precision": 0.6514695657038235,
+        "recall": 0.7206502694925808,
+        "f1-score": 0.6842108625808455,
+        "support": 4836
+      },
+      "weighted avg": {
+        "precision": 0.7440580015338297,
+        "recall": 0.8124483043837882,
+        "f1-score": 0.7766435100456833,
+        "support": 4836
+      }
+    }
+  }
+}

performance_report.md

@@ -0,0 +1,81 @@
+# Performance on Training Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| DomainMotif | 0.907 | 0.888 | 0.897 | 197 |
+| FamilyName | 0.976 | 0.980 | 0.978 | 815 |
+| Gene | 0.988 | 0.976 | 0.982 | 2243 |
+| macro avg | 0.957 | 0.948 | 0.953 | 3255 |
+| weighted avg | 0.980 | 0.972 | 0.976 | 3255 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.999 | 1.000 | 0.999 | 45948 |
+| B-DomainMotif | 0.974 | 0.949 | 0.961 | 195 |
+| I-DomainMotif | 0.996 | 0.989 | 0.993 | 536 |
+| B-FamilyName | 0.993 | 0.990 | 0.991 | 816 |
+| I-FamilyName | 0.994 | 0.997 | 0.996 | 1829 |
+| B-Gene | 0.996 | 0.992 | 0.994 | 2239 |
+| I-Gene | 0.999 | 0.998 | 0.998 | 4411 |
+| macro avg | 0.993 | 0.988 | 0.990 | 55974 |
+| weighted avg | 0.999 | 0.999 | 0.999 | 55974 |
+
+
+# Performance on Validation Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| DomainMotif | 0.613 | 0.568 | 0.590 | 81 |
+| FamilyName | 0.539 | 0.513 | 0.526 | 298 |
+| Gene | 0.849 | 0.913 | 0.880 | 768 |
+| macro avg | 0.667 | 0.665 | 0.665 | 1147 |
+| weighted avg | 0.752 | 0.785 | 0.767 | 1147 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.982 | 0.983 | 0.982 | 15288 |
+| B-DomainMotif | 0.754 | 0.642 | 0.693 | 81 |
+| I-DomainMotif | 0.846 | 0.715 | 0.775 | 207 |
+| B-FamilyName | 0.682 | 0.609 | 0.643 | 299 |
+| I-FamilyName | 0.757 | 0.666 | 0.708 | 637 |
+| B-Gene | 0.872 | 0.938 | 0.904 | 763 |
+| I-Gene | 0.886 | 0.938 | 0.912 | 1446 |
+| macro avg | 0.826 | 0.784 | 0.803 | 18721 |
+| weighted avg | 0.955 | 0.956 | 0.956 | 18721 |
+
+
+# Performance on Testing Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| DomainMotif | 0.602 | 0.670 | 0.634 | 361 |
+| FamilyName | 0.497 | 0.569 | 0.530 | 1250 |
+| Gene | 0.856 | 0.923 | 0.888 | 3225 |
+| macro avg | 0.651 | 0.721 | 0.684 | 4836 |
+| weighted avg | 0.744 | 0.812 | 0.777 | 4836 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.987 | 0.973 | 0.980 | 57537 |
+| B-DomainMotif | 0.737 | 0.761 | 0.749 | 360 |
+| I-DomainMotif | 0.847 | 0.815 | 0.831 | 979 |
+| B-FamilyName | 0.619 | 0.674 | 0.645 | 1250 |
+| I-FamilyName | 0.625 | 0.706 | 0.663 | 2481 |
+| B-Gene | 0.887 | 0.934 | 0.910 | 3231 |
+| I-Gene | 0.895 | 0.928 | 0.911 | 6028 |
+| macro avg | 0.799 | 0.827 | 0.813 | 71866 |
+| weighted avg | 0.953 | 0.950 | 0.951 | 71866 |
+
+

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3956f5f0d871975d38a47184ff093b702da31d5c65f3d8334be329cc35ea62cf
+size 14244

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

trainer_state.json

@@ -0,0 +1,203 @@
+{
+  "best_metric": 0.8025602601482168,
+  "best_model_checkpoint": "tmp_ner_fantastic-bale-17_39/run-51/checkpoint-351",
+  "epoch": 13.0,
+  "eval_steps": 500,
+  "global_step": 351,
+  "is_hyper_param_search": true,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.0,
+      "eval_accuracy": 0.8966935526948346,
+      "eval_loss": 0.3334161937236786,
+      "eval_macro_f1": 0.3481690804569024,
+      "eval_macro_precision": 0.381260717978419,
+      "eval_macro_recall": 0.35979719914273406,
+      "eval_runtime": 0.6891,
+      "eval_samples_per_second": 206.072,
+      "eval_steps_per_second": 26.122,
+      "step": 27
+    },
+    {
+      "epoch": 2.0,
+      "eval_accuracy": 0.930559264996528,
+      "eval_loss": 0.21216891705989838,
+      "eval_macro_f1": 0.6053382787959644,
+      "eval_macro_precision": 0.6107512517099946,
+      "eval_macro_recall": 0.6460256647872952,
+      "eval_runtime": 0.6859,
+      "eval_samples_per_second": 207.037,
+      "eval_steps_per_second": 26.244,
+      "step": 54
+    },
+    {
+      "epoch": 3.0,
+      "eval_accuracy": 0.9471716254473586,
+      "eval_loss": 0.17325381934642792,
+      "eval_macro_f1": 0.654390209074469,
+      "eval_macro_precision": 0.8778760893535847,
+      "eval_macro_recall": 0.6000593968379591,
+      "eval_runtime": 0.6857,
+      "eval_samples_per_second": 207.088,
+      "eval_steps_per_second": 26.251,
+      "step": 81
+    },
+    {
+      "epoch": 4.0,
+      "eval_accuracy": 0.9528337161476417,
+      "eval_loss": 0.15815088152885437,
+      "eval_macro_f1": 0.780354726682967,
+      "eval_macro_precision": 0.7951782992828399,
+      "eval_macro_recall": 0.7756083475295977,
+      "eval_runtime": 0.6835,
+      "eval_samples_per_second": 207.761,
+      "eval_steps_per_second": 26.336,
+      "step": 108
+    },
+    {
+      "epoch": 5.0,
+      "eval_accuracy": 0.9519256449975962,
+      "eval_loss": 0.17041093111038208,
+      "eval_macro_f1": 0.7750872516025015,
+      "eval_macro_precision": 0.819684829234941,
+      "eval_macro_recall": 0.7465932066328682,
+      "eval_runtime": 0.6848,
+      "eval_samples_per_second": 207.368,
+      "eval_steps_per_second": 26.286,
+      "step": 135
+    },
+    {
+      "epoch": 6.0,
+      "eval_accuracy": 0.9541156989477058,
+      "eval_loss": 0.18207110464572906,
+      "eval_macro_f1": 0.7757922057464975,
+      "eval_macro_precision": 0.8139015262713462,
+      "eval_macro_recall": 0.7479356706189721,
+      "eval_runtime": 0.6847,
+      "eval_samples_per_second": 207.398,
+      "eval_steps_per_second": 26.29,
+      "step": 162
+    },
+    {
+      "epoch": 7.0,
+      "eval_accuracy": 0.9547032743977352,
+      "eval_loss": 0.1869019716978073,
+      "eval_macro_f1": 0.7909756567968369,
+      "eval_macro_precision": 0.818653317630622,
+      "eval_macro_recall": 0.7694133266548752,
+      "eval_runtime": 0.6858,
+      "eval_samples_per_second": 207.047,
+      "eval_steps_per_second": 26.245,
+      "step": 189
+    },
+    {
+      "epoch": 8.0,
+      "eval_accuracy": 0.9542225308477111,
+      "eval_loss": 0.19935451447963715,
+      "eval_macro_f1": 0.7742556094296894,
+      "eval_macro_precision": 0.829305822509576,
+      "eval_macro_recall": 0.7366273963466895,
+      "eval_runtime": 0.6857,
+      "eval_samples_per_second": 207.102,
+      "eval_steps_per_second": 26.252,
+      "step": 216
+    },
+    {
+      "epoch": 9.0,
+      "eval_accuracy": 0.9549169381977458,
+      "eval_loss": 0.19827748835086823,
+      "eval_macro_f1": 0.7890513633726142,
+      "eval_macro_precision": 0.8190924393392907,
+      "eval_macro_recall": 0.7654512068808164,
+      "eval_runtime": 0.6844,
+      "eval_samples_per_second": 207.478,
+      "eval_steps_per_second": 26.3,
+      "step": 243
+    },
+    {
+      "epoch": 10.0,
+      "eval_accuracy": 0.9539554510976977,
+      "eval_loss": 0.21724671125411987,
+      "eval_macro_f1": 0.7951226311261627,
+      "eval_macro_precision": 0.8076887756893915,
+      "eval_macro_recall": 0.7863682941618138,
+      "eval_runtime": 0.6853,
+      "eval_samples_per_second": 207.214,
+      "eval_steps_per_second": 26.267,
+      "step": 270
+    },
+    {
+      "epoch": 11.0,
+      "eval_accuracy": 0.9528871320976443,
+      "eval_loss": 0.24698173999786377,
+      "eval_macro_f1": 0.7700880502690434,
+      "eval_macro_precision": 0.837595387901314,
+      "eval_macro_recall": 0.7281291396407018,
+      "eval_runtime": 0.6867,
+      "eval_samples_per_second": 206.792,
+      "eval_steps_per_second": 26.213,
+      "step": 297
+    },
+    {
+      "epoch": 12.0,
+      "eval_accuracy": 0.9551306019977566,
+      "eval_loss": 0.23617151379585266,
+      "eval_macro_f1": 0.7840279783447406,
+      "eval_macro_precision": 0.8305447049199165,
+      "eval_macro_recall": 0.7522814462795042,
+      "eval_runtime": 0.6856,
+      "eval_samples_per_second": 207.122,
+      "eval_steps_per_second": 26.255,
+      "step": 324
+    },
+    {
+      "epoch": 13.0,
+      "eval_accuracy": 0.9563591688478179,
+      "eval_loss": 0.22979934513568878,
+      "eval_macro_f1": 0.8025602601482168,
+      "eval_macro_precision": 0.8255341113341885,
+      "eval_macro_recall": 0.784416544697495,
+      "eval_runtime": 0.6867,
+      "eval_samples_per_second": 206.792,
+      "eval_steps_per_second": 26.213,
+      "step": 351
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 864,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 32,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 3,
+        "early_stopping_threshold": 0.001
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": {
+    "learning_rate": 4.312024782506724e-05,
+    "per_device_train_batch_size": 16,
+    "warmup_ratio": 0.00046632724074153857,
+    "weight_decay": 0.004637010897989902
+  }
+}