--- license: mit language: en tags: - generated_from_trainer model-index: - name: verdict-classifier-en results: - task: type: text-classification name: Verdict Classification widget: - "One might think that this is true, but it's taken out of context." --- # English Verdict Classifier This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on 2,500 deduplicated verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into English with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/). It achieves the following results on the evaluation set, being 1,000 such verdicts translated into English, but here including duplicates to represent the true distribution: - Loss: 0.1304 - F1 Macro: 0.8868 - F1 Misinformation: 0.9832 - F1 Factual: 0.9890 - F1 Other: 0.6882 - Prec Macro: 0.8580 - Prec Misinformation: 0.9918 - Prec Factual: 0.9783 - Prec Other: 0.6038 ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - gradient_accumulation_steps: 8 - total_train_batch_size: 32 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 625 - num_epochs: 1000 ### Training results | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other | |:-------------:|:-----:|:----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:| | 1.0588 | 0.64 | 50 | 1.0803 | 0.0256 | 0.0 | 0.0 | 0.0768 | 0.0133 | 0.0 | 0.0 | 0.0400 | | 0.9885 | 1.28 | 100 | 1.0055 | 0.3497 | 0.9291 | 0.0 | 0.12 | 0.3910 | 0.8729 | 0.0 | 0.3 | | 0.971 | 1.92 | 150 | 0.9218 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 | | 0.9263 | 2.56 | 200 | 0.6035 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 | | 0.8672 | 3.2 | 250 | 0.3639 | 0.4428 | 0.9337 | 0.0 | 0.3946 | 0.3976 | 0.9217 | 0.0 | 0.2710 | | 0.743 | 3.84 | 300 | 0.2396 | 0.7944 | 0.9698 | 0.9091 | 0.5043 | 0.7893 | 0.9812 | 1.0 | 0.3867 | | 0.5106 | 4.49 | 350 | 0.1579 | 0.8399 | 0.9733 | 0.9888 | 0.5577 | 0.8130 | 0.9859 | 1.0 | 0.4531 | | 0.4215 | 5.13 | 400 | 0.1245 | 0.8174 | 0.9747 | 0.9834 | 0.4941 | 0.8076 | 0.9780 | 0.9780 | 0.4667 | | 0.3941 | 5.77 | 450 | 0.1422 | 0.8298 | 0.9678 | 1.0 | 0.5217 | 0.7960 | 0.9880 | 1.0 | 0.4 | | 0.3105 | 6.41 | 500 | 0.1352 | 0.8223 | 0.9696 | 0.9836 | 0.5138 | 0.7872 | 0.9881 | 0.9677 | 0.4058 | | 0.3126 | 7.05 | 550 | 0.1126 | 0.8423 | 0.9756 | 0.9945 | 0.5567 | 0.8162 | 0.9859 | 0.9890 | 0.4737 | | 0.2206 | 7.69 | 600 | 0.1206 | 0.8557 | 0.9761 | 0.9890 | 0.6019 | 0.8203 | 0.9905 | 0.9783 | 0.4921 | | 0.2472 | 8.33 | 650 | 0.1296 | 0.8481 | 0.9731 | 0.9945 | 0.5766 | 0.8105 | 0.9917 | 0.9890 | 0.4507 | | 0.1839 | 8.97 | 700 | 0.1357 | 0.8582 | 0.9761 | 0.9890 | 0.6095 | 0.8208 | 0.9917 | 0.9783 | 0.4923 | | 0.1282 | 9.61 | 750 | 0.1465 | 0.8481 | 0.9756 | 0.9945 | 0.5743 | 0.8175 | 0.9882 | 0.9890 | 0.4754 | | 0.1447 | 10.26 | 800 | 0.1621 | 0.8602 | 0.9767 | 0.9945 | 0.6095 | 0.8243 | 0.9917 | 0.9890 | 0.4923 | | 0.1223 | 10.9 | 850 | 0.1304 | 0.8868 | 0.9832 | 0.9890 | 0.6882 | 0.8580 | 0.9918 | 0.9783 | 0.6038 | | 0.1053 | 11.54 | 900 | 0.1640 | 0.8714 | 0.9797 | 0.9945 | 0.64 | 0.8380 | 0.9918 | 0.9890 | 0.5333 | | 0.064 | 12.18 | 950 | 0.1983 | 0.8627 | 0.9791 | 0.9889 | 0.62 | 0.8321 | 0.9906 | 0.9889 | 0.5167 | | 0.1085 | 12.82 | 1000 | 0.1811 | 0.8688 | 0.9803 | 0.9945 | 0.6316 | 0.8413 | 0.9895 | 0.9890 | 0.5455 | | 0.0885 | 13.46 | 1050 | 0.2052 | 0.8710 | 0.9821 | 0.9945 | 0.6364 | 0.8532 | 0.9872 | 0.9890 | 0.5833 | | 0.0799 | 14.1 | 1100 | 0.1826 | 0.8801 | 0.9827 | 0.9836 | 0.6742 | 0.8565 | 0.9895 | 0.9677 | 0.6122 | | 0.0737 | 14.74 | 1150 | 0.2158 | 0.8556 | 0.9761 | 0.9945 | 0.5962 | 0.8213 | 0.9905 | 0.9890 | 0.4844 | | 0.0564 | 15.38 | 1200 | 0.2283 | 0.8637 | 0.9797 | 0.9945 | 0.6170 | 0.8381 | 0.9883 | 0.9890 | 0.5370 | | 0.0547 | 16.03 | 1250 | 0.2508 | 0.8693 | 0.9785 | 0.9888 | 0.6408 | 0.8381 | 0.9906 | 1.0 | 0.5238 | | 0.0602 | 16.67 | 1300 | 0.2320 | 0.8555 | 0.9798 | 0.9889 | 0.5977 | 0.8420 | 0.9838 | 0.9889 | 0.5532 | | 0.0576 | 17.31 | 1350 | 0.2346 | 0.8737 | 0.9803 | 0.9945 | 0.6465 | 0.8411 | 0.9918 | 0.9890 | 0.5424 | ### Framework versions - Transformers 4.11.3 - Pytorch 1.9.0+cu102 - Datasets 1.9.0 - Tokenizers 0.10.2