gemma-fineweb-edu-scorer-xlm-binary-xlm-roberta-base-lr5e-05-20250411_122512

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1484
• Precision: 0.7851
• Recall: 0.7983
• F1 Macro: 0.7913
• Accuracy: 0.8438

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 128
• seed: 0
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1 Macro	Accuracy
No log	0	0	0.2350	0.3796	0.5	0.4316	0.7593
0.1059	0.7812	1000	0.1000	0.8077	0.7659	0.7830	0.8517
0.0825	1.5625	2000	0.0995	0.8129	0.7662	0.7850	0.8540
0.0563	2.3438	3000	0.1118	0.7910	0.8195	0.8031	0.8488
0.0315	3.125	4000	0.1183	0.8049	0.7861	0.7947	0.8545
0.0359	3.9062	5000	0.1234	0.8174	0.7528	0.7766	0.8524
0.0213	4.6875	6000	0.1431	0.7791	0.8259	0.7960	0.8374
0.0162	5.4688	7000	0.1312	0.7904	0.8026	0.7961	0.8478
0.0111	6.25	8000	0.1509	0.7817	0.8155	0.7955	0.8411
0.013	7.0312	9000	0.1242	0.7998	0.7994	0.7996	0.8536
0.0135	7.8125	10000	0.1292	0.8162	0.7570	0.7794	0.8530
0.0094	8.5938	11000	0.1463	0.7814	0.8172	0.7957	0.8406
0.0063	9.375	12000	0.1427	0.7799	0.8202	0.7954	0.8390
0.0082	10.1562	13000	0.1369	0.7838	0.8082	0.7944	0.8430
0.0081	10.9375	14000	0.1390	0.7880	0.7840	0.7859	0.8446
0.0058	11.7188	15000	0.1421	0.7833	0.8072	0.7937	0.8426
0.0061	12.5	16000	0.1580	0.7646	0.8190	0.7819	0.8225
0.005	13.2812	17000	0.1414	0.7924	0.7963	0.7943	0.8486
0.0031	14.0625	18000	0.1403	0.7935	0.8052	0.7990	0.8501
0.0032	14.8438	19000	0.1412	0.7958	0.7873	0.7914	0.8496
0.0046	15.625	20000	0.1458	0.7852	0.8119	0.7967	0.8441
0.0027	16.4062	21000	0.1490	0.7832	0.8115	0.7952	0.8425
0.0025	17.1875	22000	0.1442	0.7886	0.7995	0.7938	0.8463
0.0025	17.9688	23000	0.1431	0.7962	0.7811	0.7881	0.8490
0.0017	18.75	24000	0.1496	0.7834	0.8036	0.7925	0.8427
0.0022	19.5312	25000	0.1484	0.7851	0.7983	0.7913	0.8438

Framework versions

• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.3.2
• Tokenizers 0.21.1