openai-fineweb-edu-scorer-xlm-binary-xlm-roberta-base-lr5e-05-20250411_122038

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1567
• Precision: 0.8363
• Recall: 0.8275
• F1 Macro: 0.8307
• Accuracy: 0.8358

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 128
• seed: 0
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1 Macro	Accuracy
No log	0	0	0.4902	0.2156	0.5	0.3013	0.4311
0.113	0.7812	1000	0.1096	0.8549	0.8348	0.8405	0.8472
0.0936	1.5625	2000	0.1076	0.8531	0.8457	0.8486	0.8528
0.0614	2.3438	3000	0.1205	0.8425	0.8381	0.8400	0.8439
0.0431	3.125	4000	0.1294	0.8341	0.8392	0.8355	0.8371
0.0476	3.9062	5000	0.1448	0.8453	0.8226	0.8285	0.8364
0.0332	4.6875	6000	0.1394	0.8403	0.8261	0.8306	0.8368
0.0258	5.4688	7000	0.1402	0.8380	0.8331	0.8351	0.8393
0.0234	6.25	8000	0.1382	0.8411	0.8285	0.8326	0.8384
0.019	7.0312	9000	0.1398	0.8381	0.8295	0.8326	0.8377
0.0172	7.8125	10000	0.1569	0.8344	0.8134	0.8190	0.8270
0.016	8.5938	11000	0.1440	0.8399	0.8286	0.8324	0.8380
0.0127	9.375	12000	0.1500	0.8394	0.8341	0.8363	0.8405
0.0102	10.1562	13000	0.1497	0.8331	0.8330	0.8331	0.8362
0.0121	10.9375	14000	0.1506	0.8355	0.8210	0.8255	0.8320
0.008	11.7188	15000	0.1491	0.8410	0.8302	0.8339	0.8393
0.009	12.5	16000	0.1456	0.8384	0.8350	0.8364	0.8403
0.0053	13.2812	17000	0.1542	0.8377	0.8277	0.8312	0.8366
0.0062	14.0625	18000	0.1514	0.8347	0.8318	0.8331	0.8369
0.0034	14.8438	19000	0.1549	0.8325	0.8335	0.8330	0.8359
0.0045	15.625	20000	0.1530	0.8343	0.8309	0.8324	0.8364
0.0036	16.4062	21000	0.1555	0.8312	0.8315	0.8314	0.8345
0.001	17.1875	22000	0.1573	0.8329	0.8308	0.8318	0.8355
0.0029	17.9688	23000	0.1558	0.8334	0.8309	0.8320	0.8358
0.0018	18.75	24000	0.1543	0.8376	0.8357	0.8366	0.8401
0.0018	19.5312	25000	0.1567	0.8363	0.8275	0.8307	0.8358

Framework versions

• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.3.2
• Tokenizers 0.21.1