qg-isg-simp-m-base-ls0.0

This model is a fine-tuned version of bigscience/mt0-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5101

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 24
• eval_batch_size: 24
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
0.5634	0.1850	10000	0.5951
0.5066	0.3700	20000	0.5563
0.4973	0.5551	30000	0.5425
0.4243	0.7401	40000	0.5325
0.4146	0.9251	50000	0.5237
0.4271	1.1101	60000	0.5314
0.393	1.2951	70000	0.5224
0.3964	1.4801	80000	0.5217
0.3695	1.6652	90000	0.5192
0.3695	1.8502	100000	0.5171
0.3424	2.0352	110000	0.5214
0.3161	2.2202	120000	0.5084
0.3455	2.4052	130000	0.5105
0.3253	2.5902	140000	0.5096
0.2994	2.7753	150000	0.5091
0.3047	2.9603	160000	0.5101

Framework versions

• Transformers 4.51.3
• Pytorch 2.5.0a0+e000cf0ad9.nv24.10
• Tokenizers 0.21.1