O0507TESTV4

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -6.8545

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
-0.942	0.09	10	-4.0337
-4.9269	0.18	20	-5.1799
-5.2024	0.27	30	-5.2156
-5.2532	0.36	40	-5.2896
-5.3213	0.45	50	-5.3646
-5.399	0.54	60	-5.4559
-5.4992	0.63	70	-5.5550
-5.6013	0.73	80	-5.6645
-5.7233	0.82	90	-5.7934
-5.7728	0.91	100	-4.1088
-4.3845	1.0	110	-5.9599
-5.9017	1.09	120	-6.0444
-5.783	1.18	130	-5.1570
-5.82	1.27	140	-6.2132
-6.1104	1.36	150	-6.1753
-5.8551	1.45	160	-6.3266
-6.2638	1.54	170	-6.4003
-6.4124	1.63	180	-6.4657
-6.4804	1.72	190	-6.5352
-6.5576	1.81	200	-6.5940
-6.6088	1.9	210	-6.6424
-6.6357	1.99	220	-6.6778
-6.6971	2.08	230	-6.7281
-6.7339	2.18	240	-6.7584
-6.7424	2.27	250	-6.7835
-6.7862	2.36	260	-6.8037
-6.8111	2.45	270	-6.8213
-6.7991	2.54	280	-6.8343
-6.8401	2.63	290	-6.8422
-6.837	2.72	300	-6.8492
-6.8343	2.81	310	-6.8525
-6.8335	2.9	320	-6.8541
-6.8489	2.99	330	-6.8545

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0