O0507TESTV2

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -8.4955

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.863	0.09	10	-2.3940
-3.4568	0.18	20	-3.8604
-3.8975	0.27	30	-3.9271
-3.9809	0.36	40	-4.0442
-4.1019	0.45	50	-4.1819
-4.2453	0.54	60	-4.3517
-4.4388	0.63	70	-4.5620
-4.6598	0.73	80	-4.8056
-4.9318	0.82	90	-5.1072
-5.2431	0.91	100	-5.4448
-5.4952	1.0	110	-4.5478
-5.5982	1.09	120	-5.4325
-5.9031	1.18	130	-6.2539
-6.3415	1.27	140	-6.5544
-6.5926	1.36	150	-6.6084
-6.3162	1.45	160	-7.0376
-6.8327	1.54	170	-6.8154
-6.9773	1.63	180	-7.2898
-7.314	1.72	190	-7.4747
-7.5529	1.81	200	-7.6409
-7.7215	1.9	210	-7.8216
-7.851	1.99	220	-7.9723
-8.0306	2.08	230	-8.0959
-8.0824	2.18	240	-8.1964
-8.1233	2.27	250	-8.2747
-8.2767	2.36	260	-8.3412
-8.3555	2.45	270	-8.3954
-8.2976	2.54	280	-8.4392
-8.4536	2.63	290	-8.4655
-8.4442	2.72	300	-8.4827
-8.4231	2.81	310	-8.4916
-8.4211	2.9	320	-8.4945
-8.486	2.99	330	-8.4955

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0