O0506TEST6

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -4.3565

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.977	0.09	10	-1.0050
-1.6595	0.18	20	-1.8402
-1.871	0.27	30	-1.8787
-1.9097	0.36	40	-1.9390
-1.9688	0.45	50	-2.0092
-2.0406	0.54	60	-2.0941
-2.1363	0.63	70	-2.2000
-2.2492	0.73	80	-2.3229
-2.3893	0.82	90	-2.4755
-2.5463	0.91	100	-2.6467
-2.7222	1.0	110	-2.8442
-2.9328	1.09	120	-3.0124
-2.2671	1.18	130	-3.0900
-3.1533	1.27	140	-2.3725
-3.111	1.36	150	-3.3882
-3.4384	1.45	160	-3.5451
-3.5427	1.54	170	-3.6589
-3.6817	1.63	180	-3.6443
-1.9189	1.72	190	-1.9290
-0.861	1.81	200	-2.3558
-2.9997	1.9	210	-3.6258
-3.7286	1.99	220	-3.8329
-3.9009	2.08	230	-3.9615
-3.9864	2.18	240	-4.0995
-4.0806	2.27	250	-4.1864
-4.1997	2.36	260	-4.2364
-4.2574	2.45	270	-4.2755
-4.229	2.54	280	-4.3103
-4.3281	2.63	290	-4.3305
-4.3242	2.72	300	-4.3461
-4.3173	2.81	310	-4.3527
-4.319	2.9	320	-4.3550
-4.3546	2.99	330	-4.3565

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0