collapse_gemma-2-9b_hs2_replace_iter2_sftsd1

This model is a fine-tuned version of google/gemma-2-9b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1667
• Num Input Tokens Seen: 4350556

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.2335	0
1.0016	0.0563	5	1.0746	250140
0.7083	0.1127	10	1.0700	500448
0.4411	0.1690	15	1.0957	746176
0.2245	0.2254	20	1.1211	994844
0.1716	0.2817	25	1.1361	1248704
0.1605	0.3380	30	1.1311	1500268
0.1207	0.3944	35	1.1025	1749428
0.1501	0.4507	40	1.1084	1998696
0.0745	0.5070	45	1.0939	2243152
0.0467	0.5634	50	1.1048	2486984
0.0899	0.6197	55	1.0906	2726740
0.069	0.6761	60	1.1076	2974952
0.0715	0.7324	65	1.1280	3216652
0.0874	0.7887	70	1.1374	3463512
0.0896	0.8451	75	1.1419	3713776
0.06	0.9014	80	1.1572	3956984
0.0665	0.9577	85	1.1584	4200536

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1