collapse_gemma-2-2b_hs2_accumulatesubsample_iter17_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2002
• Num Input Tokens Seen: 4886264

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.2899	0.0527	5	1.2767	253456
1.1118	0.1054	10	1.2083	517968
0.8628	0.1581	15	1.2166	778368
0.7433	0.2108	20	1.2617	1045544
0.6547	0.2635	25	1.2726	1304672
0.5625	0.3162	30	1.2983	1568040
0.4794	0.3689	35	1.2925	1823840
0.3948	0.4216	40	1.2635	2081800
0.5003	0.4743	45	1.2479	2341320
0.4144	0.5270	50	1.2399	2601568
0.3792	0.5797	55	1.2391	2858360
0.3328	0.6324	60	1.2281	3113832
0.3185	0.6851	65	1.2214	3374064
0.3413	0.7378	70	1.2211	3633024
0.3521	0.7905	75	1.2056	3892408
0.2883	0.8432	80	1.2204	4155792
0.3138	0.8959	85	1.2073	4415152
0.3163	0.9486	90	1.2147	4676800

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1