GPT2WaP

This model is a gpt2 model trained from scratch on the War and peace book. It achieves the following results on the evaluation set:

• Loss: 9.0987
• Perplexity: 8943.6289

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 4
• total_train_batch_size: 512
• total_eval_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Perplexity
10.157	0.6897	10	9.2336	10235.7480
9.2581	1.3793	20	8.9452	7671.1870
8.8166	2.0690	30	9.4917	13248.7207
8.5094	2.7586	40	9.5417	13928.9434
8.0914	3.4483	50	9.5507	14054.4785
7.663	4.1379	60	9.4760	13043.2441
7.3275	4.8276	70	9.3510	11510.8203
6.9788	5.5172	80	9.0822	8797.7188
6.6639	6.2069	90	8.9803	7945.4014
6.3749	6.8966	100	8.6494	5706.8130
6.0702	7.5862	110	8.5696	5268.9268
5.9107	8.2759	120	8.3612	4277.6265
5.6724	8.9655	130	8.4294	4579.6484
5.5949	9.6552	140	8.4934	4882.4316
5.4904	10.3448	150	8.4683	4761.3862
5.3792	11.0345	160	8.4647	4744.5381
5.3091	11.7241	170	8.5767	5306.3535
5.233	12.4138	180	8.5257	5042.5068
5.2252	13.1034	190	8.5328	5078.8433
5.1445	13.7931	200	8.5871	5361.9390
5.0824	14.4828	210	8.5784	5315.4043
5.0272	15.1724	220	8.6434	5672.6934
4.979	15.8621	230	8.6836	5905.4277
4.924	16.5517	240	8.7112	6070.2261
4.9394	17.2414	250	8.7233	6144.3931
4.8663	17.9310	260	8.7411	6254.5234
4.8599	18.6207	270	8.7824	6518.7896
4.8572	19.3103	280	8.8338	6862.5586
4.8064	20.0	290	8.7774	6485.7441
4.746	20.6897	300	8.8458	6944.8892
4.7569	21.3793	310	8.8436	6930.1416
4.6954	22.0690	320	8.8618	7057.1084
4.7277	22.7586	330	8.8706	7119.4478
4.6432	23.4483	340	8.9084	7393.6138
4.6032	24.1379	350	8.9111	7413.5176
4.6198	24.8276	360	8.9526	7728.0210
4.5874	25.5172	370	8.9740	7895.1641
4.5455	26.2069	380	8.9365	7604.7129
4.5313	26.8966	390	8.9738	7893.2969
4.5297	27.5862	400	8.9659	7831.8110
4.5279	28.2759	410	8.9914	8034.0391
4.4974	28.9655	420	9.0293	8344.2529
4.4554	29.6552	430	9.0191	8259.1533
4.4651	30.3448	440	9.0236	8296.4531
4.4647	31.0345	450	9.0349	8391.1279
4.4668	31.7241	460	9.0530	8543.8340
4.4264	32.4138	470	9.0722	8709.4141
4.4008	33.1034	480	9.0876	8844.6104
4.3982	33.7931	490	9.0711	8700.4893
4.3846	34.4828	500	9.0894	8860.7441
4.3971	35.1724	510	9.0879	8847.6973
4.379	35.8621	520	9.0949	8909.6025
4.3696	36.5517	530	9.1097	9042.2295
4.3447	37.2414	540	9.1007	8961.6953
4.3796	37.9310	550	9.0869	8839.0781
4.364	38.6207	560	9.0987	8943.6289

Framework versions

• Transformers 4.40.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1