myBit-Llama2-jp-127M-2B4TLike-wiki-aozora-sort

This model is a fine-tuned version of on the None dataset. It achieves the following results on the evaluation set:

• Loss: 4.0765

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0024
• train_batch_size: 24
• eval_batch_size: 24
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 96
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 750
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
6.0948	0.0495	500	7.3161
5.2631	0.0989	1000	6.5832
5.0186	0.1484	1500	6.0528
4.8511	0.1979	2000	5.7678
4.7622	0.2473	2500	5.5096
4.664	0.2968	3000	5.3218
4.584	0.3463	3500	5.1782
4.511	0.3957	4000	5.0545
4.4587	0.4452	4500	4.9509
4.3964	0.4946	5000	4.8655
4.3395	0.5441	5500	4.7613
4.2698	0.5936	6000	4.6568
4.2505	0.6430	6500	4.5421
4.2181	0.6925	7000	4.4322
4.1992	0.7420	7500	4.3288
4.2482	0.7914	8000	4.2346
4.2172	0.8409	8500	4.1939
4.1592	0.8904	9000	4.1554
4.1238	0.9398	9500	4.1194
4.1023	0.9893	10000	4.0765

Framework versions

• Transformers 4.47.1
• Pytorch 2.6.0+cu124
• Datasets 3.5.1
• Tokenizers 0.21.1