stamina/finance_model

pretrain_sft_finance

This model is a fine-tuned version of instruction-pretrain/finance-Llama3-8B on the time_dataset dataset. It achieves the following results on the evaluation set:

• Loss: 0.2738

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
4.1092	0.0649	10	4.4912
2.1524	0.1299	20	1.8664
0.7796	0.1948	30	0.6781
0.3127	0.2597	40	0.2871
0.4223	0.3247	50	0.2762
0.2854	0.3896	60	0.2877
0.2908	0.4545	70	0.3328
0.4468	0.5195	80	0.3878
0.2962	0.5844	90	0.2747
0.2759	0.6494	100	0.2835
0.3065	0.7143	110	0.2901
0.2882	0.7792	120	0.2735
0.2945	0.8442	130	0.2920
0.2805	0.9091	140	0.2734
0.2696	0.9740	150	0.2738

Framework versions

• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1