Mistral_ei_oc_structured_train

This model is a fine-tuned version of mistralai/Mistral-7B-v0.3 on the emollms_ei_oc_structured dataset. It achieves the following results on the evaluation set:

• Loss: 0.0778

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.4849	0.3604	10	0.1087
0.0931	0.7207	20	0.0948
0.0758	1.0811	30	0.0815
0.067	1.4414	40	0.0778
0.0619	1.8018	50	0.0781
0.059	2.1622	60	0.0818
0.0497	2.5225	70	0.0838

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1