OKAI-midi-gen-v-002

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 10.7039
• Accuracy: 0.0004

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

training_config:

• max_position_embeddings: 1024
• num_attention_heads: 8
• num_hidden_layers: 8
• sliding_window: 4
• intermediate_size: 512
• hidden_size: 256
• num_key_value_heads: 4

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 16
• seed: 444
• gradient_accumulation_steps: 3
• total_train_batch_size: 24
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_ratio: 0.3
• training_steps: 2000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
10.2078	3.2283	100	10.2675	0.0004
9.6734	6.4565	200	10.0564	0.0030
9.1879	9.6848	300	9.9516	0.0036
8.8395	12.9130	400	9.9849	0.0038
8.4477	16.1304	500	10.0295	0.0037
7.9007	19.3587	600	10.0765	0.0023
7.2277	22.5870	700	10.1370	0.0016
6.5655	25.8152	800	10.2042	0.0011
5.8915	29.0326	900	10.2386	0.0010
5.1573	32.2609	1000	10.3026	0.0006
4.6311	35.4891	1100	10.3649	0.0006
4.1848	38.7174	1200	10.4672	0.0005
3.7458	41.9457	1300	10.5312	0.0003
3.487	45.1630	1400	10.5832	0.0003
3.206	48.3913	1500	10.6327	0.0005
3.0194	51.6196	1600	10.6676	0.0005
2.9182	54.8478	1700	10.6874	0.0005
2.9033	58.0652	1800	10.6992	0.0004
2.8987	61.2935	1900	10.7033	0.0005
2.8351	64.5217	2000	10.7039	0.0004

Framework versions

• Transformers 4.52.3
• Pytorch 2.6.0
• Datasets 3.6.0
• Tokenizers 0.21.1