lambda-qwen2.5-32b-dpo-test

This model is a fine-tuned version of Qwen/Qwen2.5-32B-Instruct on the tanliboy/orca_dpo_pairs dataset. It achieves the following results on the evaluation set:

• Loss: 0.0003
• Rewards/chosen: -10.1618
• Rewards/rejected: -26.1150
• Rewards/accuracies: 1.0
• Rewards/margins: 15.9532
• Logps/rejected: -3049.5271
• Logps/chosen: -1372.8903
• Logits/rejected: -0.3154
• Logits/chosen: -0.0600

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.0103	0.2618	100	0.0060	-8.5159	-18.7731	1.0	10.2572	-2315.3333	-1208.2968	-0.5485	-0.2481
0.0005	0.5236	200	0.0005	-9.9255	-24.9117	1.0	14.9862	-2929.1948	-1349.2588	-0.3723	-0.0661
0.0005	0.7853	300	0.0004	-10.0873	-25.9319	1.0	15.8446	-3031.2175	-1365.4342	-0.2882	-0.0014

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1