---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- alignment-handbook
- trl
- dpo
- generated_from_trainer
datasets:
- HuggingFaceH4/ultrafeedback_binarized
library_name: peft
model-index:
- name: zephyr-7b-nca_pair-qlora-lr5e6-beta0.1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/1014579852qq-tsinghua-university/huggingface/runs/p6zmvlxg)
# zephyr-7b-nca_pair-qlora-lr5e6-beta0.1

This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-qlora](https://huggingface.co/alignment-handbook/zephyr-7b-sft-qlora) on the HuggingFaceH4/ultrafeedback_binarized dataset.
It achieves the following results on the evaluation set:
- Loss: 1.3408
- Rewards/chosen: 0.1419
- Rewards/rejected: -0.3279
- Rewards/accuracies: 0.7450
- Rewards/margins: 0.4698
- Logps/rejected: -257.3817
- Logps/chosen: -270.7577
- Logits/rejected: -2.2321
- Logits/chosen: -2.3119

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 5
- gradient_accumulation_steps: 4
- total_train_batch_size: 40
- total_eval_batch_size: 20
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 1.3652        | 0.0654 | 100  | 1.3585          | 0.0179         | -0.2204          | 0.7200             | 0.2383          | -256.3067      | -271.9973    | -2.2040         | -2.2872       |
| 1.3457        | 0.1308 | 200  | 1.3529          | 0.1799         | -0.2015          | 0.7425             | 0.3814          | -256.1176      | -270.3774    | -2.2080         | -2.2912       |
| 1.3328        | 0.1963 | 300  | 1.3500          | 0.1269         | -0.2919          | 0.7150             | 0.4188          | -257.0218      | -270.9071    | -2.2303         | -2.3106       |
| 1.3452        | 0.2617 | 400  | 1.3536          | 0.1854         | -0.2395          | 0.7200             | 0.4249          | -256.4976      | -270.3225    | -2.2250         | -2.3062       |
| 1.3446        | 0.3271 | 500  | 1.3501          | 0.0859         | -0.3936          | 0.7275             | 0.4795          | -258.0389      | -271.3175    | -2.1984         | -2.2818       |
| 1.333         | 0.3925 | 600  | 1.3496          | 0.0493         | -0.3851          | 0.7450             | 0.4344          | -257.9544      | -271.6837    | -2.2107         | -2.2937       |
| 1.3577        | 0.4580 | 700  | 1.3457          | 0.1306         | -0.2688          | 0.7175             | 0.3994          | -256.7908      | -270.8706    | -2.2100         | -2.2934       |
| 1.343         | 0.5234 | 800  | 1.3449          | 0.0814         | -0.3810          | 0.7150             | 0.4623          | -257.9127      | -271.3629    | -2.2312         | -2.3121       |
| 1.3439        | 0.5888 | 900  | 1.3459          | 0.0385         | -0.4054          | 0.7250             | 0.4439          | -258.1573      | -271.7917    | -2.2327         | -2.3137       |
| 1.3388        | 0.6542 | 1000 | 1.3442          | 0.2150         | -0.2625          | 0.7325             | 0.4775          | -256.7277      | -270.0262    | -2.2387         | -2.3183       |
| 1.3186        | 0.7197 | 1100 | 1.3423          | 0.1242         | -0.3587          | 0.7325             | 0.4829          | -257.6895      | -270.9345    | -2.2306         | -2.3107       |
| 1.3299        | 0.7851 | 1200 | 1.3417          | 0.1468         | -0.3270          | 0.7425             | 0.4737          | -257.3728      | -270.7089    | -2.2275         | -2.3078       |
| 1.3248        | 0.8505 | 1300 | 1.3413          | 0.1555         | -0.3132          | 0.7525             | 0.4687          | -257.2347      | -270.6216    | -2.2306         | -2.3105       |
| 1.3398        | 0.9159 | 1400 | 1.3414          | 0.1409         | -0.3251          | 0.7475             | 0.4660          | -257.3535      | -270.7675    | -2.2317         | -2.3117       |
| 1.325         | 0.9814 | 1500 | 1.3409          | 0.1433         | -0.3268          | 0.7475             | 0.4701          | -257.3707      | -270.7436    | -2.2339         | -2.3137       |


### Framework versions

- PEFT 0.10.0
- Transformers 4.43.1
- Pytorch 2.1.2+cu121
- Datasets 2.18.0
- Tokenizers 0.19.1