yzw1424349151/Mistral-7B-v0_3-qlora

See axolotl config

axolotl version: 0.10.0

base_model: mistralai/Mistral-7B-v0.1
# optionally might have model_type or tokenizer_type
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name

load_in_8bit: false
load_in_4bit: true

datasets:

  - path: "cognitivecomputations/dolphin"
    name: "flan1m-alpaca-uncensored"
    type: alpaca
    split: train[:25000]

  - path: causal-lm/ultrachat
    type: alpaca
    split: train[:25000]

  
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/qlora-out-0-3

adapter: qlora
lora_model_dir:

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

bf16: auto
tf32: false

gradient_checkpointing: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.0
special_tokens:

outputs/qlora-out-0-3

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the cognitivecomputations/dolphin and the causal-lm/ultrachat datasets. It achieves the following results on the evaluation set:

• Loss: 0.9045

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 897

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0	0	0.9881
0.8599	0.2508	75	0.8906
0.9047	0.5017	150	0.8825
0.9083	0.7525	225	0.8768
0.7706	1.0033	300	0.8729
0.8222	1.2542	375	0.8810
0.8122	1.5050	450	0.8807
0.7571	1.7559	525	0.8785
0.7191	2.0067	600	0.8822
0.6796	2.2575	675	0.9030
0.6789	2.5084	750	0.9029
0.6852	2.7592	825	0.9045

Framework versions

• PEFT 0.15.2
• Transformers 4.52.3
• Pytorch 2.5.1+cu121
• Datasets 3.6.0
• Tokenizers 0.21.2