jasonhuang3/sft-lora-2k

See axolotl config

axolotl version: 0.7.0

base_model: yentinglin/Llama-3-Taiwan-8B-Instruct
#trust_remote_code: true
load_in_8bit: false
load_in_4bit: true
strict: false

hub_model_id: jasonhuang3/jason-2k-1
hub_strategy: end
wandb_name: jason 8b 2k

dataset_processes: 16
datasets:
  - path: jasonhuang3/taitung_gemini_v1v3_messages
#  - path: jasonhuang3/instruct_output_v1_messages
    type: chat_template
    field_messages: messages
chat_template: llama3

dataset_prepared_path: last_run_prepared_jason_8b
val_set_size: 0.05 #
output_dir: ./output/8b/jason/2k-1

save_safetensors: true
sequence_len: 2048 #
sample_packing: true
pad_to_sequence_len: true

wandb_project: jasontwllm
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 2
micro_batch_size: 2 #
num_epochs: 4 #
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-4

train_on_inputs: true
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 5 #
evals_per_epoch: 4 #
eval_table_size:
saves_per_epoch: 1
save_total_limit: 10
save_steps:
debug:
deepspeed: deepspeed_configs/zero3_bf16.json
weight_decay: 0.0 #
fsdp:
fsdp_config:

#
adapter: lora
lora_r: 64
lora_alpha: 64
lora_dropout: 0.0
lora_target_modules:
  - q_proj
  - v_proj
  - k_proj
  - o_proj
  - gate_proj
  - down_proj
  - up_proj

#lora_mlp_kernel: true
#lora_qkv_kernel: true
#lora_o_kernel: true

eval_sample_packing: False #

jason-2k-1

This model is a fine-tuned version of yentinglin/Llama-3-Taiwan-8B-Instruct on the jasonhuang3/taitung_gemini_v1v3_messages dataset. It achieves the following results on the evaluation set:

• Loss: 1.6743

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• total_eval_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 5
• num_epochs: 4.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.5468	0.0690	1	2.7452
2.2849	0.2759	4	2.2667
1.8185	0.5517	8	1.9545
1.705	0.8276	12	1.8034
1.5654	1.0690	16	1.7497
1.4897	1.3448	20	1.7219
1.4709	1.6207	24	1.7026
1.4316	1.8966	28	1.6768
1.3117	2.1379	32	1.6765
1.2819	2.4138	36	1.6759
1.2825	2.6897	40	1.6815
1.2934	2.9655	44	1.6787
1.2153	3.2069	48	1.6747
1.2287	3.4828	52	1.6745
1.1894	3.7586	56	1.6743

Framework versions

• PEFT 0.14.0
• Transformers 4.48.3
• Pytorch 2.4.0+cu121
• Datasets 3.2.0
• Tokenizers 0.21.1