Onuii/DAMI-base-merge-0408

Uploaded model

• Developed by: Onuii

• License: apache-2.0

• Finetuned from model : kakaocorp/kanana-nano-2.1b-base

  --run_name "DAMI-base-merge-0408"
  --model_name "Onuii/DAMI-base-merge-0408"
  --dataset_name "Onuii/DAMI-Pretrain-Dataset-Tokenize8192"
  --max_seq_length 8192
  --dtype "bfloat16"
  --load_in_4bit True
  --r 128
  --lora_alpha 32
  --lora_dropout 0
  --use_rslora True
  --batch_size 32
  --gradient_accumulation_steps 4
  --learning_rate 1e-4
  --embedding_learning_rate 1e-5
  --warmup_ratio 0.1
  --num_train_epochs 3
  --optimizer "adamw_8bit"
  --weight_decay 0.01
  --lr_scheduler_type "linear"
  --output_dir "../outputs"
  --logging_dir "../logs"
  --logging_steps 1
  --save_steps 200
  --save_total_limit 2
  --report_to "wandb"
  --use_gradient_checkpointing False
  --save_strategy "steps"
  --random_seed 562

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.