See axolotl config

axolotl version: 0.12.0.dev0

base_model: minpeter/tiny-ko-124m-base-muon

hub_model_id: minpeter/tiny-ko-124m-sft-muon
output_dir: ./outputs/tiny-ko-124m-sft-muon
wandb_project: "axolotl"
wandb_entity: "kasfiekfs-e"

model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

strict: false

chat_template: chatml
datasets:
  - path: HuggingFaceTB/smol-smoltalk
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: trillionlabs/multisystem-curated
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: allenai/tulu-3-sft-personas-instruction-following
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: lemon-mint/smol-koreantalk
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: lemon-mint/Korean-FineTome-100k
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: heegyu/open-korean-instructions-v20231020
    type: chat_template
    split: train
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value
    roles:
      user: ["human", "user"]
      assistant: ["gpt", "assistant", "bot"]
      system: ["system", "input"]

  - path: coastral/korean-writing-style-instruct
    type: chat_template
    split: train
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value

  - path: devngho/korean-instruction-mix
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: from
      content: value

dataset_prepared_path: last_run_prepared
val_set_size: 0.001
save_safetensors: true
sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
use_pose: true
pose_max_context_len: 65536

overrides_of_model_config:
  rope_theta: 10000.0
  max_position_embeddings: 65536

gradient_accumulation_steps: 8
micro_batch_size: 32
num_epochs: 1
optimizer: muon
lr_scheduler: cosine
learning_rate: 3e-4

train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: true

gradient_checkpointing: false
gradient_checkpointing_kwargs:
  use_reentrant: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
sdp_attention:
s2_attention:

save_steps: 200
warmup_steps: 20
eval_steps: 200
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:

tiny-ko-124m-sft-muon

This model is a fine-tuned version of minpeter/tiny-ko-124m-base-muon on the HuggingFaceTB/smol-smoltalk, the trillionlabs/multisystem-curated, the allenai/tulu-3-sft-personas-instruction-following, the lemon-mint/smol-koreantalk, the lemon-mint/Korean-FineTome-100k, the heegyu/open-korean-instructions-v20231020, the coastral/korean-writing-style-instruct and the devngho/korean-instruction-mix datasets. It achieves the following results on the evaluation set:

• Loss: 1.6461

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 1024
• total_eval_batch_size: 128
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 20
• training_steps: 6865

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0	0	2.4581
1.8892	0.1165	200	1.9059
1.802	0.2331	400	1.8333
1.7906	0.3496	600	1.7918
1.7761	0.4661	800	1.7638
1.7145	0.5827	1000	1.7423
1.7114	0.6992	1200	1.7255
1.6798	0.8157	1400	1.7123
1.6722	0.9323	1600	1.7006
1.6821	1.0484	1800	1.6928
1.6414	1.1649	2000	1.6864
1.6473	1.2814	2200	1.6794
1.6202	1.3980	2400	1.6729
1.6141	1.5145	2600	1.6689
1.6415	1.6310	2800	1.6645
1.6165	1.7476	3000	1.6603
1.6292	1.8641	3200	1.6573
1.6277	1.9806	3400	1.6541
1.6033	2.0967	3600	1.6537
1.6432	2.2133	3800	1.6517
1.602	2.3298	4000	1.6505
1.6435	2.4463	4200	1.6493
1.5941	2.5629	4400	1.6481
1.594	2.6794	4600	1.6473
1.5986	2.7959	4800	1.6468
1.586	2.9125	5000	1.6464
1.6146	3.0286	5200	1.6462
1.5985	3.1451	5400	1.6462
1.574	3.2616	5600	1.6462
1.5823	3.3782	5800	1.6460
1.597	3.4947	6000	1.6460
1.5859	3.6112	6200	1.6460
1.5769	3.7277	6400	1.6459
1.572	3.8443	6600	1.6459
1.6111	3.9608	6800	1.6461

Framework versions

• Transformers 4.53.1
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.1