Midm-KCulture-2.0-Base-Instruct

• This model is fine-tuned from KT/Midm-2.0-Base-Instruct on the 'Korean Culture Q&A Corpus' using the LoRA (Low-Rank Adaptation) methodology.

GitHub

Check out the full training code here.

Training Hyperparameters

Hyperparameter	Value
SFTConfig
torch_dtype	bfloat16
seed	42
epoch	3
per_device_train_batch_size	2
per_device_eval_batch_size	2
learning_rate	0.0002
lr_scheduler_type	"linear"
max_grad_norm	1.0
neftune_noise_alpha	None
gradient_accumulation_steps	1
gradient_checkpointing	False
max_seq_length	1024
LoraConfig
r	16
lora_alpha	16
lora_dropout	0.1
target_modules	["q_proj", "v_proj"]

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "jjae/Midm-KCulture-2.0-Base-Instruct"
model = AutoModelForCausalLM.from_pretrained(
      model_name,
      torch_dtype=torch.bfloat16,
      trust_remote_code=True,
      device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)