Training Details
I only used LoRA. I adjusted some parameters in the TrainingArguments, and this is the best configuration I tried: training_arguments = TrainingArguments( output_dir=new_model_id, per_device_train_batch_size=2, gradient_accumulation_steps=10, optim="paged_adamw_32bit", num_train_epochs=40, logging_strategy="steps", logging_steps=20, warmup_steps=20, save_steps=10, save_total_limit = 40, max_steps = 300, learning_rate=5e-5, fp16=False, bf16=True, seed = 3407, group_by_length=True, no_cuda=False, report_to=None, )
Training Data
https://liat-aip.sakura.ne.jp/wp/llmのための日本語インストラクションデータ作成/llmのための日本語インストラクションデータ-公開/ [More Information Needed]
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API was unable to determine this model’s pipeline type.
Model tree for yurrika/llm-jp-3-13b-finetune2300
Base model
llm-jp/llm-jp-3-13b