llm-jp-3-13b-it

Model Description

This is a fine-tuned version of llm-jp/llm-jp-3-13b trained using Unsloth for improved performance.

Training Details

The model was trained using:

• Unsloth for 2x faster training
• Hugging Face's TRL library for supervised fine-tuning
• QLoRA 4-bit quantization
• Maximum sequence length of 512 tokens

Training Configuration

• Base Model: llm-jp/llm-jp-3-13b
• Training Type: Supervised Fine-tuning (SFT)
• LoRA Configuration:
  • Rank: 32
  • Alpha: 32
  • Dropout: 0.05
  • Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("your-username/llm-jp-3-13b-it")
tokenizer = AutoTokenizer.from_pretrained("your-username/llm-jp-3-13b-it")

Training Framework

This model was trained using Unsloth, which provided 2x faster training speed compared to traditional methods.

License

This model is licensed under Apache 2.0.