🇮🇳 Gemma 4 12B Hindi LoRA: experimental v1
🧪 Experimental v1, not production-ready yet. It works: answers stay in pure Hindi with no English words in brackets, and it follows requested list counts. It also has known issues (below), and a v2 is in progress. In my 25-prompt evaluation, base Gemma 4 12B with a short Hindi system prompt scored higher, so use that setup if you need the best quality today.
Part of my weekly 🇮🇳 Hindi LLM Series.
TL;DR
v1 of my Gemma 4 12B Hindi fine-tune, trained on 12.5k Hindi instruction pairs. It meets its main goals: no English words in brackets, and requested counts followed (7/7). Because much of the training data is machine-translated, it also picked up some translation habits: extra length, a few artifacts, and occasional factual slips. A 4-line system prompt on the base model currently scores higher (24 vs 20 fully correct out of 25). v2 will fix this by training on the model's own clean Hindi answers (self-distillation).
Results (25 Hindi prompts)
| Base + system prompt | This LoRA (temp 0.3) | |
|---|---|---|
| Manual review: correct / partly wrong / wrong | 24 / 1 / 0 | 20 / 3 / 2 |
| English words in brackets | 0 | 0 |
| Requested list counts met | 7/7 | 7/7 |
| Visarga used as a colon ("हैः") | 2 | 45 |
| Average answer length | 219 chars | 628 chars |
| Chatbot filler ("नमस्ते! मुझे खुशी है…") | 0 | 5 answers |
- 💬 Try it and compare all answers (Chat + Evaluation tabs): pankajpandey-dev/gemma-4-12b-hindi-chat
- 📂 All 25 answers + metrics: pankajpandey-dev/gemma-4-12b-hindi-eval25
Known issues (fixing in v2)
Most of the training data is machine-translated into Hindi, and the 12B base already writes better Hindi than that data. So v1 picked up some of the data's habits:
- Translation artifacts: visarga (ः) instead of colons; "order" translated as "आदेश" (a command); lists missing their "1."
- Western content in Hindi: advised wearing a sweater during the Indian monsoon, missing mosquitoes and water-borne disease
- Factual slips: expanded UPI as "Universal Payments Interface" (it is Unified Payments Interface)
- Verbosity: about 3× longer than needed; 4 of 25 answers hit the token limit
- Foreign-script tokens at temperature 1.0: Russian and Chinese tokens mid-sentence. These disappear at temperature 0.7 and below
The same data improved my smaller models (0.6B–4B). At 12B, the base model is already stronger than the data.
For the best quality today
Base Gemma 4 12B with this system prompt:
हमेशा केवल शुद्ध हिंदी में उत्तर दें। कोष्ठक में अंग्रेज़ी शब्द न लिखें।
जितना पूछा जाए, ठीक उतना ही दें। Markdown का उपयोग न करें।
गणित या तर्क के प्रश्नों में अंतिम उत्तर से पहले एक-दो पंक्तियों में हल दिखाएँ।
The last line fixes the base model being too terse on maths. It was added after the evaluation.
Using this LoRA
The adapter is in the last-checkpoint folder.
from huggingface_hub import snapshot_download
from unsloth import FastModel
snapshot_download("pankajpandey-dev/gemma-4-12b-hindi-instruct-checkpoints",
allow_patterns=["last-checkpoint/*"], local_dir="ckpt")
model, tokenizer = FastModel.from_pretrained("ckpt/last-checkpoint",
max_seq_length=2048, load_in_4bit=True)
Recommended settings: temperature 0.3 or lower (at 1.0 it can produce foreign-script tokens), max new tokens around 400. Review outputs before using them in anything user-facing.
Training details
| Base | unsloth/gemma-4-12b-it (Gemma 4 12B instruct), 4-bit QLoRA |
| Data | AI4Bharat indic-instruct-data-v0.1: anudesh Hindi (7,577) + 4,000 dolly Hindi; Cohere Aya Hindi (1,153). 12,478 examples after removing those over 1,024 tokens |
| LoRA | r=16, alpha=16, dropout 0, attention + MLP layers, vision layers frozen; 65.6M trainable parameters (0.55%) |
| Loss | Response-only (assistant tokens only), Gemma 4 chat template with empty thought channel |
| Optimiser | AdamW 8-bit, LR 1e-4, cosine, 20 warmup steps, weight decay 0.01 |
| Batch | 1 × 8 gradient accumulation, max length 1,024 |
| Schedule | 1 epoch, 1,560 steps |
| Precision | float32 compute (Gemma 4 can't train in fp16; the T4 has no bf16) |
| Hardware | Free Kaggle T4, 9 h 41 min |
| Final loss | 2.60 → ~1.2 plateau; 1.23 average over the run |
What's next
v2 with self-distillation: train on the 12B's own clean Hindi answers (base + system prompt, filtered) instead of translated data, so the pure-Hindi behaviour is built in without losing quality.
Credits
Built with Unsloth and TRL. Data by AI4Bharat and Cohere Aya. 🙏
- Downloads last month
- 23