dbhaskarganesh/TinyLlama1BTeluguFineTuneModel

🪔 Adapting TinyLlama-1B for Telugu

Model Description

This model is a fine-tuned version of TinyLlama-1.1B-Chat trained on a custom Telugu TinyStories dataset.
It was developed as part of CISC7021 – Applied Natural Language Processing, University of Macau to explore low-resource language adaptation of lightweight LLMs.

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Language: Telugu (te)
• Model type: Decoder-only transformer (LLaMA-style)
• Training objective: Continual pre-training on Telugu corpus for better language modeling and text generation

---

Intended Uses

• Text generation in Telugu (stories, descriptions, prompts).
• Research on low-resource language adaptation.
• Educational purposes for understanding continual pre-training with Hugging Face & PyTorch.

⚠️ Not recommended for production or sensitive applications (e.g., medical, financial, or legal use).

---

Training Data

• Dataset: dbhaskarganesh/TeluguTinnystories
• Approx. size: 11,500 tokens
• Derived from TinyStories-style narratives adapted into Telugu.

---

Training Procedure

• Base model: TinyLlama-1.1B-Chat
• Framework: PyTorch + Hugging Face Transformers
• GPU: Google Colab (free tier) and NVIDIA RTX 4090 24GB
• Settings:
  • Batch size = 3
  • Max sequence length = 512
  • Learning rate = 2e-5
  • Optimizer = AdamW
  • Decoding examples: temperature = 0.6, max_new_tokens = 850

---

Evaluation

• Metrics: accuracy, perplexity

• Perplexity results:

  • English test set: ~4.92
  • Telugu test set: ~2.42

• Qualitative evaluation:
  Model generates coherent Telugu sentences, though with occasional repetition or off-topic responses.

---

Limitations

• Small model (1B parameters) → not competitive with large LLMs.
• Limited dataset coverage → may not generalize well.
• Possible biases and hallucinations due to training data.

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "dbhaskarganesh/tinyllama-telugu"  # replace with your repo path
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "ఒక చిన్న కథను వ్రాయండి."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))