---
license: apache-2.0
datasets:
- dbhaskarganesh/TeluguTinnystories
language:
- te
metrics:
- accuracy
- perplexity
base_model:
- TinyLlama/TinyLlama-1.1B-Chat-v1.0
new_version: TinyLlama/TinyLlama-1.1B-Chat-v1.0
pipeline_tag: text-classification
tags:
- TinnyStories
- NLP
---

# 🪔 Adapting TinyLlama-1B for Telugu

## Model Description
This model is a **fine-tuned version of TinyLlama-1.1B-Chat** trained on a custom **Telugu TinyStories dataset**.  
It was developed as part of **CISC7021 – Applied Natural Language Processing, University of Macau** to explore **low-resource language adaptation** of lightweight LLMs.

- **Base model:** [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)  
- **Language:** Telugu (te)  
- **Model type:** Decoder-only transformer (LLaMA-style)  
- **Training objective:** Continual pre-training on Telugu corpus for better language modeling and text generation  

---

## Intended Uses
- **Text generation** in Telugu (stories, descriptions, prompts).  
- **Research** on low-resource language adaptation.  
- **Educational purposes** for understanding continual pre-training with Hugging Face & PyTorch.  

⚠️ **Not recommended** for production or sensitive applications (e.g., medical, financial, or legal use).

---

## Training Data
- Dataset: [`dbhaskarganesh/TeluguTinnystories`](https://huggingface.co/datasets/dbhaskarganesh/TeluguTinnystories)  
- Approx. size: 11,500 tokens  
- Derived from TinyStories-style narratives adapted into Telugu.  

---

## Training Procedure
- **Base model:** TinyLlama-1.1B-Chat  
- **Framework:** PyTorch + Hugging Face Transformers  
- **GPU:** Google Colab (free tier) and NVIDIA RTX 4090 24GB  
- **Settings:**  
  - Batch size = 3  
  - Max sequence length = 512  
  - Learning rate = 2e-5  
  - Optimizer = AdamW  
  - Decoding examples: temperature = 0.6, max\_new\_tokens = 850  

---

## Evaluation
- **Metrics:** accuracy, perplexity  
- **Perplexity results:**  
  - English test set: ~4.92  
  - Telugu test set: ~2.42  

- **Qualitative evaluation:**  
  Model generates coherent Telugu sentences, though with occasional repetition or off-topic responses.  

---

## Limitations
- Small model (1B parameters) → not competitive with large LLMs.  
- Limited dataset coverage → may not generalize well.  
- Possible biases and hallucinations due to training data.  

---

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "dbhaskarganesh/tinyllama-telugu"  # replace with your repo path
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "ఒక చిన్న కథను వ్రాయండి."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))