---
language:
- en
license: mit
model-index:
- name: Nano-Llama
  results: []
tags:
- pytorch
- causal-lm
- text-generation
- fineweb
datasets:
- HuggingFaceFW/fineweb
library_name: transformers
---

# Nano-Llama

A compact 67M parameter LLaMA-2-style language model pretrained on FineWeb dataset.

## Model Details

- **Architecture**: LLaMA-2-style transformer
- **Parameters**: 678M
- **Training Data**: FineWeb dataset (~100M tokens)
- **Context Length**: 1024 tokens
- **Layers**: 6
- **Hidden Size**: 768
- **Attention Heads**: 12

## Training

- **Dataset**: FineWeb (web-crawled high-quality text)
- **Tokens Trained**: ~110M tokens
- **Training Time**: ~6 hours on RTX 3090
- **Optimizer**: AdamW
- **Learning Rate**: 1e-4

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("vishesh-t27/Nano-Llama")
model = AutoModelForCausalLM.from_pretrained("vishesh-t27/Nano-Llama")

model.eval()

# Test prompt
text = "The future of artificial intelligence is"
inputs = tokenizer(text, return_tensors="pt")

# Generate text
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.8,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

# Decode and print
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
```

## Limitations

- Small model size (67M parameters)
- Limited training data compared to larger models
- May generate repetitive or nonsensical text 

## License

MIT License