Redgerd/llama3-roman-urdu-finetuned

Redgerd/llama3-roman-urdu-finetuned is a multilingual instruction-tuned LLaMA 3 model fine-tuned on a combination of Roman Urdu QA pairs and English examples from the Stanford Alpaca dataset.

This model is designed to enhance performance in low-resource, multilingual, and instruction-following tasks, especially involving Roman Urdu.

Model Details

• Base Model: Meta LLaMA 3
• Architecture: Decoder-only transformer
• Fine-tuned On: Custom Alpaca-style dataset
• Languages: Roman Urdu 🇵🇰 & English 🇬🇧
• Format: Instruction-tuning (compatible with Alpaca style)

Dataset Overview

A custom dataset with ~500 instruction-based examples in Roman Urdu and ~500 from Stanford Alpaca at from Redgerd/roman-urdu-alpaca-qa-mix

Training Setup

• Frameworks: transformers, unsloth
• Format: Instruction-based fine-tuning (instruction, input, output)
• Environment: A100 GPU with bfloat16 precision
• Checkpointing: Supported

Credits

Developed by Muhammad Salaar using:

• Stanford Alpaca Dataset
• GPT-4 for Roman Urdu generation
• LLaMA 3.1 as the base model

For feedback or collaboration, visit: github.com/Redgerd