README.md

---
license: gemma
language: en
library_name: transformers
base_model: unsloth/gemma-3-1b-it
datasets:
- iamtarun/code_instructions_120k_alpaca
tags:
- gemma
- fine-tune
- instruction-following
- code
- alpaca
- unsloth
---

# Gemma-3-1B-Code-Alpaca-FT

This is a fine-tuned version of the `unsloth/gemma-3-1b-it` model. It was fine-tuned by [prathmesh-nik](https://huggingface.co/prathmesh-nik) on the [`iamtarun/code_instructions_120k_alpaca`](https://huggingface.co/datasets/iamtarun/code_instructions_120k_alpaca) dataset to enhance its ability to follow instructions related to code generation and general programming tasks.

## Model Details

### Technical Specifications

- **Context Window:** The model's configuration (`config.json`) specifies a theoretical maximum context of 32,768 tokens. However, it is based on `unsloth/gemma-3-1b-it`, which is optimized for an **8192 token** context. Furthermore, this fine-tune was performed with a `max_seq_length` of only **2048 tokens**. For best results, it is recommended to use this model with sequences close to the 2048 token training length.
- **Max New Tokens:** This is a parameter set during inference, not an intrinsic property of the model. The example code uses a default, but you can set this value higher or lower based on your needs (up to the context window limit).

### Intended Use

This model is designed for general-purpose instruction following, with a focus on coding assistance. It can be used for tasks such as:
- Generating code snippets in various programming languages.
- Answering questions about programming concepts.
- Acting as a backend for a development-focused chatbot.

### How to Use

You can use this model with the `transformers` library just like the base Gemma model.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Make sure you have logged in to huggingface-cli
# huggingface-cli login

model_id = "prathmesh_nik/Gemma-3-1B-Code-Alpaca-FT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

chat = [
    { "role": "user", "content": "Write a Python function to calculate the factorial of a number." },
]

prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)

outputs = model.generate(input_ids=inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
```

### Limitations and Bias

This model inherits the limitations and biases of the base `unsloth/gemma-3-1b-it` model and the training dataset.

- **Short Context Training:** Since the model was fine-tuned on sequences of only 2048 tokens, its performance on tasks requiring a very long context (i.e., approaching the 8192 token limit) may be degraded.
- **Quantization-Aware Training:** The adapters were trained on a 4-bit model. While the final model is full-precision, this process can introduce subtle differences compared to training in full precision.
- **General Limitations:** As a model fine-tuned on a specific dataset, it may not be as effective for general-purpose tasks and may not always provide correct answers. It may generate incorrect or insecure code and should not be used for mission-critical applications without human oversight.

## Fine-tuning Details

### Training Data

This model was fine-tuned on the [`iamtarun/code_instructions_120k_alpaca`](https://huggingface.co/datasets/iamtarun/code_instructions_120k_alpaca) dataset, which contains a large set of instruction-response pairs for coding tasks.

### Training Procedure

The model was fine-tuned using the LoRA (Low-Rank Adaptation) methodology. Key aspects of the training include:

- **Unsloth Optimization:** The training was performed using the [Unsloth](https://github.com/unslothai/unsloth) library, which enables significantly faster training and lower memory usage.
- **4-bit Quantization:** For maximum efficiency, the LoRA adapters were trained on a 4-bit quantized version of the base model. The final merged model is in full precision (`bfloat16`).
- **Parameter-Efficient Fine-Tuning (PEFT):** Only a small fraction of the model's parameters (the LoRA adapters) were trained, making the process highly efficient.

## Repository and Training Code

This model was fine-tuned by [prathmesh_nik](https://huggingface.co/prathmesh-nik). The complete source code for the fine-tuning process, including data preparation, training, and merging scripts, is available on GitHub.

- **GitHub Repository:** [https://github.com/prathmeshnik]

## Original Model Citation

Please cite the original Gemma 3 report if you use this model in your research:

```
@article{gemma_2025,
    title={Gemma 3},
    url={https://goo.gle/Gemma3Report},
    publisher={Kaggle},
    author={Gemma Team},
    year={2025}
}
```