Gemma-3-4b-4bit-nepal_civil_law_finetuned

This is a fine-tuned version of the Gemma 3 4B model [2], specifically adapted for Nepal civil law question answering tasks using the dataset: chhatramani/nepal_civilcode_en [1]. The fine-tuning process was aimed at enhancing the model's capabilities in understanding and responding to legal questions based on Nepal’s civil code.

Model Description

Gemma 3 4B is a lightweight, state-of-the-art open model built from the same research and technology as Gemini 2.0, offering high-quality results across various tasks [2]. This particular model has been further refined through fine-tuning to better understand and generate responses relevant to the civil laws of Nepal, making it suitable for legal QA and instruction-following tasks in this domain.

The base model was fine-tuned using 4-bit quantization and QLoRA techniques, enabling efficient training and inference while maintaining performance [7].

Dataset Used

The model was trained on the chhatramani/nepal_civilcode_en dataset [1], which contains approximately 2.4K question-answer pairs extracted from Nepal’s civil code. Each sample consists of a legal question and its corresponding answer derived from official civil law documentation.

Training Details

The model was fine-tuned using a combination of Hugging Face Transformers and Unsloth for efficient low-Rank Adaptation (LoRA) and 4-bit training [7]. Below are the key training parameters used:

Training Setup

Base Model: google/gemma-3-4b-it
Dataset: chhatramani/nepal_civilcode_en [1]
Training Framework: Hugging Face Transformers + TRL + Unsloth
Quantization: 4-bit (LLM.int4)
Adapter Method: QLoRA
LoRA Parameters:
- Rank (r): 64
- Dropout: 0.1
Batch Size per Device: 2
Gradient Accumulation Steps: 4
Total Batch Size: 8
Learning Rate: 2e-4
Optimizer: AdamW_8bit
Weight Decay: 0.01
Number of Epochs: 3
Max Sequence Length: 512
Warmup Steps: 100
LR Scheduler: Linear with warmup

How to Use

To utilize this model, you can load it via the Hugging Face Transformers library. Here’s a simple example:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("chhatramani/gemma-3-4b-4bit-nepal_civil_law_finetuned")
model = AutoModelForCausalLM.from_pretrained("chhatramani/gemma-3-4b-4bit-nepal_civil_law_finetuned")

input_text = "What is the basic price according to the hire-purchase agreement?"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---
license: apache-2.0
---