Llama3.1-8B-nep_civillaw_finetuned

This repository contains a fine-tuned version of the Llama3.1-8B model, specifically adapted for understanding and generating responses related to Nepal Civil Law. The model has been fine-tuned using a custom-generated dataset derived from the Nepal Civil Code.

Model Description

Llama3.1-8B-nep_civillaw_finetuned is a specialized language model designed to provide accurate and contextually relevant information on the Nepal Civil Law. It builds upon the powerful Llama3.1-8B architecture, leveraging its extensive pre-training and adapting it to the nuances of legal texts specific to Nepal.

This model is ideal for applications requiring:

Question Answering on Nepal Civil Law topics.
Summarization of legal provisions from the Nepal Civil Code.
Generating explanations or interpretations of specific legal articles.

Usage

To use this model, you can load it directly from Hugging Face Transformers.

First, ensure you have the transformers and accelerate libraries installed:

!pip install transformers accelerate bitsandbytes -U

Then, you can load the model for inference:

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

# Load the tokenizer and model
model_id = "chhatramani/Llama3.1-8B-nep_civillaw_finetuned"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,  # Use bfloat16 for Gemma models if your GPU supports it
    device_map="auto"
)

# Create a pipeline for text generation
# Ensure the chat template is applied for proper instruction following
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,  # Adjust max_new_tokens as needed for answer length
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
    top_k=50,
)

Use Alpa syle to chat with model with text streaming

alpaca_prompt = """Below is an instruction that describes a task, coupled with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Response:
{}"""

inputs = tokenizer(
[
    alpaca_prompt.format(
        "What are the Duty to operate and manage trust property properly", # instruction
        "", # input
        "", # output - leave this blank for generation!
    )
], return_tensors = "pt").to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 500)

Fine-tuning Details

The Llama3.1-8B-nep_civillaw_finetuned model was fine-tuned from the base Llama3.1-8B model using a Supervised Fine-Tuning (SFT) approach with the trl library's SFTTrainer.

Base Model: Llama3.1-8B
Fine-tuning Dataset: A custom dataset of 2712 Question-Answer pairs related to Nepal Civil Law, generated from the civil_code_en.pdf document.
Fine-tuning Method: LoRA (Low-Rank Adaptation) was used to efficiently adapt the model to the new domain.
Average Training Loss: 0.863

Hyperparameters Used for Fine-tuning

The following SFTConfig parameters were used during the training process:

Parameter	Value
`dataset_text_field`	`"text"`
`per_device_train_batch_size`	4
`gradient_accumulation_steps`	4
`warmup_ratio`	0.03
`num_train_epochs`	3
`learning_rate`	2e-4
`logging_steps`	10
`optim`	`"adamw_8bit"`
`weight_decay`	0.01
`lr_scheduler_type`	`"cosine"`
`seed`	3407
`report_to`	`"none"`
`fp16`	True

Explanation of Key Parameters:

dataset_text_field: Specifies the column in the training dataset that contains the formatted text for training (instruction + response).
per_device_train_batch_size: The batch size per GPU during training. Set to 4.
gradient_accumulation_steps: Gradients are accumulated over 4 steps, leading to an effective batch size of 16 (4 * 4) per GPU.
warmup_ratio: The proportion of training steps over which the learning rate linearly increases from 0. Set to 0.03 (3% of total steps).
num_train_epochs: The total number of training epochs to perform. Set to 3 to ensure the model sees the entire 2712-row dataset multiple times.
learning_rate: The initial learning rate for the optimizer. A value of 2e-4 is common for LoRA fine-tuning.
logging_steps: Logs training metrics every 10 steps.
optim: The optimizer used, adamw_8bit for memory-efficient AdamW.
weight_decay: L2 regularization applied to the weights to prevent overfitting.
lr_scheduler_type: The learning rate scheduler type, cosine for a cosine annealing schedule.
seed: Random seed for reproducibility.
report_to: Specifies integration with experiment tracking platforms (set to none here).
fp16: Enables mixed-precision training using float16, which speeds up training and reduces memory usage.

Limitations and Bias

As with all large language models, this fine-tuned model may exhibit certain limitations and biases:

Domain Specificity: While fine-tuned on Nepal Civil Law, its knowledge is limited to the content of the training data. It may not generalize well to other legal domains or new information outside its training scope.
Hallucinations: The model might occasionally generate plausible but incorrect or non-existent legal information. Always verify critical outputs with official legal sources.
Bias from Data: Any biases present in the original Nepal Civil Code or introduced during the dataset generation process may be reflected in the model's responses.
Language Nuances: Primarily trained on English translations of the Civil Code, it might lack deep understanding of specific Nepali legal terminology or cultural contexts if not explicitly present in the training data.

chhatramani
/

Llama3.1-8B-nep_civillaw_finetuned