Llama-PaperSummarization-LoRA

Model Details

This is a LoRA fine-tuned adapter built on meta-llama/Llama-3.2-1B-Instruct. It is designed for scientific paper summarization tasks and leverages Low-Rank Adaptation (LoRA) to enhance model performance efficiently while maintaining a low computational overhead.

Performance comparison

Model	ROUGE-1	ROUGE-2	ROUGE-3	ROUGE-L
Llama-3.2-1B-Instruct	36.69	7.47	1.95	19.36
Llama-PaperSummarization-LoRA	41.56	11.31	2.67	21.86

The model was evaluated on a 6K-sample test set using ROUGE scores with beam search (beam size = 4).

How to load

from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
peft_model_id = "gabe-zhang/Llama-PaperSummarization-LoRA"
model = PeftModel.from_pretrained(base_model, peft_model_id)
model.merge_and_unload()

Dataset

The model was fine-tuned on the armanc/scientific_papers dataset. Below are the details of the dataset splits:

• Training Set: 20K samples
• Validation Set: 6K samples
• Test Set: 6K samples

LoRA Configuration

• Trainable Parameters: 850K (~7% of base model parameters)
• Context Length: 10K tokens
• Rank: 8
• Target Modules: Query and Value matrices
• Optimization Settings:
  • Gradient Accumulation: 4 steps
  • Training Steps: 5K

Training Setup

• Hardware: NVIDIA RTX A6000 GPU
• Evaluation Frequency: Every 20 steps
• Training Duration: 28 hours
• Training Scripts: gabe-zhang/paper2summary

License

This repository contains a LoRA fine-tuned adapter derived from the Llama 3.2 model.

• Llama 3.2 Materials: Governed by the Llama 3.2 Community License.
• All other content: Licensed under the MIT License.

Attribution

• The model prominently incorporates Llama 3.2 as its base.
• "Built with Llama" is displayed as required by the Llama 3.2 Community License.