hilaryc112/LoRA-GPT2-Project

GPT2-Large LoRA Fine-tuned for Natural Language Inference

This model is a LoRA (Low-Rank Adaptation) fine-tuned version of GPT2-large for Natural Language Inference (NLI) on the MultiNLI dataset.

Model Details

• Base Model: GPT2-large (774M parameters)
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Trainable Parameters: ~2.3M (0.3% of total parameters)
• Dataset: MultiNLI (50K training samples)
• Task: Natural Language Inference (3-class classification)

Performance

• Test Accuracy (Matched): ~79.22%
• Test Accuracy (Mismatched): ~80.38%
• Training Method: Parameter-efficient fine-tuning with LoRA
• Hardware: Trained on 36G vGPU

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained("gpt2-large")
base_model = AutoModelForCausalLM.from_pretrained("gpt2-large")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "hilaryc112/LoRA-GPT2-Project")

# Format input
premise = "A person is outdoors, on a horse."
hypothesis = "A person is at a diner, ordering an omelette."
input_text = f"Premise: {premise}\nHypothesis: {hypothesis}\nRelationship:"

# Tokenize and generate
inputs = tokenizer(input_text, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=10, pad_token_id=tokenizer.eos_token_id)
    
prediction = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True).strip()
print(f"Prediction: {prediction}")  # Should output: contradiction, neutral, or entailment

Training Configuration

{ "model_name": "gpt2-large", "max_length": 512, "lora_r": 16, "lora_alpha": 32, "lora_dropout": 0.1, "target_modules": [ "c_attn", "c_proj", "c_fc" ], "num_epochs": 3, "train_batch_size": 4, "eval_batch_size": 12, "gradient_accumulation_steps": 6, "learning_rate": 0.0002, "weight_decay": 0.01, "max_grad_norm": 1.0, "use_fp16": true, "gradient_checkpointing": true, "logging_steps": 100, "eval_steps": 500, "save_steps": 500, "save_total_limit": 3, "early_stopping_patience": 5, "data_dir": "./processed_data", "output_dir": "./gpt2_lora_multinli", "seed": 42, "use_wandb": false, "_comments": { "effective_batch_size": "6 * 6 = 36 (optimized for 36G vGPU)", "memory_optimization": "FP16 + gradient checkpointing enabled", "lora_config": "Rank 16 with alpha 32 for good performance/efficiency balance", "target_modules": "GPT2 attention and MLP layers for comprehensive adaptation", "training_data": "Uses 50K samples from MultiNLI training set (configured in preprocessing)", "evaluation_data": "Uses local dev files for matched/mismatched evaluation", "training_adjustments": "Reduced epochs to 2 and LR to 1e-4 for better training with real data", "eval_frequency": "Less frequent evaluation (every 500 steps) due to larger dataset" } }

Dataset Format

The model was trained on text-to-text format:

Premise: [premise text]
Hypothesis: [hypothesis text]
Relationship: [entailment/neutral/contradiction]

Files

• adapter_config.json: LoRA adapter configuration
• adapter_model.safetensors: LoRA adapter weights
• training_config.json: Training hyperparameters and settings

Citation

If you use this model, please cite:

@misc{gpt2-lora-multinli,
  title={GPT2-Large LoRA Fine-tuned for Natural Language Inference},
  author={HilaryKChen},
  year={2024},
  howpublished={\url{https://huggingface.co/hilaryc112/LoRA-GPT2-Project}}
}

License

This model is released under the MIT License.