This model is part of LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation.
LoRI (LoRA with Reduced Interference) is a simple yet effective parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs). It addresses common issues like notable overhead and parameter interference in multi-task scenarios by freezing the projection matrices A as random projections and sparsifying the matrices B using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance, minimizing cross-task interference in adapter merging, and supporting continual learning by mitigating catastrophic forgetting.
LoRI-S_code_llama3_rank_64 is a specific LoRI-S (Sparse) adapter trained for code generation tasks. It is built upon the meta-llama/Meta-Llama-3-8B base model with an adapter rank of 64. The LoRI approach has been demonstrated to outperform full fine-tuning and existing PEFT methods, using up to 95% fewer trainable parameters than standard LoRA. This model is part of a broader set of LoRI adapters that cover natural language understanding, mathematical reasoning, code generation, and safety alignment tasks.
meta-llama/Meta-Llama-3-8BThis model is intended to be used as a PEFT adapter to efficiently fine-tune or enhance the meta-llama/Meta-Llama-3-8B base model specifically for code generation tasks. It should be loaded using the Hugging Face PEFT library on top of the base LLM.
LoRI adapters are particularly designed for multi-task scenarios and continual learning, where they enable effective adapter merging and reduce cross-task interference. This model can be combined with other LoRI adapters for different tasks to build more robust multi-task systems.
This model is not intended for standalone use; it strictly requires the meta-llama/Meta-Llama-3-8B as its base model. Like all large language models, it may generate biased, harmful, or factually incorrect content, and should not be used in critical applications without thorough evaluation and additional safeguards.
While LoRI aims to reduce interference and parameter overhead, the model may still inherit biases present in its pre-training or fine-tuning data (e.g., CodeAlpaca, Meta-Llama-3-8B's pre-training data). Potential risks and limitations include:
Users (both direct and downstream) should be aware of these potential issues and implement appropriate validation and filtering mechanisms for the model's outputs. It is recommended to apply responsible AI practices and conduct task-specific evaluations.
Use the code below to get started with the model:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# 1. Load the base model
base_model_name = "meta-llama/Meta-Llama-3-8B"
base_model = AutoModelForCausalLM.from_pretrained(
base_model_name,
torch_dtype=torch.bfloat16, # Llama 3 models often use bfloat16
device_map="auto", # Load model onto available devices (GPU if available)
low_cpu_mem_usage=True # Optimize CPU memory usage
)
# 2. Load the LoRI adapter
# Replace "tomg-group-umd/LoRI-S_code_llama3_rank_64" with the correct model ID if different
adapter_model_id = "tomg-group-umd/LoRI-S_code_llama3_rank_64"
adapter_model = PeftModel.from_pretrained(base_model, adapter_model_id)
# 3. Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
# Set pad_token if not already set, crucial for batching/generation
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token # Or another appropriate token
# 4. Set the model to evaluation mode
adapter_model.eval()
# 5. Prepare your input prompt for code generation
prompt = '''
def bubble_sort(arr):
n = len(arr)
for i in range(n - 1):
for j in range(0, n - i - 1):
if arr[j] > arr[j + 1]:
arr[j], arr[j + 1] = arr[j + 1], arr[j]
return arr
# Write a docstring for the function above, describing its purpose and parameters.
'''
# Encode the prompt and move to the model's device
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(adapter_model.device)
# 6. Generate output
with torch.no_grad():
output_ids = adapter_model.generate(
input_ids,
max_new_tokens=100,
do_sample=True, # Sample outputs
temperature=0.01, # Low temperature for less randomness, more deterministic code
top_p=0.95, # Nucleus sampling
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
# Decode and print the generated text
generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(generated_text)
# Optional: Merge adapter weights into the base model for easier deployment
# merged_model = adapter_model.merge_and_unload()
# merged_model.save_pretrained("path/to/merged-lori-model")
This LoRI-S_code_llama3_rank_64 adapter was specifically fine-tuned on the CodeAlpaca dataset for code generation tasks. The LoRI paper also describes experiments on:
LoRI training typically involves a two-stage process, implemented using Fully Sharded Data Parallel (FSDP) for efficient multi-GPU training:
A are frozen as random projections, and the B matrices are trained densely.LoRI-D training, sparse masks are extracted from the learned B matrices. For LoRI-S models, a high sparsity level (e.g., 90%) is typically applied.LoRI-S_code_llama3_rank_64, is the result of this sparsified training phase.meta-llama/Meta-Llama-3-8Br): 64lora_alpha): 128lora_dropout): 0.05LoRI models have been extensively evaluated across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks. Experiments demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using significantly fewer trainable parameters (up to 95% less than LoRA). In multi-task settings, LoRI enables effective adapter merging and continual learning with reduced cross-task interference.
For detailed quantitative results, specific metrics (e.g., HumanEval for code generation, SuperGLUE for NLU, GSM8K for math), and comprehensive comparisons against baselines, please refer to the official paper.
LoRI introduces a modification to the standard LoRA architecture where the projection matrices A are fixed as random projections, and the matrices B are sparsified using task-specific masks. This design is aimed at reducing cross-task interference in multi-task learning and mitigating catastrophic forgetting in continual learning scenarios.
If you use LoRI in your work, please cite:
@article{zhang2025lori,
title={LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation},
author={Zhang, Juzheng and You, Jiacheng and Panda, Ashwinee and Goldstein, Tom},
journal={arXiv preprint arXiv:2504.07448},
year={2025}
}
Niels Rogge (Hugging Face Community Science Team)
Base model
meta-llama/Meta-Llama-3-8B