This model is part of LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation.
Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference.
LoRI (LoRA with Reduced Interference) is a simple yet effective variant of Low-Rank Adaptation (LoRA) for fine-tuning Large Language Models (LLMs). It improves efficiency and performance by freezing projection matrices (A) as random projections and sparsifying matrices (B) using task-specific masks. This design significantly reduces trainable parameters while maintaining strong task performance. LoRI also minimizes cross-task interference in adapter merging and supports continual learning by mitigating catastrophic forgetting through sparsity.
meta-llama/Meta-Llama-3-8BLoRI adapters can be directly loaded with a compatible base LLM (e.g., meta-llama/Meta-Llama-3-8B) using the peft library. This model, LoRI-S_code_llama3_rank_32, is specifically fine-tuned for code generation tasks. LoRI is designed for efficient fine-tuning across various tasks including natural language understanding, mathematical reasoning, code generation, and safety alignment, and supports effective adapter merging and continual learning.
LoRI can be integrated into larger AI systems and applications requiring efficient multi-task learning or continual adaptation of LLMs. Its reduced cross-task interference makes it suitable for complex scenarios where multiple capabilities are needed from a single adapter.
This model is designed for text-based generation and understanding tasks, specifically in the context of code generation. Using it for tasks outside of its trained modalities, for applications requiring very high precision in domains not covered by its training data, or for generating harmful content is not recommended.
As with any model finetuned from a large language model, this adapter may inherit biases present in its underlying training data (meta-llama/Meta-Llama-3-8B) and the specific finetuning datasets. While the LoRI paper mentions "safety alignment tasks", comprehensive evaluation for all potential risks is recommended.
Users should be aware of the inherent biases and limitations of large language models. It is recommended to perform further evaluation in specific deployment contexts and to implement appropriate safeguards, especially in sensitive applications.
Use the code below to get started with the model.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load the base model (e.g., Llama-3-8B)
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
# Load the LoRI adapter on top of the base model
# This example loads the LoRI-S adapter for code generation, rank 32
adapter = PeftModel.from_pretrained(base_model, "tomg-group-umd/LoRI-S_code_llama3_rank_32")
# Load the tokenizer for the base model
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
# Example usage (for text generation with the adapted model)
# from transformers import pipeline
# generator = pipeline("text-generation", model=adapter, tokenizer=tokenizer)
# print(generator("def fibonacci(n):", max_new_tokens=50))
LoRI adapters are trained on various datasets relevant to different tasks. This specific adapter (LoRI-S_code_llama3_rank_32) was trained for code generation using the CodeAlpaca dataset. Other datasets mentioned in the paper/repo include GSM8K (mathematical reasoning) and SaferPaca (safety alignment).
LoRI employs a two-stage training procedure:
A are frozen as random projections, and matrices B are trained to discover task-specific masks.Training is implemented using Fully Sharded Data Parallel (FSDP) to support multi-GPU environments.
LoRI uses up to 95% fewer trainable parameters than traditional LoRA while maintaining strong task performance.
For code generation tasks, evaluation was performed on the HumanEval benchmark. LoRI was also evaluated across natural language understanding, mathematical reasoning, and safety alignment tasks on various datasets.
No specific disaggregated factors (e.g., subpopulations) were explicitly mentioned for evaluation in the provided context.
For code generation, the primary metric is typically pass@k, which measures the percentage of generated code samples that pass unit tests for a given problem.
LoRI consistently outperforms full fine-tuning and existing PEFT methods, while using significantly fewer trainable parameters. It also demonstrates reduced cross-task interference in adapter merging and improved resistance to catastrophic forgetting in continual learning. For detailed quantitative results, please refer to the paper.
LoRI modifies the standard LoRA architecture by freezing the projection matrices A as random projections and sparsifying the matrices B using task-specific masks. This design aims to achieve substantial reduction in trainable parameters, minimize cross-task interference between different adaptations, and support continual learning by mitigating catastrophic forgetting.
Training and inference are supported on multi-GPU environments, leveraging technologies like FSDP.
The project builds on PyTorch, transformers, and peft.
If you use LoRI in your work, please cite:
BibTeX:
@article{zhang2025lori,
title={LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation},
author={Zhang, Juzheng and You, Jiacheng and Panda, Ashwinee and Goldstein, Tom},
journal={arXiv preprint arXiv:2504.07448},
year={2025}
}
APA: Zhang, J., You, J., Panda, A., & Goldstein, T. (2025). LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation. arXiv preprint arXiv:2504.07448.
Niels Drost (huggingface.co/nielsr)
Base model
meta-llama/Meta-Llama-3-8B