z-lab/sparselora-svd-estimator

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

• Paper
• GitHub
• Project Page

This repository contains the pre-computed SVD predictors for all 4 models used in our paper. By default, the required predictors are downloaded to your local machine when you first launch the training script.

We have precomputed the SVD predictors at Rank 8 for the following models, as used in the main paper:

• "NousResearch/Llama-2-7b-hf"
• "NousResearch/Llama-2-13b-hf"
• "NousResearch/Meta-Llama-3-8B-Instruct"
• "NousResearch/Meta-Llama-3.1-8B"