mit-han-lab/svdquant-models

Quantization Library: DeepCompressor   Inference Engine: Nunchaku

[Paper]  [Code]  [Website]  [Blog]

SVDQuant seamlessly integrates with off-the-shelf LoRAs without requiring re-quantization. When applying LoRAs, it matches the image quality of the original 16-bit FLUX.1-dev.

Model Description

This reposity contains a converted LoRA collection for SVDQuant INT4 FLUX.1-dev. The LoRA style includes Realism, Ghibsky Illustration, Anime, Children Sketch, and Yarn Art.

Usage

Diffusers

Please follow the instructions in mit-han-lab/nunchaku to set up the environment. Then you can run the model with

import torch
from diffusers import FluxPipeline

from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel

transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit-han-lab/svdq-int4-flux.1-dev")
pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", transformer=transformer, torch_dtype=torch.bfloat16
).to("cuda")
image = pipeline("A cat holding a sign that says hello world", num_inference_steps=50, guidance_scale=3.5).images[0]
image.save("example.png")

Comfy UI

Work in progress.

Limitations

• The model is only runnable on NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See this issue for more details.
• You may observe some slight differences from the BF16 models in details.

Citation

If you find this model useful or relevant to your research, please cite

@article{
  li2024svdquant,
  title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models},
  author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song},
  journal={arXiv preprint arXiv:2411.05007},
  year={2024}
}