File size: 2,658 Bytes
7c5a090 6030126 7c5a090 6030126 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
license: apache-2.0
language:
- en
tags:
- diffusion model
- stable diffusion
- SCEdit
- Scepter
- Scepter studio
- Controllable
- ControlNet
- Lora
---
<p align="center">
<h2 align="center">馃獎SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing</h2>
<p align="center">
<strong>Zeyinzi Jiang</strong>
路
<strong>Chaojie Mao</strong>
路
<strong>Yulin Pan</strong>
路
<strong>Zhen Han</strong>
路
<strong>Jingfeng Zhang</strong>
<br>
<br>
<a href="https://arxiv.org/abs/2312.11392"><img src='https://img.shields.io/badge/arXiv-SCEdit-red' alt='Paper PDF'></a>
<a href='https://scedit.github.io/'><img src='https://img.shields.io/badge/Project_Page-SCEdit-green' alt='Project Page'></a>
<a href='https://github.com/modelscope/scepter'><img src='https://img.shields.io/badge/scepter-SCEdit-yellow'></a>
<a href='https://github.com/modelscope/swift'><img src='https://img.shields.io/badge/swift-SCEdit-blue'></a>
<br>
<b>Alibaba Group</b>
</p>
<table align="center">
<tr>
<td>
<img src="assets/figures/show.jpg">
</td>
</tr>
</table>
SCEdit is an efficient generative fine-tuning framework proposed by Alibaba TongYi Vision Intelligence Lab. This framework enhances the fine-tuning capabilities for text-to-image generation downstream tasks and enables quick adaptation to specific generative scenarios, **saving 30%-50% of training memory costs compared to LoRA**. Furthermore, it can be directly extended to controllable image generation tasks, **requiring only 7.9% of the parameters that ControlNet needs for conditional generation and saving 30% of memory usage**. It supports various conditional generation tasks including edge maps, depth maps, segmentation maps, poses, color maps, and image completion.
#### Code Example
```shell
git clone https://github.com/modelscope/scepter.git
cd scepter
PYTHONPATH=. python scepter/tools/run_train.py --cfg scepter/methods/SCEdit/t2i_sdxl_1024_sce.yaml
```
To prepare the training dataset.
```python
# pip install modelscope
from modelscope.msdatasets import MsDataset
ms_train_dataset = MsDataset.load('style_custom_dataset', namespace='damo', subset_name='3D', split='train_short')
print(next(iter(ms_train_dataset)))
```
## BibTeX
```bibtex
@article{jiang2023scedit,
title = {SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing},
author = {Jiang, Zeyinzi and Mao, Chaojie and Pan, Yulin and Han, Zhen and Zhang, Jingfeng},
year = {2023},
journal = {arXiv preprint arXiv:2312.11392}
}
``` |