Add pipeline tag
#2
by
nielsr
HF staff
- opened
This view is limited to 50 files because it contains too many changes.
See the raw diff here.
- .gitattributes +0 -3
- .gitignore +0 -2
- README.md +4 -119
- assets/test_imgs/ghibli_output/00_0.png +0 -3
- assets/test_imgs/ghibli_output/00_1.png +0 -3
- assets/test_imgs/ghibli_output/00_2.png +0 -3
- assets/test_imgs/ghibli_output/00_3.png +0 -3
- assets/test_imgs/ghibli_output/00_4.png +0 -3
- assets/test_imgs/ghibli_output/02_0.png +0 -3
- assets/test_imgs/ghibli_output/02_1.png +0 -3
- assets/test_imgs/ghibli_output/02_2.png +0 -3
- assets/test_imgs/ghibli_output/02_3.png +0 -3
- assets/test_imgs/ghibli_output/02_4.png +0 -3
- assets/test_imgs/ghibli_output/03_0.png +0 -3
- assets/test_imgs/ghibli_output/03_1.png +0 -3
- assets/test_imgs/ghibli_output/03_2.png +0 -3
- assets/test_imgs/ghibli_output/03_3.png +0 -3
- assets/test_imgs/ghibli_output/03_4.png +0 -3
- assets/test_imgs/ghibli_output/04_0.png +0 -3
- assets/test_imgs/ghibli_output/04_1.png +0 -3
- assets/test_imgs/ghibli_output/04_2.png +0 -3
- assets/test_imgs/ghibli_output/04_3.png +0 -3
- assets/test_imgs/ghibli_output/04_4.png +0 -3
- assets/test_imgs/ghibli_output/06_0.png +0 -3
- assets/test_imgs/ghibli_output/06_1.png +0 -3
- assets/test_imgs/ghibli_output/06_2.png +0 -3
- assets/test_imgs/ghibli_output/06_3.png +0 -3
- assets/test_imgs/ghibli_output/06_4.png +0 -3
- assets/test_imgs/ghibli_output/07_0.png +0 -3
- assets/test_imgs/ghibli_output/07_1.png +0 -3
- assets/test_imgs/ghibli_output/07_2.png +0 -3
- assets/test_imgs/ghibli_output/07_3.png +0 -3
- assets/test_imgs/ghibli_output/07_4.png +0 -3
- assets/test_imgs/ghibli_output/08_0.png +0 -3
- assets/test_imgs/ghibli_output/08_1.png +0 -3
- assets/test_imgs/ghibli_output/08_2.png +0 -3
- assets/test_imgs/ghibli_output/08_3.png +0 -3
- assets/test_imgs/ghibli_output/08_4.png +0 -3
- assets/test_imgs/ghibli_output/09_0.png +0 -3
- assets/test_imgs/ghibli_output/09_1.png +0 -3
- assets/test_imgs/ghibli_output/09_2.png +0 -3
- assets/test_imgs/ghibli_output/09_3.png +0 -3
- assets/test_imgs/ghibli_output/09_4.png +0 -3
- assets/test_imgs/ghibli_ref/00.png +0 -3
- assets/test_imgs/ghibli_ref/02.png +0 -3
- assets/test_imgs/ghibli_ref/03.png +0 -3
- assets/test_imgs/ghibli_ref/04.png +0 -3
- assets/test_imgs/ghibli_ref/06.png +0 -3
- assets/test_imgs/ghibli_ref/07.png +0 -3
- assets/test_imgs/ghibli_ref/08.png +0 -3
.gitattributes
CHANGED
@@ -34,6 +34,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
assets/*.jpg filter=lfs diff=lfs merge=lfs -text
|
37 |
-
*.jpg filter=lfs diff=lfs merge=lfs -text
|
38 |
-
*.jpeg filter=lfs diff=lfs merge=lfs -text
|
39 |
-
*.png filter=lfs diff=lfs merge=lfs -text
|
|
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
assets/*.jpg filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
.gitignore
DELETED
@@ -1,2 +0,0 @@
|
|
1 |
-
.DS_Store
|
2 |
-
**/.DS_Store
|
|
|
|
|
|
README.md
CHANGED
@@ -1,5 +1,6 @@
|
|
1 |
---
|
2 |
license: apache-2.0
|
|
|
3 |
---
|
4 |
# Implementation of EasyControl
|
5 |
|
@@ -7,8 +8,6 @@ EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer
|
|
7 |
|
8 |
<a href='https://arxiv.org/pdf/2503.07027'><img src='https://img.shields.io/badge/Technique-Report-red'></a>
|
9 |
<a href="https://github.com/Xiaojiu-z/EasyControl/tree/dev"><img src="https://img.shields.io/badge/GitHub-Code-blue.svg?logo=github&" alt="GitHub"></a>
|
10 |
-
<a href='https://huggingface.co/spaces/jamesliu1217/EasyControl'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'></a>
|
11 |
-
<a href='https://huggingface.co/spaces/jamesliu1217/EasyControl_Ghibli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Ghibli_Spaces-purple'></a>
|
12 |
|
13 |
> *[Yuxuan Zhang](https://xiaojiu-z.github.io/YuxuanZhang.github.io/), [Yirui Yuan](https://github.com/Reynoldyy), [Yiren Song](https://scholar.google.com.hk/citations?user=L2YS0jgAAAAJ), [Haofan Wang](https://haofanwang.github.io/), [Jiaming Liu](https://scholar.google.com/citations?user=SmL7oMQAAAAJ&hl=en)*
|
14 |
> <br>
|
@@ -82,7 +81,6 @@ control_models = {
|
|
82 |
"seg": f"{lora_path}/seg.safetensors",
|
83 |
"inpainting": f"{lora_path}/inpainting.safetensors",
|
84 |
"subject": f"{lora_path}/subject.safetensors",
|
85 |
-
"Ghibli": f"{lora_path}/Ghibli.safetensors"
|
86 |
}
|
87 |
```
|
88 |
|
@@ -95,7 +93,7 @@ set_single_lora(pipe.transformer, path, lora_weights=[1], cond_size=512)
|
|
95 |
|
96 |
# Generate image
|
97 |
prompt = "A nice car on the beach"
|
98 |
-
spatial_image =
|
99 |
|
100 |
image = pipe(
|
101 |
prompt,
|
@@ -121,8 +119,8 @@ paths = [control_models["subject"], control_models["inpainting"]]
|
|
121 |
set_multi_lora(pipe.transformer, paths, lora_weights=[[1], [1]], cond_size=512)
|
122 |
|
123 |
prompt = "A SKS on the car"
|
124 |
-
subject_images = [
|
125 |
-
spatial_images = [
|
126 |
|
127 |
image = pipe(
|
128 |
prompt,
|
@@ -150,119 +148,6 @@ clear_cache(pipe.transformer)
|
|
150 |
- Use `num_inference_steps=25` for a good balance of quality and speed
|
151 |
- When using set_multi_lora api, make sure the subject lora path(subject) is before the spatial lora path(canny, depth, hedsketch, etc.).
|
152 |
|
153 |
-
<br>
|
154 |
-
|
155 |
-
# 🎨 Ghibli-Style LoRA for Portraits
|
156 |
-
|
157 |
-
[NEW!!] New **Ghibli LoRA model** is now released!! Transform Asian portraits into Studio Ghibli-style artwork using this LoRA model. Trained on **only 100 real Asian faces** paired with **GPT-4o-generated Ghibli-style counterparts**, it preserves facial features while applying the iconic anime aesthetic.
|
158 |
-
|
159 |
-
## 🖼️ Example Gallery
|
160 |
-
> **8 complete transformation sets available** in `/assets/test_imgs/` directory
|
161 |
-
> *All outputs generated with identical prompt weights* with same prompt: `"Ghibli Studio style, Charming hand-drawn anime-style illustration"`
|
162 |
-
### Case1
|
163 |
-
<table>
|
164 |
-
<tr>
|
165 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/00.png"></td>
|
166 |
-
<td><img src="./assets/test_imgs/ghibli_output/00_0.png"></td>
|
167 |
-
<td><img src="./assets/test_imgs/ghibli_output/00_1.png"></td>
|
168 |
-
</tr>
|
169 |
-
<tr>
|
170 |
-
<td><img src="./assets/test_imgs/ghibli_output/00_2.png"></td>
|
171 |
-
<td><img src="./assets/test_imgs/ghibli_output/00_3.png"></td>
|
172 |
-
</tr>
|
173 |
-
</table>
|
174 |
-
|
175 |
-
### Case2
|
176 |
-
<table>
|
177 |
-
<tr>
|
178 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/02.png"></td>
|
179 |
-
<td><img src="./assets/test_imgs/ghibli_output/02_0.png"></td>
|
180 |
-
<td><img src="./assets/test_imgs/ghibli_output/02_1.png"></td>
|
181 |
-
</tr>
|
182 |
-
<tr>
|
183 |
-
<td><img src="./assets/test_imgs/ghibli_output/02_2.png"></td>
|
184 |
-
<td><img src="./assets/test_imgs/ghibli_output/02_3.png"></td>
|
185 |
-
</tr>
|
186 |
-
</table>
|
187 |
-
|
188 |
-
### Case3
|
189 |
-
<table>
|
190 |
-
<tr>
|
191 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/03.png"></td>
|
192 |
-
<td><img src="./assets/test_imgs/ghibli_output/03_0.png"></td>
|
193 |
-
<td><img src="./assets/test_imgs/ghibli_output/03_1.png"></td>
|
194 |
-
</tr>
|
195 |
-
<tr>
|
196 |
-
<td><img src="./assets/test_imgs/ghibli_output/03_2.png"></td>
|
197 |
-
<td><img src="./assets/test_imgs/ghibli_output/03_3.png"></td>
|
198 |
-
</tr>
|
199 |
-
</table>
|
200 |
-
|
201 |
-
### Case4
|
202 |
-
<table>
|
203 |
-
<tr>
|
204 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/04.png"></td>
|
205 |
-
<td><img src="./assets/test_imgs/ghibli_output/04_0.png"></td>
|
206 |
-
<td><img src="./assets/test_imgs/ghibli_output/04_1.png"></td>
|
207 |
-
</tr>
|
208 |
-
<tr>
|
209 |
-
<td><img src="./assets/test_imgs/ghibli_output/04_2.png"></td>
|
210 |
-
<td><img src="./assets/test_imgs/ghibli_output/04_3.png"></td>
|
211 |
-
</tr>
|
212 |
-
</table>
|
213 |
-
|
214 |
-
### Case5
|
215 |
-
<table>
|
216 |
-
<tr>
|
217 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/06.png"></td>
|
218 |
-
<td><img src="./assets/test_imgs/ghibli_output/06_0.png"></td>
|
219 |
-
<td><img src="./assets/test_imgs/ghibli_output/06_1.png"></td>
|
220 |
-
</tr>
|
221 |
-
<tr>
|
222 |
-
<td><img src="./assets/test_imgs/ghibli_output/06_2.png"></td>
|
223 |
-
<td><img src="./assets/test_imgs/ghibli_output/06_3.png"></td>
|
224 |
-
</tr>
|
225 |
-
</table>
|
226 |
-
|
227 |
-
### Case6
|
228 |
-
<table>
|
229 |
-
<tr>
|
230 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/07.png"></td>
|
231 |
-
<td><img src="./assets/test_imgs/ghibli_output/07_0.png"></td>
|
232 |
-
<td><img src="./assets/test_imgs/ghibli_output/07_1.png"></td>
|
233 |
-
</tr>
|
234 |
-
<tr>
|
235 |
-
<td><img src="./assets/test_imgs/ghibli_output/07_2.png"></td>
|
236 |
-
<td><img src="./assets/test_imgs/ghibli_output/07_3.png"></td>
|
237 |
-
</tr>
|
238 |
-
</table>
|
239 |
-
|
240 |
-
### Case7
|
241 |
-
<table>
|
242 |
-
<tr>
|
243 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/08.png"></td>
|
244 |
-
<td><img src="./assets/test_imgs/ghibli_output/08_0.png"></td>
|
245 |
-
<td><img src="./assets/test_imgs/ghibli_output/08_1.png"></td>
|
246 |
-
</tr>
|
247 |
-
<tr>
|
248 |
-
<td><img src="./assets/test_imgs/ghibli_output/08_2.png"></td>
|
249 |
-
<td><img src="./assets/test_imgs/ghibli_output/08_3.png"></td>
|
250 |
-
</tr>
|
251 |
-
</table>
|
252 |
-
|
253 |
-
### Case8
|
254 |
-
<table>
|
255 |
-
<tr>
|
256 |
-
<td rowspan="2"><img src="./assets/test_imgs/ghibli_ref/09.png"></td>
|
257 |
-
<td><img src="./assets/test_imgs/ghibli_output/09_0.png"></td>
|
258 |
-
<td><img src="./assets/test_imgs/ghibli_output/09_1.png"></td>
|
259 |
-
</tr>
|
260 |
-
<tr>
|
261 |
-
<td><img src="./assets/test_imgs/ghibli_output/09_2.png"></td>
|
262 |
-
<td><img src="./assets/test_imgs/ghibli_output/09_3.png"></td>
|
263 |
-
</tr>
|
264 |
-
</table>
|
265 |
-
|
266 |
|
267 |
## Disclaimer
|
268 |
The code of EasyControl is released under [Apache License](https://github.com/Xiaojiu-Z/EasyControl?tab=Apache-2.0-1-ov-file#readme) for both academic and commercial usage. Our released checkpoints are for research purposes only. Users are granted the freedom to create images using this tool, but they are obligated to comply with local laws and utilize it responsibly. The developers will not assume any responsibility for potential misuse by users.
|
|
|
1 |
---
|
2 |
license: apache-2.0
|
3 |
+
pipeline_tag: text-to-image
|
4 |
---
|
5 |
# Implementation of EasyControl
|
6 |
|
|
|
8 |
|
9 |
<a href='https://arxiv.org/pdf/2503.07027'><img src='https://img.shields.io/badge/Technique-Report-red'></a>
|
10 |
<a href="https://github.com/Xiaojiu-z/EasyControl/tree/dev"><img src="https://img.shields.io/badge/GitHub-Code-blue.svg?logo=github&" alt="GitHub"></a>
|
|
|
|
|
11 |
|
12 |
> *[Yuxuan Zhang](https://xiaojiu-z.github.io/YuxuanZhang.github.io/), [Yirui Yuan](https://github.com/Reynoldyy), [Yiren Song](https://scholar.google.com.hk/citations?user=L2YS0jgAAAAJ), [Haofan Wang](https://haofanwang.github.io/), [Jiaming Liu](https://scholar.google.com/citations?user=SmL7oMQAAAAJ&hl=en)*
|
13 |
> <br>
|
|
|
81 |
"seg": f"{lora_path}/seg.safetensors",
|
82 |
"inpainting": f"{lora_path}/inpainting.safetensors",
|
83 |
"subject": f"{lora_path}/subject.safetensors",
|
|
|
84 |
}
|
85 |
```
|
86 |
|
|
|
93 |
|
94 |
# Generate image
|
95 |
prompt = "A nice car on the beach"
|
96 |
+
spatial_image = "./test_imgs/canny.png"
|
97 |
|
98 |
image = pipe(
|
99 |
prompt,
|
|
|
119 |
set_multi_lora(pipe.transformer, paths, lora_weights=[[1], [1]], cond_size=512)
|
120 |
|
121 |
prompt = "A SKS on the car"
|
122 |
+
subject_images = ["./test_imgs/subject_1.png"]
|
123 |
+
spatial_images = ["./test_imgs/inpainting.png"]
|
124 |
|
125 |
image = pipe(
|
126 |
prompt,
|
|
|
148 |
- Use `num_inference_steps=25` for a good balance of quality and speed
|
149 |
- When using set_multi_lora api, make sure the subject lora path(subject) is before the spatial lora path(canny, depth, hedsketch, etc.).
|
150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
151 |
|
152 |
## Disclaimer
|
153 |
The code of EasyControl is released under [Apache License](https://github.com/Xiaojiu-Z/EasyControl?tab=Apache-2.0-1-ov-file#readme) for both academic and commercial usage. Our released checkpoints are for research purposes only. Users are granted the freedom to create images using this tool, but they are obligated to comply with local laws and utilize it responsibly. The developers will not assume any responsibility for potential misuse by users.
|
assets/test_imgs/ghibli_output/00_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/00_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/00_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/00_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/00_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/02_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/02_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/02_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/02_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/02_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/03_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/03_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/03_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/03_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/03_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/04_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/04_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/04_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/04_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/04_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/06_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/06_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/06_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/06_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/06_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/07_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/07_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/07_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/07_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/07_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/08_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/08_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/08_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/08_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/08_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/09_0.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/09_1.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/09_2.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/09_3.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_output/09_4.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/00.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/02.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/03.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/04.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/06.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/07.png
DELETED
Git LFS Details
|
assets/test_imgs/ghibli_ref/08.png
DELETED
Git LFS Details
|