|
--- |
|
language: |
|
- en |
|
license: creativeml-openrail-m |
|
library_name: diffusers |
|
tags: |
|
- art |
|
- diffusion |
|
- Interior |
|
--- |
|
|
|
# KuJiaLe Layout ControlNet |
|
|
|
Given the structural elements of the room, such as the walls, floors, and ceilings. Our model auto-completes the furnishing of the room. |
|
The model is trained on [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) for interior designs. |
|
|
|
### Layout ControlNet Example |
|
Keep the room layout consistent, re-furnish the room. |
|
<table style="table-layout: fixed;"> |
|
<tr> |
|
<td style="text-align: center; vertical-align: middle; width: 50%">Input</td> |
|
<td style="text-align: center; vertical-align: middle; width: 50%">Output</td> |
|
</td> |
|
<tr> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/layout_input.jpg" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/layout_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_and_dining_room_input.jpg" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_and_dining_room_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_room_input.png" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_room_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
</tr> |
|
<tr> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/kitchen_input.jpg" alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
<td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/kitchen_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td> |
|
</tr> |
|
</tr> |
|
</table> |
|
|
|
## News🔥🔥🔥 |
|
|
|
* June.06, 2024. Our checkpoint Layout-ControlNet are publicly available on [HuggingFace Repo](https://huggingface.co/kujiale-ai/controlnet-layout). |
|
* June.06, 2024. Our Layout-ControlNet demo are publicly available on [HuggingFace Space](https://huggingface.co/spaces/ysmao/Layout-Control). |
|
|
|
## Try our HuggingFace demo: |
|
[HuggingFace Space Demo](https://huggingface.co/spaces/ysmao/Layout-Control) |
|
|
|
## Checkpoints |
|
* `control_v1_sd15_layout_fp16`: Layout ControlNet checkpoint, for SD15 models. |
|
|
|
## Using in 🧨 diffusers |
|
### Layout ControlNet |
|
```python |
|
import torch |
|
from diffusers.utils import load_image |
|
import numpy as np |
|
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline, UniPCMultistepScheduler |
|
|
|
controlnet_checkpoint = "kujiale-ai/controlnet-layout" |
|
# Load original image |
|
image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_input.jpg") |
|
depth_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_depth.jpg").convert("L") |
|
normal_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_normal.jpg") |
|
segm_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_segm.jpg") |
|
W, H = image.size |
|
depth_image = depth_image.resize((W, H)) |
|
normal_image = normal_image.resize((W, H)) |
|
segm_image = segm_image.resize((W, H)) |
|
# Prepare Layout Control Image |
|
depth_image = np.array(depth_image, dtype=np.float32) / 255.0 |
|
depth_image = torch.from_numpy(depth_image[:, :, None])[None].permute(0, 3, 1, 2) |
|
normal_image = np.array(normal_image, dtype=np.float32) |
|
normal_image = normal_image / 127.5 - 1.0 |
|
normal_image = torch.from_numpy(normal_image)[None].permute(0, 3, 1, 2) |
|
segm_image = np.array(segm_image, dtype=np.float32) / 255.0 |
|
segm_image = torch.from_numpy(segm_image)[None].permute(0, 3, 1, 2) |
|
control_image = torch.cat([depth_image, normal_image, segm_image], dim=1) |
|
# Initialize pipeline |
|
controlnet = ControlNetModel.from_pretrained(controlnet_checkpoint, subfolder="control_v1_sd15_layout_fp16", torch_dtype=torch.float16) |
|
pipe = StableDiffusionControlNetPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16).to("cuda") |
|
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) |
|
image = pipe("A modern bedroom,best quality", num_inference_steps=30, image=control_image, guidance_scale=7).images[0] |
|
image.save('layout_output.jpg') |
|
``` |
