CaptainZZZ/sd3-instructpix2pix

Example Usage

import torch
from diffusers import SD3Transformer2DModel
from diffusers import DiffusionPipeline
from diffusers.utils import load_image


resolution = 512
image = load_image("https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png").resize(
    (resolution, resolution)
)
edit_instruction = "Turn sky into a sunny one"


pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers", custom_pipeline="pipeline_stable_diffusion_3_instruct_pix2pix", torch_dtype=torch.float16).to('cuda')

pipe.transformer = SD3Transformer2DModel.from_pretrained("CaptainZZZ/sd3-instructpix2pix",torch_dtype=torch.float16).to('cuda')

edited_image = pipe(
    prompt=edit_instruction,
    image=image,
    height=resolution,
    width=resolution,
    guidance_scale=7.5,
    image_guidance_scale=1.5,
    num_inference_steps=30,
).images[0]

edited_image.save("edited_image.png")

Original	Edited

Note

This model is trained on 512x512, so input size is better on 512x512. For better editing performance, please refer to this powerful model https://huggingface.co/BleachNick/SD3_UltraEdit_freeform and Paper "UltraEdit: Instruction-based Fine-Grained Image Editing at Scale", many thanks to their contribution!