Add diffusers example (#20)

- Add diffusers example (e9a76799bc397335fbfa74b52d7ef473c636a4d2)


Co-authored-by: Patrick von Platen <[email protected]>

README.md

@@ -16,6 +16,61 @@ zeroscope_v2_XL uses 15.3gb of vram when rendering 30 frames at 1024x576
 2. Replace the respective files in the 'stable-diffusion-webui\models\ModelScope\t2v' directory.
 ### Upscaling recommendations
 For upscaling, it's recommended to use the 1111 extension. It works best at 1024x576 with a denoise strength between 0.66 and 0.85. Remember to use the same prompt that was used to generate the original clip.
+
+### Usage in 🧨 Diffusers
+
+Let's first install the libraries required:
+
+```bash
+$ pip install git+https://github.com/huggingface/diffusers.git
+$ pip install transformers accelerate torch
+```
+
+Now, let's first generate a low resolution video using [cerspense/zeroscope_v2_576w](https://huggingface.co/cerspense/zeroscope_v2_576w).
+
+```py
+import torch
+from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
+from diffusers.utils import export_to_video
+
+pipe = DiffusionPipeline.from_pretrained("cerspense/zeroscope_v2_576w", torch_dtype=torch.float16)
+pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
+pipe.enable_model_cpu_offload()
+pipe.enable_vae_slicing()
+
+prompt = "Darth Vader is surfing on waves"
+video_frames = pipe(prompt, num_inference_steps=40, height=320, width=576, num_frames=36).frames
+video_path = export_to_video(video_frames)
+```
+
+Next, we can upscale it using [cerspense/zeroscope_v2_XL](https://huggingface.co/cerspense/zeroscope_v2_XL).
+
+```py
+pipe = DiffusionPipeline.from_pretrained("cerspense/zeroscope_v2_XL", torch_dtype=torch.float16)
+pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
+pipe.enable_model_cpu_offload()
+pipe.enable_vae_slicing()
+
+video = [Image.fromarray(frame).resize((1024, 576)) for frame in video_frames]
+
+video_frames = pipe(prompt, video=video, strength=0.6).frames
+video_path = export_to_video(video_frames, output_video_path="/home/patrick/videos/video_1024_darth_vader_36.mp4")
+```
+
+Here are some results:
+
+<table>
+    <tr>
+        Darth vader is surfing on waves.
+        <br>
+        <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/darth_vader_36_1024.gif"
+            alt="Darth vader surfing in waves."
+            style="width: 576;" />
+        </center></td>
+    </tr>
+</table>
+
+
 ### Known issues
 Rendering at lower resolutions or fewer than 24 frames could lead to suboptimal outputs. <br />