File size: 3,186 Bytes
1e4a04f 247c1aa 1e4a04f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
---
license: other
license_name: flux-1-dev-non-commercial-license
license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.
datasets:
- raulc0399/open_pose_controlnet
language:
- en
pipeline_tag: text-to-image
tags:
- Stable Diffusion
- image-generation
- Flux
- diffusers
- controlnet
---
# openpose controlnet for flux.dev
(big thanks to [oxen.ai](https://www.oxen.ai/) for sponsoring the GPU for the training)
## inference
an openpose controlnet for flux-dev, trained on https://huggingface.co/datasets/raulc0399/open_pose_controlnet
the controlnet model is trained for the xlabs ai pipeline https://github.com/XLabs-AI/x-flux
to install the pipeline, execute the following:
```
git clone https://github.com/XLabs-AI/x-flux.git
cd x-flux
python3 -m venv xflux_env
source xflux_env/bin/activate
pip install -r requirements.txt
```
to run the pipeline with controlnet:
```
python3 main.py \
--prompt "person enjoying a day at the park, full hd, cinematic" \
--image ~/open_pose_controlnet_dataset/validation_images/pose/3_pose_1024.jpg --control_type openpose \
--local_path ./model.safetensors \
--use_controlnet --model_type flux-dev \
--width 1024 --height 1024 --timestep_to_start_cfg 2 \
--num_steps 50 --true_gs 4 --guidance 4 \
--save_path ~/gen_imgs
```
if the image has already been preprocessed comment out the line #146 from src/flux/xflux_pipeline.py
```
# self.annotator = Annotator(control_type, self.other_device)
```
## training
```
oxen clone https://hub.oxen.ai/raulc/open_pose_controlnet_dataset
git clone https://github.com/raulc0399/x-flux.git
cd x-flux
git checkout open_pose_training
python3 -m venv xflux_env
source xflux_env/bin/activate
pip install -r requirements.txt
huggingface-cli login
accelerate config
mkdir images
rsync -r ~/open_pose_controlnet_dataset/train/images/ images/
cp train_configs/test_openpose_controlnet.yaml train_configs/openpose_controlnet.yaml
accelerate launch train_flux_deepspeed_controlnet.py --config "train_configs/openpose_controlnet.yaml"
```
note 1: check the file train_configs/openpose_controlnet.yaml before starting
note 2: rsync is needed, cp does not work with that many files
note 3: the oxen repo has the caption files as json as expected by the training script
## results
using these 2 images:
with these prompts:
"two friends sitting by each other enjoying a day at the park, full hd, cinematic"
"person enjoying a day at the park, full hd, cinematic"
resulted in these images:
## License
Weights fall under the [FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) Non-Commercial License<br/>
|