Add pipeline tag, improve model card

README.md

@@ -1,6 +1,12 @@
 ---
+library_name: pytorch
 license: mit
-library_name: liveportrait
+pipeline_tag: image-to-video
+tags:
+- portrait-animation
+- video-generation
+- keypoint-based
+- efficient
 ---
 
 <h1 align="center">LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control</h1>
@@ -15,7 +21,7 @@ library_name: liveportrait
 
 <div align='center'>
     <a href='https://scholar.google.com/citations?user=P6MraaYAAAAJ' target='_blank'><strong>Pengfei Wan</strong></a><sup> 1</sup>&emsp;
-    <a href='https://openreview.net/profile?id=~Di_ZHANG3' target='_blank'><strong>Di Zhang</strong></a><sup> 1</sup>&emsp;
+    <a href='https://openreview.net/profile?id=~Di_ZHANG3\' target='_blank'><strong>Di Zhang</strong></a><sup> 1</sup>&emsp;
 </div>
 
 <div align='center'>
@@ -43,195 +49,11 @@ library_name: liveportrait
 
 
 ## 🔥 Updates
-- **`2024/08/02`**: 😸 We released a version of the **Animals model**, along with several other updates and improvements. Check out the details [**here**](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-08-02.md)!
-- **`2024/07/25`**: 📦 Windows users can now download the package from [HuggingFace](https://huggingface.co/cleardusk/LivePortrait-Windows/tree/main) or [BaiduYun](https://pan.baidu.com/s/1FWsWqKe0eNfXrwjEhhCqlw?pwd=86q2). Simply unzip and double-click `run_windows.bat` to enjoy!
-- **`2024/07/24`**: 🎨 We support pose editing for source portraits in the Gradio interface. We’ve also lowered the default detection threshold to increase recall. [Have fun](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-07-24.md)!
-- **`2024/07/19`**: ✨ We support 🎞️ portrait video editing (aka v2v)! More to see [here](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-07-19.md).
-- **`2024/07/17`**: 🍎 We support macOS with Apple Silicon, modified from [jeethu](https://github.com/jeethu)'s PR [#143](https://github.com/KwaiVGI/LivePortrait/pull/143).
-- **`2024/07/10`**: 💪 We support audio and video concatenating, driving video auto-cropping, and template making to protect privacy. More to see [here](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-07-10.md).
-- **`2024/07/09`**: 🤗 We released the [HuggingFace Space](https://huggingface.co/spaces/KwaiVGI/liveportrait), thanks to the HF team and [Gradio](https://github.com/gradio-app/gradio)!
-- **`2024/07/04`**: 😊 We released the initial version of the inference code and models. Continuous updates, stay tuned!
-- **`2024/07/04`**: 🔥 We released the [homepage](https://liveportrait.github.io) and technical report on [arXiv](https://arxiv.org/pdf/2407.03168).
-
-
-## Introduction 📖
-This repo, named **LivePortrait**, contains the official PyTorch implementation of our paper [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://arxiv.org/pdf/2407.03168).
-We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.
-
-## Getting Started 🏁
-### 1. Clone the code and prepare the environment
-```bash
-git clone https://github.com/KwaiVGI/LivePortrait
-cd LivePortrait
-
-# create env using conda
-conda create -n LivePortrait python==3.9
-conda activate LivePortrait
-
-# install dependencies with pip
-# for Linux and Windows users
-pip install -r requirements.txt
-# for macOS with Apple Silicon users
-pip install -r requirements_macOS.txt
-```
-
-**Note:** make sure your system has [FFmpeg](https://ffmpeg.org/download.html) installed, including both `ffmpeg` and `ffprobe`!
-
-### 2. Download pretrained weights
-
-The easiest way to download the pretrained weights is from HuggingFace:
-```bash
-# first, ensure git-lfs is installed, see: https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage
-git lfs install
-# clone and move the weights
-git clone https://huggingface.co/KwaiVGI/LivePortrait temp_pretrained_weights
-mv temp_pretrained_weights/* pretrained_weights/
-rm -rf temp_pretrained_weights
-```
-
-Alternatively, you can download all pretrained weights from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn). Unzip and place them in `./pretrained_weights`.
-
-Ensuring the directory structure is as follows, or contains:
-```text
-pretrained_weights
-├── insightface
-│   └── models
-│       └── buffalo_l
-│           ├── 2d106det.onnx
-│           └── det_10g.onnx
-└── liveportrait
-    ├── base_models
-    │   ├── appearance_feature_extractor.pth
-    │   ├── motion_extractor.pth
-    │   ├── spade_generator.pth
-    │   └── warping_module.pth
-    ├── landmark.onnx
-    └── retargeting_models
-        └── stitching_retargeting_module.pth
-```
-
-### 3. Inference 🚀
-
-#### Fast hands-on
-```bash
-# For Linux and Windows
-python inference.py
-
-# For macOS with Apple Silicon, Intel not supported, this maybe 20x slower than RTX 4090
-PYTORCH_ENABLE_MPS_FALLBACK=1 python inference.py
-```
-
-If the script runs successfully, you will get an output mp4 file named `animations/s6--d0_concat.mp4`. This file includes the following results: driving video, input image or video, and generated result.
-
-<p align="center">
-  <img src="./docs/inference.gif" alt="image">
-</p>
-
-Or, you can change the input by specifying the `-s` and `-d` arguments:
-
-```bash
-# source input is an image
-python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4
-
-# source input is a video ✨
-python inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving/d0.mp4
-
-# more options to see
-python inference.py -h
-```
-
-#### Driving video auto-cropping 📢📢📢
-To use your own driving video, we **recommend**: ⬇️
- - Crop it to a **1:1** aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-cropping by `--flag_crop_driving_video`.
- - Focus on the head area, similar to the example videos.
- - Minimize shoulder movement.
- - Make sure the first frame of driving video is a frontal face with **neutral expression**.
-
-Below is a auto-cropping case by `--flag_crop_driving_video`:
-```bash
-python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d13.mp4 --flag_crop_driving_video
-```
-
-If you find the results of auto-cropping is not well, you can modify the `--scale_crop_driving_video`, `--vy_ratio_crop_driving_video` options to adjust the scale and offset, or do it manually.
-
-#### Motion template making
-You can also use the auto-generated motion template files ending with `.pkl` to speed up inference, and **protect privacy**, such as:
-```bash
-python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d5.pkl # portrait animation
-python inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving/d5.pkl # portrait video editing
-```
-
-### 4. Gradio interface 🤗
-
-We also provide a Gradio <a href='https://github.com/gradio-app/gradio'><img src='https://img.shields.io/github/stars/gradio-app/gradio'></a> interface for a better experience, just run by:
-
-```bash
-# For Linux and Windows users (and macOS with Intel??)
-python app.py
-
-# For macOS with Apple Silicon users, Intel not supported, this maybe 20x slower than RTX 4090
-PYTORCH_ENABLE_MPS_FALLBACK=1 python app.py
-```
-
-You can specify the `--server_port`, `--share`, `--server_name` arguments to satisfy your needs!
-
-🚀 We also provide an acceleration option `--flag_do_torch_compile`. The first-time inference triggers an optimization process (about one minute), making subsequent inferences 20-30% faster. Performance gains may vary with different CUDA versions.
-```bash
-# enable torch.compile for faster inference
-python app.py --flag_do_torch_compile
-```
-**Note**: This method is not supported on Windows and macOS.
-
-**Or, try it out effortlessly on [HuggingFace](https://huggingface.co/spaces/KwaiVGI/LivePortrait) 🤗**
-
-### 5. Inference speed evaluation 🚀🚀🚀
-We have also provided a script to evaluate the inference speed of each module:
-
-```bash
-# For NVIDIA GPU
-python speed.py
-```
-
-Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with `torch.compile`:
-
-| Model                             | Parameters(M) | Model Size(MB) | Inference(ms) |
-|-----------------------------------|:-------------:|:--------------:|:-------------:|
-| Appearance Feature Extractor      |     0.84      |       3.3      |     0.82      |
-| Motion Extractor                  |     28.12     |       108      |     0.84      |
-| Spade Generator                   |     55.37     |       212      |     7.59      |
-| Warping Module                    |     45.53     |       174      |     5.21      |
-| Stitching and Retargeting Modules |     0.23      |       2.3      |     0.31      |
-
-*Note: The values for the Stitching and Retargeting Modules represent the combined parameter counts and total inference time of three sequential MLP networks.*
-
-## Community Resources 🤗
-
-Discover the invaluable resources contributed by our community to enhance your LivePortrait experience:
-
-- [ComfyUI-LivePortraitKJ](https://github.com/kijai/ComfyUI-LivePortraitKJ) by [@kijai](https://github.com/kijai)
-- [comfyui-liveportrait](https://github.com/shadowcz007/comfyui-liveportrait) by [@shadowcz007](https://github.com/shadowcz007)
-- [LivePortrait In ComfyUI](https://www.youtube.com/watch?v=aFcS31OWMjE) by [@Benji](https://www.youtube.com/@TheFutureThinker)
-- [LivePortrait hands-on tutorial](https://www.youtube.com/watch?v=uyjSTAOY7yI) by [@AI Search](https://www.youtube.com/@theAIsearch)
-- [ComfyUI tutorial](https://www.youtube.com/watch?v=8-IcDDmiUMM) by [@Sebastian Kamph](https://www.youtube.com/@sebastiankamph)
-- [Replicate Playground](https://replicate.com/fofr/live-portrait) and [cog-comfyui](https://github.com/fofr/cog-comfyui) by [@fofr](https://github.com/fofr)
-
-And many more amazing contributions from our community!
-
-## Acknowledgements 💐
-We would like to thank the contributors of [FOMM](https://github.com/AliaksandrSiarohin/first-order-model), [Open Facevid2vid](https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis), [SPADE](https://github.com/NVlabs/SPADE), [InsightFace](https://github.com/deepinsight/insightface) repositories, for their open research and contributions.
-
-## Citation 💖
-If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:
-```bibtex
-@article{guo2024liveportrait,
-  title   = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},
-  author  = {Guo, Jianzhu and Zhang, Dingyun and Liu, Xiaoqiang and Zhong, Zhizhou and Zhang, Yuan and Wan, Pengfei and Zhang, Di},
-  journal = {arXiv preprint arXiv:2407.03168},
-  year    = {2024}
-}
-```
-
-*Long live in arXiv.*
-
-## Contact 📧
-[**Jianzhu Guo (郭建珠)**](https://guojianzhu.com); **[email protected]**
+- **`2025/01/01`**: 🐶 We updated a new version of the Animals model with more data, see [**here**](./assets/docs/changelog/2025-01-01.md).
+- **`2024/10/18`**: ❗ We have updated the versions of the `transformers` and `gradio` libraries to avoid security vulnerabilities. Details [here](https://github.com/KwaiVGI/LivePortrait/pull/421/files).
+- **`2024/08/29`**: 📦 We update the Windows [one-click installer](https://huggingface.co/cleardusk/LivePortrait-Windows/blob/main/LivePortrait-Windows-v20240829.zip) and support auto-updates, see [changelog](https://huggingface.co/cleardusk/LivePortrait-Windows#20240829).
+- **`2024/08/19`**: 🖼️ We support **image driven mode** and **regional control**. For details, see [**here**](./assets/docs/changelog/2024-08-19.md).
+- **`2024/08/06`**: 🎨 We support **precise portrait editing** in the Gradio interface, inspired by [ComfyUI-AdvancedLivePortrait](https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait). See [**here**](./assets/docs/changelog/2024-08-06.md).
+- **`2024/08/05`**: 📦 Windows users can now download the [one-click installer](https://huggingface.co/cleardusk/LivePortrait-Windows/blob/main/LivePortrait-Windows-v20240806.zip) for Humans mode and **Animals mode** now! For details, see [**here**](./assets/docs/changelog/2024-08-05.md).
+- **`2024/08/02`**: 😸 We released a version of the **Animals model**, along with several other updates and improvements. Check out the details [**here**](./assets/docs/changelog/2024-08-02.md)!
+- **`2024/07/25`**: 📦 Windows users can now download the package from [HuggingFace](https://huggingface.co/cleardusk/LivePortrait-Windows/tree/main) or [BaiduYun](https://pan.baidu.com/s/1FWsWqKe0eNfXrwjEhhCqlw?pwd=86q2). Simply unzip and double-click `run_windows.bat` to enjoy!\n- **`2024/07/24`**: 🎨 We support pose editing for source portraits in the Gradio interface. We’ve also lowered the default detection threshold to increase recall. [Have fun](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-07-24.md)!\n- **`2024/07/19`**: ✨ We support 🎞️ portrait video editing (aka v2v)! More to see [here](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-07-19.md).\n- **`2024/07/17`**: 🍎 We support macOS with Apple Silicon, modified from [jeethu](https://github.com/jeethu)\'s PR [#143](https://github.com/KwaiVGI/LivePortrait/pull/143).\n- **`2024/07/10`**: 💪 We support audio and video concatenating, driving video auto-cropping, and template making to protect privacy. More to see [here](https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-07-10.md).\n- **`2024/07/09`**: 🤗 We released the [HuggingFace Space](https://huggingface.co/spaces/KwaiVGI/liveportrait), thanks to the HF team and [Gradio](https://github.com/gradio-app/gradio)!\n- **`2024/07/04`**: 😊 We released the initial version of the inference code and models. Continuous updates, stay tuned!\n- **`2024/07/04`**: 🔥 We released the [homepage](https://liveportrait.github.io) and technical report on [arXiv](https://arxiv.org/pdf/2407.03168).\n\n\n## Introduction 📖\nThis repo, named **LivePortrait**, contains the official PyTorch implementation of our paper [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://arxiv.org/pdf/2407.03168).\nWe are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.\n\n## Getting Started 🏁\n### 1. Clone the code and prepare the environment\n```bash\ngit clone https://github.com/KwaiVGI/LivePortrait\ncd LivePortrait\n\n# create env using conda\nconda create -n LivePortrait python==3.9\nconda activate LivePortrait\n\n# install dependencies with pip\n# for Linux and Windows users\npip install -r requirements.txt\n# for macOS with Apple Silicon users\npip install -r requirements_macOS.txt\n```\n\n**Note:** make sure your system has [FFmpeg](https://ffmpeg.org/download.html) installed, including both `ffmpeg` and `ffprobe`!\n\n### 2. Download pretrained weights\n\nThe easiest way to download the pretrained weights is from HuggingFace:\n```bash\n# first, ensure git-lfs is installed, see: https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage\ngit lfs install\n# clone and move the weights\ngit clone https://huggingface.co/KwaiVGI/LivePortrait temp_pretrained_weights\nmv temp_pretrained_weights/* pretrained_weights/\nrm -rf temp_pretrained_weights\n```\n\nAlternatively, you can download all pretrained weights from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn). Unzip and place them in `./pretrained_weights`.\n\nEnsuring the directory structure is as follows, or contains:\n```text\npretrained_weights\n├── insightface\n│   └── models\n│       └── buffalo_l\n│           ├── 2d106det.onnx\n│           └── det_10g.onnx\n└── liveportrait\n    ├── base_models\n    │   ├── appearance_feature_extractor.pth\n    │   ├── motion_extractor.pth\n    │   ├── spade_generator.pth\n    │   └── warping_module.pth\n    ├── landmark.onnx\n    └── retargeting_models\n        └── stitching_retargeting_module.pth\n```\n\n### 3. Inference 🚀\n\n#### Fast hands-on\n```bash\n# For Linux and Windows\npython inference.py\n\n# For macOS with Apple Silicon, Intel not supported, this maybe 20x slower than RTX 4090\nPYTORCH_ENABLE_MPS_FALLBACK=1 python inference.py\n```\n\nIf the script runs successfully, you will get an output mp4 file named `animations/s6--d0_concat.mp4`. This file includes the following results: driving video, input image or video, and generated result.\n\n<p align="center">\n  <img src="./docs/inference.gif" alt="image">\n</p>\n\nOr, you can change the input by specifying the `-s` and `-d` arguments:\n\n```bash\n# source input is an image\npython inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4\n\n# source input is a video ✨\npython inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving/d0.mp4\n\n# more options to see\npython inference.py -h\n```\n\n#### Driving video auto-cropping 📢📢📢\nTo use your own driving video, we **recommend**: ⬇️\n - Crop it to a **1:1** aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-cropping by `--flag_crop_driving_video`.\n - Focus on the head area, similar to the example videos.\n - Minimize shoulder movement.\n - Make sure the first frame of driving video is a frontal face with **neutral expression**.\n\nBelow is a auto-cropping case by `--flag_crop_driving_video`:\n```bash\npython inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d13.mp4 --flag_crop_driving_video\n```\n\nIf you find the results of auto-cropping is not well, you can modify the `--scale_crop_driving_video`, `--vy_ratio_crop_driving_video` options to adjust the scale and offset, or do it manually.\n\n#### Motion template making\nYou can also use the auto-generated motion template files ending with `.pkl` to speed up inference, and **protect privacy**, such as:\n```bash\npython inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d5.pkl # portrait animation\npython inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving/d5.pkl # portrait video editing\n```\n\n### 4. Gradio interface 🤗\n\nWe also provide a Gradio <a href=\'https://github.com/gradio-app/gradio\'><img src=\'https://img.shields.io/github/stars/gradio-app/gradio\'></a> interface for a better experience, just run by:\n\n```bash\n# For Linux and Windows users (and macOS with Intel??)\npython app.py\n\n# For macOS with Apple Silicon users, Intel not supported, this maybe 20x slower than RTX 4090\nPYTORCH_ENABLE_MPS_FALLBACK=1 python app.py\n```\n\nYou can specify the `--server_port`, `--share`, `--server_name` arguments to satisfy your needs!\n\n🚀 We also provide an acceleration option `--flag_do_torch_compile`. The first-time inference triggers an optimization process (about one minute), making subsequent inferences 20-30% faster. Performance gains may vary with different CUDA versions.\n```bash\n# enable torch.compile for faster inference\npython app.py --flag_do_torch_compile\n```\n**Note**: This method is not supported on Windows and macOS.\n\n**Or, try it out effortlessly on [HuggingFace](https://huggingface.co/spaces/KwaiVGI/liveportrait) 🤗**\n\n### 5. Inference speed evaluation 🚀🚀🚀\nWe have also provided a script to evaluate the inference speed of each module:\n\n```bash\n# For NVIDIA GPU\npython speed.py\n```\n\nBelow are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with `torch.compile`:\n\n| Model                             | Parameters(M) | Model Size(MB) | Inference(ms) |\n|-----------------------------------|:-------------:|:--------------:|:-------------:|\n| Appearance Feature Extractor      |     0.84      |       3.3      |     0.82      |\n| Motion Extractor                  |     28.12     |       108      |     0.84      |\n| Spade Generator                   |     55.37     |       212      |     7.59      |\n| Warping Module                    |     45.53     |       174      |     5.21      |\n| Stitching and Retargeting Modules |     0.23      |       2.3      |     0.31      |\n\n*Note: The values for the Stitching and Retargeting Modules represent the combined parameter counts and total inference time of three sequential MLP networks.*\n\n## Community Resources 🤗\n\nDiscover the invaluable resources contributed by our community to enhance your LivePortrait experience:\n\n- [ComfyUI-LivePortraitKJ](https://github.com/kijai/ComfyUI-LivePortraitKJ) by [@kijai](https://github.com/kijai)\n- [comfyui-liveportrait](https://github.com/shadowcz007/comfyui-liveportrait) by [@shadowcz007](https://github.com/shadowcz007)\n- [LivePortrait In ComfyUI](https://www.youtube.com/watch?v=aFcS31OWMjE) by [@Benji](https://www.youtube.com/@TheFutureThinker)\n- [LivePortrait hands-on tutorial](https://www.youtube.com/watch?v=uyjSTAOY7yI) by [@AI Search](https://www.youtube.com/@theAIsearch)\n- [ComfyUI tutorial](https://www.youtube.com/watch?v=8-IcDDmiUMM) by [@Sebastian Kamph](https://www.youtube.com/@sebastiankamph)\n- [Replicate Playground](https://replicate.com/fofr/live-portrait) and [cog-comfyui](https://github.com/fofr/cog-comfyui) by [@fofr](https://github.com/fofr)\n\nAnd many more amazing contributions from our community!\n\n## Acknowledgements 💐\nWe would like to thank the contributors of [FOMM](https://github.com/AliaksandrSiarohin/first-order-model), [Open Facevid2vid](https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis), [SPADE](https://github.com/NVlabs/SPADE), [InsightFace](https://github.com/deepinsight/insightface) and [X-Pose](https://github.com/IDEA-Research/X-Pose) repositories, for their open research and contributions.\n\n## Ethics Considerations 🛡️\nPortrait animation technologies come with social risks, particularly the potential for misuse in creating deepfakes. To mitigate these risks, it’s crucial to follow ethical guidelines and adopt responsible usage practices. At present, the synthesized results contain visual artifacts that may help in detecting deepfakes. Please note that we do not assume any legal responsibility for the use of the results generated by this project.\n\n## Citation 💖\nIf you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:\n```bibtex\n@article{guo2024liveportrait,\n  title   = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},\n  author  = {Guo, Jianzhu and Zhang, Dingyun and Liu,Xiaoqiang and Zhong, Zhizhou and Zhang, Yuan and Wan, Pengfei and Zhang, Di},\n  journal = {arXiv preprint arXiv:2407.03168},\n  year    = {2024}\n}\n```\n\n*Long live in arXiv.*\n\n## Contact 📧\n[**Jianzhu Guo (郭建珠)**](https://guojianzhu.com); **[email protected]**