Animagus XL 1.0

Animagus is an anime-themed finetuned model built on top of Stable Diffusion XL, created over the Animagine XL series. The model merges over 20 specialized checkpoints (including various versions of Animagine, babesByStableYogi, bigasp, juggernaut, tofuANIMEBASEMODEL, kohakuXLZeta, starryXL, hassakuXL, animeIllustDiffusion, RaeDiffusion-XL, UrangDiffusion, clandestine, and ranimeXL) using multistage merging methods (karcher_mean, ties_sum and task_arithmetic), and then finetuned on carefully selected illustration datasets. This results in more stable outputs, accurate anatomy, and rich colors, while preserving the “anime” style.

Get Animagus XL 1.0

Civitai hub
Explore model details, community examples, and usage tips on Civitai:
https://civitai.com/models/1783095
Download untrained merge (Animagus XL Zero)
A raw merge of all source checkpoints before fine-tuning:
https://huggingface.co/rityak/animagus/resolve/main/animagus-zero.safetensors?download=true
Download fine-tuned model (Animagus XL 1.0)
The final anime-optimized checkpoint after fine-tuning:
https://huggingface.co/rityak/animagus/resolve/main/animagus.safetensors?download=true

Model Details

Model type: diffusion-based text-to-image
Base checkpoint: Stable Diffusion XL 1.0
Initial finetuned models: Animagine XL 3.1 and Animagine XL 4.0 Opt
License: CreativeML Open RAIL++-M

Training Pipeline

Step 1: Model Merging
Multistage merging of over 20 anime-specific checkpoints using the following methods:

Karcher mean (Riemannian center of mass): a generalization of the arithmetic mean to curved spaces (Riemannian manifolds), finding the point that minimizes the sum of squared Riemannian distances to all checkpoints.
subtract and add_difference to preserve the unique characteristics of each checkpoint
fallback and task_arithmetic for flexible combination of UNet, VAE, and CLIP components

Step 2: Fine-tuning
A few short iterations on a small but high-quality dataset:

pixiv – a collection of illustrations with high aesthetic scores
latum – expressive artworks with pleasant tones and fine line work
artists – artworks by asou, hara_harayutaka, hiroki_yyqw7151 in a vanilla anime screencap style
aiga – AI-generated images with high aesthetic scores
anime – screenshots and illustrations from anime and hentai

Note: UNet and CLIP were fine-tuned separately due to an initial configuration error.

Styles Reference

You can influence the visual style of generated images by including artist or style tags in your prompt. For a curated list of artist tags that have been tested with Animagus (compiled from the various checkpoints used in the merge), download the reference here:

Download artist list:
https://huggingface.co/rityak/animagus/raw/main/artist_list.txt

Note: This list was gathered from models used during merging and serves as a helpful starting point, but not all tags are guaranteed to work equally well. Feel free to experiment and discover which styles best suit your use case.

Usage Guidelines

The usage guidelines will be the same as for Animagine 3.1 and 4.0. For more details, please refer to:

Recipe

Step 1. Merging Animagus XL Zero - Non-trained merging of checkpoints.

version: 0.1.0
model "SDXL\animagine\animagine-xl-4.0-opt.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\animagine\animagine-xl-3.1.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\babesByStableYogiSDXL_v50FP16.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\bigasp_v20.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\juggernautXL_ragnarokBy.safetensors" model_config="sdxl-sgm" merge_space="weight"
merge "karcher_mean_with_blocks" &2 &3 &4 alpha_clip_l=null alpha_clip_g=null alpha_in=null alpha_mid=null alpha_out=null alpha_embed_time=null alpha_embed_label=null max_iter=30.000000000000007 tol="3e-08" alpha_global="0.4, 0.3, 0.3"
merge "merge_checkpointing" &5
model "SDXL\sd_xl_base_1.0.safetensors" model_config="sdxl-sgm" merge_space="weight"
merge "subtract" &6 &7
literal 0.0 model_config="sdxl-sgm" merge_space="param"
merge "pick_component" &9 "clip_l"
merge "pick_component" &9 "clip_g"
merge "fallback" &10 &11
merge "pick_component" &9 "vae"
merge "fallback" &12 &13
literal 1.0000000000000002 model_config="sdxl-sgm" merge_space="param"
merge "pick_component" &15 "diffuser"
merge "fallback" &14 &16
merge "add_difference" &1 &8 &17
merge "fallback" &18 &1
model "SDXL\tofuANIMEBASEMODEL_050.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\kohakuXLZeta_rev1.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\starryXLV52_v52.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\hassakuXL_betaV06.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\animeIllustDiffusion_v08.safetensors" model_config="sdxl-sgm" merge_space="weight"
merge "karcher_mean_with_blocks" &1 &20 &21 &22 &23 &24 alpha_clip_l=null alpha_clip_g=null alpha_in=null alpha_mid=null alpha_out=null alpha_embed_time=null alpha_embed_label=null max_iter=30.000000000000007 tol="3e-08" alpha_global="0.1675, 0.1665, 0.1665, 0.1665, 0.1665, 0.1665"
merge "merge_checkpointing" &25
merge "subtract" &26 &1
merge "add_difference" &19 &27 1.0
merge "fallback" &28 &19
model "SDXL\animagine\RaeDiffusion-XL-v2.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\animagine\UrangDiffusion-2.0.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\animagine\clandestine-xl-1.0.safetensors" model_config="sdxl-sgm" merge_space="weight"
model "SDXL\animagine\ranimeXLBaseOnAnimagine_v10.safetensors" model_config="sdxl-sgm" merge_space="weight"
merge "task_arithmetic" &1 &30 &31 &32 &33 lambda_=1.0 weights="0.22, 0.26, 0.26, 0.26"
merge "merge_checkpointing" &34
merge "subtract" &30 &1
merge "subtract" &31 &1
merge "subtract" &32 &1
merge "subtract" &33 &1
merge "ties_sum_extended" &36 &37 &38 &39 k=0.20000000000000004 vote_sgn=true apply_stock=false apply_median=true cos_eps=1e-06 eps=1e-06 maxiter=100 ftol=1e-20
merge "add_difference" &35 &40 1.0
merge "fallback" &41 &35
merge "subtract" &42 &1
merge "add_difference" &29 &43 1.0
merge "fallback" &44 &29
merge "karcher_mean" &0 &45 max_iter=30.000000000000007 tol="1e-07" alphas="0.35, 0.65"

Step 2. Fine-tuning Animagus XL A few short iterations on a small, high-quality dataset.

rityak
/

animagus