pi05-beat-drum-seedance100-v1

A vision-language-action (VLA) policy for a bimanual AgileX Piper arm setup, fine-tuned from the π0.5 (pi05_base) foundation model. Given camera images and a natural-language instruction, it outputs a 50-step chunk of end-effector actions for both arms.

The task it was trained on is beating a drum: the robot must locate a drum in the scene and strike it with a stick held in its gripper.

Why this checkpoint exists

Collecting real teleoperated robot demonstrations is slow and expensive. This model is part of a sim2video line of work: instead of recording real demonstrations, trajectories are produced in simulation and then re-rendered into photorealistic video with a video generation model (here, Seedance), so the policy trains on imagery that looks like the real world without ever recording it. This checkpoint is the policy that comes out the other end of that pipeline.

The interesting question this checkpoint is evidence for is whether a VLA trained purely on generated video transfers to a real robot. See Evaluation status before drawing conclusions — that question is not answered by anything published here.

Model at a glance

Base model	π0.5 (pi05_base, openpi)
Architecture	PaliGemma VLM backbone + flow-matching action expert
Robot	Bimanual AgileX Piper (two 6-DoF arms + 2 grippers)
Task	Beat a drum
Training steps	50,000 (this is the final checkpoint, step=49999)
Global batch size	128
Precision / framework	JAX, Orbax checkpoint (OCDBT)
Params size	~12 GB

Inputs and outputs

Inputs

• Images — three independent camera views, each 224×224 RGB.
• Language instruction — a natural-language prompt, tokenized with the PaliGemma tokenizer, up to 200 tokens.
• Robot state — ⚠️ not used. This policy was trained with state conditioning disabled (--model.no-discrete-state-input; the training manifest records state_conditioning: none). The policy is vision- and language-only: it does not read joint encoders. A norm_stats.json with 14-dim state statistics ships with the checkpoint because the data pipeline computes it, but the model does not consume the state field.

Outputs

• An action chunk: 50 future timesteps × 14 active dimensions.
• The 14 dimensions are, per arm (left then right): 3 end-effector translation + 3 end-effector rotation + 1 gripper.
• The model's internal action vector is padded to 32 dimensions; dimensions 14–31 are padding and must be discarded.

Actions are normalized. You must un-normalize them with the shipped assets/.../norm_stats.json before sending anything to a robot.

Repository layout

.
├── params/                        # Orbax OCDBT checkpoint — the weights (~12 GB)
├── assets/
│   └── agentic-sim2video/
│       └── piper_beat_drum_20260923_barrel_seedance100_inline224_v1/
│           └── norm_stats.json    # action/state mean, std, q01, q99 — REQUIRED
└── _CHECKPOINT_METADATA

params/ is a JAX/Orbax checkpoint, not safetensors — it will not load with transformers. You need openpi.

Note: only inference weights are published. The optimizer state (train_state/, ~31 GB) was not uploaded, so you can run this policy but cannot resume training from step 49999.

Usage

from huggingface_hub import snapshot_download

ckpt = snapshot_download("xiwenyoumu/pi05-beat-drum-seedance100-v1")

Then load it with openpi, using the training config this checkpoint was produced under:

from openpi.training import config as _config
from openpi.policies import policy_config

cfg = _config.get_config("pi05_piper_dual_beat_drum_eef_xyz3d_100")
policy = policy_config.create_trained_policy(cfg, ckpt)

action_chunk = policy.infer({
    # three 224x224x3 uint8 views — key names must match your config's data transform
    "images": {...},
    "prompt": "beat the drum",
})["actions"]          # (50, 32) -> use [:, :14], then un-normalize

The config name pi05_piper_dual_beat_drum_eef_xyz3d_100 refers to a project-local openpi config that is not part of upstream openpi. To use this checkpoint you will need an equivalent config declaring: pi05=True, action_dim=32, action_horizon=50, max_token_len=200, discrete state input disabled, and a Piper bimanual EEF policy transform over three camera views.

Training details

Hyperparameter	Value
Initialized from	pi05_base params (openpi assets)
Training steps	50,000
Global batch size	128
Hardware	8 GPUs, FSDP across 8 devices
Optimizer	AdamW — β₁ 0.9, β₂ 0.95, ε 1e-8, weight decay 1e-10
Gradient clipping	global norm 1.0
LR schedule	warmup 1,000 steps → peak 2.5e-5 → cosine decay over 30,000 steps → 2.5e-6
Seed	42
Action horizon	50
Action dim	32 (14 active, 18 padding)
Max token length	200
Samples seen	~6.39M

Training data — an internal dataset, agentic-sim2video/piper_beat_drum_20260923_barrel_seedance100_inline224_v1: simulated bimanual drum-beating trajectories re-rendered into photorealistic video with Seedance, at 224×224, three views. This dataset is not public.

Final training loss — ≈0.00115 (flow-matching loss, mean over the last logged steps). Per-dimension losses were highest on right-arm rotation (≈0.006) and lowest on translation and gripper dimensions (≈0.001).

Evaluation status

Read this before using or citing this model.

• ❌ No held-out evaluation was run. The periodic "eval" during training was computed on a fixed first training batch with a fixed RNG — the training manifest records it verbatim as "fixed first transformed batch with fixed RNG; not held-out". It is a numerical-stability probe, not a measure of generalization.
• ❌ No real-robot success rate is reported here. Nothing in this repository demonstrates that the policy beats a drum on physical hardware.
• ❌ No simulation benchmark score is reported here.
• ✅ What is verified: training ran to completion for 50,000 steps with finite loss and gradients throughout, and the loss converged to ≈0.00115.

Low training loss on generated video says the policy fits its training distribution. It does not establish sim2real transfer. Treat this checkpoint as a research artifact, not a validated controller.

Limitations and intended use

• Single task, single embodiment. Trained only on drum-beating with a specific bimanual Piper setup, specific camera placement, and 224×224 three-view input. It is not a general-purpose manipulation policy and should not be expected to follow arbitrary instructions.
• Trained on generated, not real, video. Systematic artifacts of the video generator are baked into what the policy expects to see. Real-world appearance gaps are an open risk.
• No state feedback. Being vision-only, the policy cannot recover from a mismatch between where it believes the arms are and where they actually are.
• Safety. This model drives physical robot arms. Any real-hardware deployment needs workspace limits, velocity/force limits, and an accessible emergency stop. Do not run it on hardware near people without those safeguards.

License

Released under Apache-2.0, following the license of the upstream openpi π0.5 base model it was fine-tuned from.

Acknowledgements

Built on openpi and the π0.5 base model by Physical Intelligence.