UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations

Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts.

Website  |   Paper  |   Github  

• UniSkill is a universal skill representation learning approach that enables the use of large-scale video data by removing the need for labels or any form of alignment constraints.

• UniSkill shows effective human-to-robot and robot-to-robot imitation in both simulation and real-world experiments through its embodiment-agnostic skill representation.

Model Summary

• Developed by: Yonsei University
• Model type: image generation (language, image => robot actions)
• License: MIT
• Finetuned from: InstructPix2Pix
• Website: https://kimhanjung.github.io/UniSkill/
• Paper: https://arxiv.org/abs/2505.08787
• Code: https://github.com/KimHanjung/UniSkill

Uses

UniSkill is designed to learn cross-embodiment skill representations from large-scale human and robot video datasets. Its embodiment-agnostic skill representations can be used in various ways, such as training policies and generating sub-goal images.

License

The model is licensed under the MIT license.