depth-anything/camera-depth-model-zed2i-quality

Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots

This repository contains the Camera Depth Models (CDMs) presented in the paper Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots.

Project Page: https://manipulation-as-in-simulation.github.io/ Code: https://github.com/ByteDance-Seed/manip-as-in-sim-suite

Abstract

Modern robotic manipulation primarily relies on visual observations in a 2D color space for skill learning but suffers from poor generalization. In contrast, humans, living in a 3D world, depend more on physical properties-such as distance, size, and shape-than on texture when interacting with objects. Since such 3D geometric information can be acquired from widely available depth cameras, it appears feasible to endow robots with similar perceptual capabilities. Our pilot study found that using depth cameras for manipulation is challenging, primarily due to their limited accuracy and susceptibility to various types of noise. In this work, we propose Camera Depth Models (CDMs) as a simple plugin on daily-use depth cameras, which take RGB images and raw depth signals as input and output denoised, accurate metric depth. To achieve this, we develop a neural data engine that generates high-quality paired data from simulation by modeling a depth camera's noise pattern. Our results show that CDMs achieve nearly simulation-level accuracy in depth prediction, effectively bridging the sim-to-real gap for manipulation tasks. Notably, our experiments demonstrate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots on two challenging long-horizon tasks involving articulated, reflective, and slender objects, with little to no performance degradation. We hope our findings will inspire future research in utilizing simulation data and 3D information in general robot policies.

Key Features

• Sim-to-Real Depth Transfer: Clean, metric depth estimation that matches simulation quality
• Multi-Camera Support: Pre-trained models for various depth sensors (RealSense, ZED, Kinect)
• Automated Data Generation: Scalable demonstration generation using enhanced MimicGen
• Whole-Body Control: Unified control for mobile manipulators for mimicgen
• Multi-GPU Parallelization: Distributed simulation for faster data collection
• VR Teleoperation: Intuitive demonstration recording using Meta Quest controllers

Usage

This section provides an example of how to run depth inference using the Camera Depth Model (CDM). For more details, refer to the Model inference guide in the GitHub repository.

To run depth inference on RGB-D camera data, use the following command:

cd cdm
python infer.py \
    --encoder vitl \
    --model-path /path/to/model.pth \
    --rgb-image /path/to/rgb.jpg \
    --depth-image /path/to/depth.png \
    --output result.png

Citation

If you use this work in your research, please cite:

@article{liu2025manipulation,
  title={Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots},
  author={Liu, Minghuan and Zhu, Zhengbang and Han, Xiaoshen and Hu, Peng and Lin, Haotong and 
          Li, Xinyao and Chen, Jingxiao and Xu, Jiafeng and Yang, Yichu and Lin, Yunfeng and 
          Li, Xinghang and Yu, Yong and Zhang, Weinan and Kong, Tao and Kang, Bingyi},
  journal={arXiv preprint},
  year={2025}
}