St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World

@inproceedings{st4rtrack2025,
  title={St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World},
  author={Feng*, Haiwen and Zhang*, Junyi and Wang, Qianqian and Ye, Yufei and Yu, Pengcheng and Black, Michael J. and Darrell, Trevor and Kanazawa, Angjoo},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2025}
}

Model info

Github page: https://github.com/HavenFeng/St4rTrack-Release
Project page: https://st4rtrack.github.io/
Paper: https://arxiv.org/abs/2504.13152

How to Use

First install St4rTrack. To load the model (Seq):

from dust3r.models import AsymmetricCroCo3DStereo

# Loads default Seq checkpoint
model = AsymmetricCroCo3DStereo.from_pretrained("yupengchengg147/St4rTrack")

run following code to load checkpoint trained with Pair Mode:

from huggingface_hub import hf_hub_download
import tempfile
import os
from dust3r.model import AsymmetricCroCo3DStereo

# Create a temporary directory for the model files
temp_dir = os.path.join(tempfile.gettempdir(), "st4rtrack_pair")
os.makedirs(temp_dir, exist_ok=True)

# Download the config and model files from the Pair subfolder
config_path = hf_hub_download(
    repo_id="yupengchengg147/St4rTrack",
    filename="Pair/config.json",
    cache_dir=temp_dir
)
# Load the model from the downloaded path
model_dir = os.path.dirname(config_path)
model = AsymmetricCroCo3DStereo.from_pretrained(model_dir)