Enhance model card for Look, Focus, Act robot learning model

README.md

@@ -1,10 +1,87 @@
 ---
+license: apache-2.0
 tags:
 - model_hub_mixin
 - pytorch_model_hub_mixin
+- lerobot
+pipeline_tag: robotics
+library_name: lerobot
 ---
 
-This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
-- Code: [More Information Needed]
-- Paper: [More Information Needed]
-- Docs: [More Information Needed]
+# Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers
+
+This repository contains the `gaze_model_av_aloha_sim_cube_transfer` model, a pre-trained gaze model for the AV-ALOHA simulation environment. It is part of the research presented in the paper [Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://huggingface.co/papers/2507.15833).
+
+This work proposes a human-inspired *foveated vision framework* for robot learning that combines human gaze, foveated Vision Transformers (ViTs), and robotic control to enable policies that are both efficient and robust.
+
+[\ud83d\udcda Paper](https://huggingface.co/papers/2507.15833) | [\ud83c\udf10 Project Website](https://ian-chuang.github.io/gaze-av-aloha/) | [\ud83d\udcbb Code](https://github.com/ian-chuang/gaze-av-aloha)
+
+<p align="center">
+  <img src="https://github.com/ian-chuang/gaze-av-aloha/raw/main/media/hero.gif" alt="Hero GIF" width="100%">
+</p>
+
+## Abstract
+Human vision is a highly active process driven by gaze, which directs attention and fixation to task-relevant regions and dramatically reduces visual processing. In contrast, robot learning systems typically rely on passive, uniform processing of raw camera images. In this work, we explore how incorporating human-like active gaze into robotic policies can enhance both efficiency and performance. We build on recent advances in foveated image processing and apply them to an Active Vision robot system that emulates both human head movement and eye tracking. Extending prior work on the AV-ALOHA robot simulation platform, we introduce a framework for simultaneously collecting eye-tracking data and robot demonstrations from a human operator as well as a simulation benchmark and dataset for training robot policies that incorporate human gaze. Given the widespread use of Vision Transformers (ViTs) in robot learning, we integrate gaze information into ViTs using a foveated patch tokenization scheme inspired by recent work in image segmentation. Compared to uniform patch tokenization, this significantly reduces the number of tokens-and thus computation-without sacrificing visual fidelity near regions of interest. We also explore two approaches to gaze imitation and prediction from human data. The first is a two-stage model that predicts gaze to guide foveation and action; the second integrates gaze into the action space, allowing the policy to jointly predict gaze and actions end-to-end. Our results show that our method for foveated robot vision not only drastically reduces computational overhead, but also improves performance for high precision tasks and robustness to unseen distractors. Together, these findings suggest that human-inspired visual processing offers a useful inductive bias for robotic vision systems.
+
+## Installation
+
+To work with this model within the `gaze-av-aloha` framework, you need to set up the environment by following the instructions from the official GitHub repository:
+
+```bash
+# Clone the repository and initialize submodules
+git clone https://github.com/ian-chuang/gaze-av-aloha.git
+cd gaze-av-aloha
+git submodule init
+git submodule update
+
+# Create and activate a new Conda environment
+conda create -n gaze python=3.10
+conda activate gaze
+
+# Install LeRobot
+pip install git+https://github.com/huggingface/lerobot.git@483be9aac217c2d8ef16982490f22b2ad091ab46
+
+# Install FFmpeg for video logging
+conda install ffmpeg=7.1.1 -c conda-forge
+
+# Install AV-ALOHA packages
+pip install -e ./gym_av_aloha
+pip install -e ./gaze_av_aloha
+```
+
+## Sample Usage
+
+This model (`iantc104/gaze_model_av_aloha_sim_cube_transfer`) is a pre-trained gaze prediction model. It is designed to be integrated into the `gaze-av-aloha` framework's policy training scripts. For example, to train a robot policy using this pre-trained gaze model (following the Fov-UNet approach) for the `cube_transfer` task, you would use a command similar to the following:
+
+```bash
+python gaze_av_aloha/scripts/train.py \
+  policy=foveated_vit_policy \
+  task=av_aloha_sim_cube_transfer \
+  policy.use_gaze_as_action=false \
+  policy.gaze_model_repo_id=iantc104/gaze_model_av_aloha_sim_cube_transfer \
+  policy.vision_encoder_kwargs.repo_id=iantc104/mae_vitb_foveated_vit \
+  policy.optimizer_lr_backbone=1e-5 \
+  wandb.enable=true \
+  wandb.project=<your_project_name> \
+  wandb.entity=<your_wandb_entity> \
+  wandb.job_name=fov-unet \
+  device=cuda
+```
+
+For more detailed usage, alternative policy training methods (like Fov-Act), and other available tasks, please refer to the comprehensive documentation and scripts in the official [gaze-av-aloha GitHub repository](https://github.com/ian-chuang/gaze-av-aloha).
+
+## Citation
+
+If you find this work useful for your research, please consider citing the original paper:
+
+```bibtex
+@misc{chuang2025lookfocusactefficient,
+      title={Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers},
+      author={Ian Chuang and Andrew Lee and Dechen Gao and Jinyu Zou and Iman Soltani},
+      year={2025},
+      eprint={2507.15833},
+      archivePrefix={arXiv},
+      primaryClass={cs.RO},
+      url={https://arxiv.org/abs/2507.15833},
+}
+```