Kuangji Zuo * , Gen Li * , Bofan Lyu , Yanshuo Lu , Boyu Ma , Shijia Han , Xinyu Zhou , Xichen Yuan , Chuhao Zhou , Jiaqi Bai , Geng Li , Jianfei Yang †
MARS Lab, Nanyang Technological University, Singapore
* Equal contribution. † Corresponding author.
Human gaze acts as a natural, low-effort intent interface for human-in-the-loop VLA control, guiding object-level disambiguation, part-level interaction, and dynamic intent steering.
Language tells the robot what to do. Gaze tells it which object or part you mean, without extra commands.
"Pick up the cup" is underspecified when there are three similar cups, transparent objects, or a tool whose handle, head, and neck imply different actions. Language gives the robot the task, but it often leaves the intended object, part, or target state unresolved.
Human gaze is a natural signal for this missing intent. In everyday collaboration, people disambiguate references by looking, not by issuing extra spatial commands. Gaze2Act brings that HRI cue into VLA manipulation: the user keeps language for the high-level instruction, while gaze supplies the object-level, part-level, and dynamic intent needed for grounded robot action.
Gaze2Act keeps language for the high-level task, while gaze specifies the target referent at inference time. It first uses visual foundation models to ground first-person gaze in the robot view without task-specific training or camera calibration, then injects the grounded gaze signal at both the perception level and the action level.
Uses visual foundation models to translate first-person gaze into a robot-view target mask and fixation point, without task-specific training, camera calibration, or external markers.
Renders the grounded gaze signal onto the robot observation, using object contours for coarse target selection and heatmaps for fine-grained interaction cues.
Feeds gaze-derived spatial features directly into the DiT action head through a zero-initialized decoupled cross-attention branch.
We evaluate Gaze2Act on a real Unitree G1 across seven settings: 15 static manipulation tasks covering object-level intent, compositional intent, and part-level intent, plus a separate dynamic steering setting.
2 or 3 near-identical objects, one target
instances never seen in training
no texture for a detector to grab
object and placement target together
handle vs. head vs. neck of one tool
the part you look at changes the action
switch the target mid-execution
For each task, we report intent accuracy, whether the policy reaches the intended target, and task success, whether the full manipulation is completed. For language-derived baselines, precise object descriptions or grounding prompts are provided to obtain the intended object before policy execution.
Part intent is not applicable to methods without a part-specific condition. Their detector-derived object masks remain the same no matter whether the user intends the handle, head, neck, or body, so they can report task success but do not expose a part-level target signal.
Dynamic steering is trained from 57 demonstrations and evaluated on 30 target-switch trials. The robot has already started moving when the user changes intent, so the policy must revise an ongoing action rather than choose a target from rest. The setting is difficult because the cups are visually similar, the human egocentric gaze view and the robot's exocentric camera view differ substantially, nearby and distant cups appear at different image scales, and the moving hand can partially occlude the target during re-grounding.
In this setting, updating a language-derived mask is often not salient enough to redirect an action already in progress; Gaze2Act keeps the updated gaze target active through both perception-level prompting and action-level conditioning.
Success count over 30 trials where the intended cup changes during execution.
We use Pick Bread Place Bowl and Hammer parts to separate two roles of gaze: selecting object/placement targets and preserving fine-grained contact points. Gaze prompting makes the intended region explicit in the visual input, while gaze conditioning keeps that spatial intent available during action generation. The full model performs best on both tasks, showing that the two pathways are complementary: prompting tells the policy where to look, and conditioning helps it act consistently on that intent.
Values are success rates converted from 60 trials per ablation setting; no error bars are shown.
On a Pick Bread Place Bowl task with distractors, the baseline attention remains spread across the scene even with a specific language instruction. Gaze2Act uses the generic outlined-object instruction, but gaze prompting marks the selected regions and action-level conditioning keeps these spatial cues active during denoising, producing a more concentrated focus over the intended object and placement target.
