Hierarchical robotic control system for filtering generated visual subgoals and executing robust actions

Background

The field of hierarchical imitation learning focuses on enabling robots to perform complex tasks by learning control policies from visual data and language instructions. In this domain, robotic systems increasingly rely on generative artificial intelligence, such as diffusion and video prediction models, to translate human commands into actionable steps. There is a critical need for these systems to autonomously break down complex instructions into intermediate visual subgoals. This hierarchical framework is essential because it bridges the gap between abstract language and the precise, low-level physical actions a robot must execute in dynamic environments.

Despite the promise of generative models, current approaches struggle significantly with execution reliability. Existing systems typically feed generated visual subgoals directly to low-level control policies without prior validation. Because generative models are prone to hallucination, they frequently produce irrelevant, physically impossible, or visually inconsistent subgoals. Furthermore, there is a severe distribution shift between synthetic generated images and actual real-world camera observations. Current low-level policies are highly brittle when encountering these synthetic visual artifacts. Consequently, the unvalidated transfer of flawed data causes the low-level policy to fail, severely limiting the robot’s ability to complete physical tasks.

Technology

This hierarchical robotic control system enhances imitation learning by utilizing a dual-policy framework to execute language-instructed tasks. A high-level policy leverages image-editing diffusion or video prediction models to generate multiple candidate visual subgoals based on the current environment and language commands. To translate these subgoals into physical actions, a low-level control policy dictates the robot’s movements. A core feature of this solution is its contrastive classifier, which evaluates candidate subgoals against the current state. The classifier filters out irrelevant images and selects the single subgoal most likely to achieve meaningful task progress for execution.

This solution is differentiated by its unique approach to mitigating synthetic image artifacts and distribution shifts, which frequently cause failures in traditional generative robotic controls. The subgoal filter is uniquely trained on successful trajectories and complex negative examples, such as reversed states or mismatched instructions, ensuring highly reliable selection. Furthermore, the system employs image augmentation de-synchronization during training. By applying random visual transformations independently to the current state and goal images, the control policy becomes highly robust. This prevents the robot from being derailed by visual inconsistencies between generated subgoals and real-world camera observations.

Benefits

• Enhances overall execution reliability by actively filtering out hallucinated, incorrect, or irrelevant subgoals before the robot takes action.
• Increases system robustness to visual artifacts and distribution shifts between AI-generated subgoals and real-world camera observations.
• Significantly improves task success rates, outperforming prior state-of-the-art methods particularly in challenging, cluttered physical environments.
• Boosts the resilience of the low-level control policy against errors and inconsistencies through de-synchronized image augmentation during training.
• Facilitates more generalizable and scalable hierarchical imitation learning for complex robotic tasks using large-scale generative models.

Applications

• Autonomous warehouse order picking
• Flexible manufacturing assembly robotics
• Automated retail shelf stocking
• Domestic chore assistance robots
• Hazardous material handling robots

Additional info

This hierarchical robotic control system generates visual subgoals via image-editing diffusion models. A contrastive classifier filters candidates by predicting task progress. To ensure robustness, the low-level policy utilizes de-synchronized data augmentation, applying independent transformations to state and goal images. This approach mitigates synthetic artifacts and distribution shifts, enhancing reliability during real-world execution by bridging generative vision models with grounded physical control policies.