A joint hardware-software system for optimizing structured illumination and depth reconstruction in active stereo 3D imaging

Background

Active stereo 3D imaging is a critical technology in fields such as robotics, autonomous vehicles, augmented reality, and mobile devices, where accurate depth perception is essential for navigation, object recognition, and interaction with the environment. These systems typically employ a pair of cameras and a structured light projector to cast patterns onto the scene, enabling reliable correspondence matching even on surfaces lacking natural texture. The demand for robust 3D sensing has grown as applications increasingly require operation in diverse and challenging environments, including those with strong ambient illumination, low-reflectance materials, or dynamic, cluttered scenes. As a result, there is a pressing need for depth sensing solutions that can maintain high accuracy and reliability under a wide range of real-world conditions.

Despite their widespread use, current active stereo systems face significant limitations. Traditional approaches rely on hand-crafted illumination patterns—such as pseudo-random dot arrays or regular grids—designed independently of the specific scene characteristics, environmental lighting, or the reconstruction algorithms used. This lack of joint optimization often results in suboptimal performance, particularly in scenarios with high ambient light, reflective or transparent surfaces, or occlusions. Furthermore, the separation of hardware and software design means that the system cannot adapt its illumination strategy to varying conditions or fabrication constraints, leading to reduced depth accuracy and robustness. Existing methods also struggle to resolve ambiguities in textureless or occluded regions, and are often unable to leverage advances in machine learning for end-to-end system improvement. These challenges highlight the need for more integrated, adaptive, and learning-based approaches to active stereo 3D imaging.

Technology

This technology is an advanced active stereo 3D imaging system that integrates the joint optimization of structured illumination patterns and depth reconstruction algorithms within a single, end-to-end differentiable framework. At its core, the system employs a diffractive optical element (DOE) designed through wave-optics simulations to project learned “Polka Lines” illumination patterns, which are specifically tailored to environmental factors such as ambient light and surface reflectance. The imaging setup consists of two near-infrared stereo cameras and a laser projector equipped with the DOE. A trinocular neural network processes the stereo camera images alongside the known illumination pattern, constructing dual cost volumes for both wide and narrow baselines. This enables robust depth estimation, even in challenging scenarios with textureless surfaces or occlusions. The system’s hybrid image formation model simulates the entire optical pipeline—from laser propagation and DOE modulation to scene interaction and camera capture—allowing for gradient-based optimization of both the DOE phase profile and the neural network, with further fine-tuning enabled by self-supervised learning on real-world data.

What differentiates this technology is its holistic, data-driven approach to optimizing both hardware and software components for 3D sensing. Unlike traditional active stereo systems that rely on hand-crafted, static illumination patterns and separately designed reconstruction algorithms, this solution jointly learns the optimal illumination pattern and depth estimation network in a closed loop, tailored to specific environmental and operational constraints. The differentiable simulation of the full imaging process—including wave-optics, geometric optics, and environmental factors—enables precise, environment-specific DOE designs that maximize depth accuracy and robustness to noise, ambient light, and surface variability. The trinocular neural network’s dual cost volume fusion further enhances performance in difficult regions, such as occlusions and low-texture areas. Experimental results demonstrate that this system outperforms commercial and random-pattern baselines, achieving real-time, high-accuracy depth estimation in diverse conditions, and providing a flexible foundation for next-generation 3D imaging applications.

Benefits

• Enhanced depth estimation accuracy through joint optimization of illumination patterns and reconstruction algorithms.
• Robust performance in challenging environments including textureless, specular, transparent surfaces, and strong ambient light conditions.
• Adaptive illumination patterns tailored to specific environments (indoor, outdoor, or general use) for improved sensing reliability.
• Use of a trinocular neural network leveraging stereo camera inputs and known illumination for superior disparity resolution, especially near occlusions.
• End-to-end differentiable hybrid image formation model enabling precise simulation and optimization of optical elements and imaging pipeline.
• Self-supervised fine-tuning capability to adapt the system to real-world conditions and fabrication variances.
• Real-time depth estimation suitable for applications like robotics, autonomous vehicles, and augmented reality.
• Flexible DOE design process allowing fabrication of customized structured light patterns for enhanced 3D sensing performance.

Applications

• Autonomous vehicle obstacle detection
• Robotic navigation in dynamic environments
• Augmented reality spatial mapping
• Mobile device 3D scanning
• Industrial quality control inspection

Additional info

This active stereo 3D system employs an end-to-end differentiable framework to jointly optimize structured illumination and depth reconstruction. A diffractive optical element projects learned “Polka Lines” patterns, while a trinocular neural network processes stereo inputs and illumination data. Integrating wave and geometric optics, the model enables gradient-based optimization of the phase profile and reconstruction algorithm, facilitating robust depth estimation in textureless, occluded, or high-ambient light environments.