A machine-learned microlens mask and neural network system for reducing flying pixels in time-of-flight imaging

Background

Time-of-flight (ToF) imaging is a foundational technology in the field of 3D vision, enabling applications ranging from robotics and augmented reality to industrial automation and consumer electronics. ToF cameras estimate depth by measuring the time it takes for emitted light to travel to an object and back to the sensor, producing dense depth maps in real time. However, the accuracy of ToF systems is critically challenged at object boundaries, where abrupt changes in depth occur. These regions are particularly susceptible to artifacts known as “flying pixels,” which arise when a single sensor pixel receives light reflected from both foreground and background surfaces. As a result, the measured depth at these discontinuities is often a spurious value that does not correspond to any physical point in the scene, undermining the reliability of 3D reconstructions and downstream tasks like segmentation or pose estimation.

Current approaches to mitigating flying pixel artifacts typically involve optical or computational tradeoffs. One common method is to reduce the camera’s aperture size, which narrows the angle of incoming light and lessens the mixing of signals from different depths. While this can decrease the prevalence of flying pixels, it also significantly reduces the amount of light reaching the sensor, leading to lower signal-to-noise ratio (SNR) and increased measurement noise, especially in low-light environments. Other strategies, such as post-processing filters or hand-crafted mask patterns, often fail to fully resolve the ambiguity at depth discontinuities or introduce additional artifacts. Consequently, there remains an inherent and unsatisfactory compromise in commodity ToF cameras between minimizing flying pixel errors and maintaining high SNR, limiting their effectiveness in demanding real-world scenarios.

Technology

This technology, known as Mask-ToF, is an advanced system for improving the accuracy of time-of-flight (ToF) imaging by reducing flying pixel artifacts, which are erroneous depth measurements that commonly occur at object boundaries. The core of the solution is a machine-learned microlens amplitude mask, fabricated as a binary spatial pattern and integrated directly onto the surface of a ToF sensor. Each pixel on the sensor is assigned a custom aperture size via the mask, creating a spatially multiplexed array where some pixels are optimized for low flying pixel susceptibility (with smaller apertures) and others for higher signal-to-noise ratio (with larger apertures). A jointly optimized neural network decodes the raw sensor data, leveraging the diverse spatial information from neighboring pixels to accurately separate mixed foreground and background signals. The mask pattern and decoding algorithm are co-designed through a differentiable simulation pipeline, which iteratively refines both components to minimize depth errors while preserving overall light throughput.

What differentiates this technology is its holistic, data-driven approach to overcoming the fundamental tradeoff between flying pixel suppression and signal-to-noise ratio in ToF imaging. Unlike traditional methods that uniformly narrow the camera aperture—thus sacrificing light efficiency for artifact reduction—Mask-ToF uses spatially varying apertures and a neural network that exploits the resulting diversity in pixel responses. This joint optimization enables the system to halve flying pixel counts compared to standard open aperture designs, without increasing noise or reducing light collection. The mask is manufacturable using standard lithography techniques and can be integrated into existing ToF sensor fabrication processes, ensuring practical scalability and minimal impact on device form factor or cost. Additionally, the decoding network is lightweight enough for real-time implementation on standard image signal processors, making the solution suitable for a wide range of 3D vision applications where high-fidelity depth sensing is critical.

Benefits

• Significantly reduces flying pixel artifacts at object boundaries, improving depth accuracy in time-of-flight imaging.
• Maintains high signal-to-noise ratio by spatially varying pixel apertures, avoiding the traditional tradeoff between noise and flying pixel count.
• Jointly optimized microlens amplitude mask and neural network decoding enable effective disambiguation of mixed foreground and background signals.
• Validated both in simulation and real-world prototypes, demonstrating robust generalization without retraining.
• Compatible with standard CMOS sensor fabrication and can be integrated into commodity ToF devices with minimal form factor or cost impact.
• Enables real-time depth refinement through on-board computational decoding, supporting practical applications like segmentation, pose estimation, and classification.
• Improves edge fidelity and 3D imaging quality, especially in challenging scenes with object discontinuities.

Applications

• 3D object segmentation for robotics
• Augmented reality depth sensing
• Automotive driver-assistance systems
• Industrial machine vision inspection
• Biometric facial recognition

Additional info

This system integrates a machine-learned microlens amplitude mask onto time-of-flight sensors to eliminate flying pixel artifacts. A spatially multiplexed binary pattern assigns unique aperture sizes to pixels, balancing signal-to-noise ratios with depth precision. A jointly optimized neural network decodes measurements, using spatial data from adjacent pixels to resolve mixed light signals. This differentiable pipeline minimizes depth errors while maintaining light throughput for high-fidelity 3D imaging.