Role of Transients in Two-Bounce Non-Line-of-Sight Imaging
CVPR 2023, Vancouver, Canada
- Siddharth Somasundaram MIT
- Akshat Dave Rice
- Connor Henley MIT
- Ashok Veeraraghavan Rice
- Ramesh Raskar MIT
Abstract
The goal of non-line-of-sight (NLOS) imaging is to image objects occluded from the camera's field of view using multiply scattered light. Recent works have demonstrated the feasibility of two-bounce (2B) NLOS imaging by scanning a laser and measuring cast shadows of occluded objects in scenes with two relay surfaces. In this work, we study the role of time-of-flight (ToF) measurements, i.e. transients, in 2B-NLOS under multiplexed illumination. Specifically, we study how ToF information can reduce the number of measurements and spatial resolution needed for shape reconstruction. We present our findings with respect to tradeoffs in (1) temporal resolution, (2) spatial resolution, and (3) number of image captures by studying SNR and recoverability as functions of system parameters. This leads to a formal definition of the mathematical constraints for 2B lidar. We believe that our work lays an analytical groundwork for design of future NLOS imaging systems, especially as ToF sensors become increasingly ubiquitous.
[Prior Work] Two-Bounce NLOS Imaging Using Individual Shadows
In a two-bounce NLOS configuration, a hidden object lies behind an occluder but is surrounded by two relay surfaces. By scanning a laser point across one relay surface and measuring cast shadows on the other wall, it is possible to reconstruct the 3D shape of the occluded object.
[Our Work] Two-Bounce NLOS Imaging with Multiplexed Shadows
We obviate the need for laser scanning inherent to prior NLOS methods by using multiplexed illumination. We use a single-photon avalanche diode (SPAD) to demultiplex light paths based on their time of flight. Our method is a promising avenue for few-shot two-bounce NLOS imaging.
Linear Forward Model w.r.t. Shadow Transients
We decompose the measured time-of-flight profile into two transient profiles: light transients and shadow transients. The sum of these two transients is the empty transient, which is what would have been observed by the sensor if there was no hidden object. The light transient measures light that arrives at the sensor (i.e. what the sensor measures), whereas the shadow transient measures light that doesn't arrive at the sensor because it was blocked by the hidden object. The above animations show what the empty transient, light transient, and shadow transient would look like under flash illumination. Note that all aforementioned transients are two-bounce transients. A linear model can be derived in terms of the shadow transients in the form
Backprojection Reconstruction
Simulated Results: Effect of Temporal Resolution
Experimental Results: Number of Measurements vs. Temporal Resolution
Experimental Results: Spatial Resolution vs. Temporal Resolution
Applications
There are many real-world situations with two relay surfaces in which one would want to navigate their surroundings quickly. For example, imagine an autonomous vehicle being able to see in front of the car ahead of it by using two-bounce light off the sides of a tunnel. We believe that the ideas introduced in this paper will inspire future work in few-shot NLOS imaging.