SLIDE 1
1
2
CS-ToF: High-resolution Compressive time-of-flight imaging Fengqiang - - PowerPoint PPT Presentation
CS-ToF: High-resolution Compressive time-of-flight imaging Fengqiang - - PowerPoint PPT Presentation
CS-ToF: High-resolution Compressive time-of-flight imaging Fengqiang Li, Chia-kai Yeh, Kuan He, Oliver Cossairt (Northwestern University) Huaijin Chen, Adithya Pediredla, Ashok Veeraraghavan (Rice University) fengqiang.li@u.northwestern.edu 1
SLIDE 2
SLIDE 3
3
Time-of-flight (ToF) imaging
Goyer et al., Bulletin of the American Meteorological Society, 1963
Pulsed laser Detector Object Clock
SLIDE 4
4
Time-of-flight (ToF) imaging
Goyer et al., Bulletin of the American Meteorological Society, 1963
Pulsed laser Detector Object d Clock
SLIDE 5
5
Time-of-flight (ToF) imaging
Goyer et al., Bulletin of the American Meteorological Society, 1963
Pulsed laser Detector Object d
d = 1 2 ⋅ c ⋅ t
Clock
SLIDE 6
6
Time-of-flight (ToF) imaging
Goyer et al., Bulletin of the American Meteorological Society, 1963
Pulsed laser Detector Object d
d = 1 2 ⋅ c ⋅ t
Clock
Pulsed light based ToF
q Raster scanning
SLIDE 7
7
Continuous-wave ToF
Laser diode ToF camera Controller
!(#) %(#)
Schwarte et al., SPIE Proceeding, 1997
# = 0 !(#) r(#)
SLIDE 8
8
Continuous-wave ToF
Laser diode ToF camera Controller
!(#) %(#)
p Schwarte et al., SPIE Proceeding, 1997
# = 0 !(#) s(#) r(#) () * ( # )
()
SLIDE 9
9
SLIDE 10
10
Robotics Autonomous cars HCI VR/AR
SLIDE 11
11
Robotics Autonomous cars HCI VR/AR Non-Line-of-Sight imaging Transient imaging
SLIDE 12
12
Low spatial resolution
- 640✕480 pixels (0.3 mega) vs 120-megapixel CMOS
SLIDE 13
13
Low spatial resolution
- 640✕480 pixels (0.3 mega) vs 120-megapixel CMOS
- Extra circuits for each pixel
- Limited wafer size
SLIDE 14
14
Work directly on ToF camera output:
- Edge guidance
- Defocus debluring
Fuse ToF output with a second camera:
- RGB camera
- Stereo, Photometric stereo, Shape from polarization
Previous work
SLIDE 15
15
Optical multiplexing
Objects Spatial light modulator-SLM (High resolution) ToF camera (low resolution)
q Multiple Pixels (e.g. 3✕3) on spatial light modulator projected on One Pixel of ToF camera
SLIDE 16
16
Motivation for optical multiplexing
q Higher resolution compared to algorithm based method q No need to fusion with different image modalities q Resolution is dependent on SLM
SLIDE 17
17
Pattern on modulator (M) ToF output (y)
Compressive sensing … …
SLIDE 18
18
Phasor
Intensity (a) depth (d) q ToF output: intensity (a) and depth (d) q Phasor: a ∘ e$%&'
( ) = a ∘ e$+
SLIDE 19
19
Imaging forward model
x = !" ∘ e%&' q High resolution scene projection on modulator (x)
(
SLIDE 20
20
Imaging forward model
x = !" ∘ e%&' q High resolution scene projection on modulator (x) q Spatial light modulator pattern (M)
M
()
SLIDE 21
21
Imaging forward model
x = !" ∘ e%&' q High resolution scene projection on modulator (x) q Spatial light modulator pattern (M) q Translation matrix from modulator to ToF camera (C)
M C
()*
SLIDE 22
22
Imaging forward model
x = !" ∘ e%&' q High resolution scene projection on modulator (x) q Spatial light modulator pattern (M) q Translation matrix from modulator to ToF camera (C) q ToF camera output (y)
M
( = *+,
C
y = !- ∘ e%&.
SLIDE 23
23
Imaging forward model
!" !# !$ . . !& = (" (# ($ . . (& )
* = +,) = ()
q ToF Output y. with modulation pattern M. 01234567 01834569 01:3456; . . 01<3456= = ("0>2345?7 (#0>8345@9 ($0>:345@; . . (&0><345@A = (" (# ($ . . (& 0>345@
SLIDE 24
24
Optimization
! x = arg min
*
+ − -. / + 1Φ(x) Φ . = TV . = 7
8
G: x8
/ + G; x8 /
SLIDE 25
25
Simulation
q High resolution scene
- Middlebury 3D Datasets
- Size: 1140x912
q At – simulated system A = CMt
- Mt – Hadamard multiplexing patterns on DMD at t-th measurement
- C – mapping matrix, defined as spatial down-sampling by averaging
q Low resolution measurements
- Simulated through via yt = Atx
- Size: 120×153
SLIDE 26
26
(b). Original LR ToF measurement Intensity (a). Ground truth Intensity (c). HR reconstruction (50%) Intensity (d). HR reconstruction (25%) Intensity (e). HR reconstruction (15%) Intensity
SLIDE 27
27
(b). Original LR ToF measurement Intensity (a). Ground truth Intensity (c). HR reconstruction (50%) Intensity (d). HR reconstruction (25%) Intensity (e). HR reconstruction (15%) Intensity (g). Original LR ToF measurement Phase (f). Ground truth Phase (h). HR reconstruction (50%) Phase (i). HR reconstruction (25%) Phase (j). HR reconstruction (15%) Phase
SLIDE 28
28
Original LR ToF measurement Ground truth HR reconstruction (15%)
SLIDE 29
29
Original LR ToF measurement Ground truth HR reconstruction (15%)
SLIDE 30
30
Prototype
q Texas Instrument DLP 4500: 1140×912 pixels q ToF camera: Texas Instrument OPT 8241 320×240 (186×200 pixels)
SLIDE 31
31
Original LR ToF Measurement Pixel scanning
SLIDE 32
32
3D scene
SLIDE 33
33
Original LR ToF measurement
HR Reconstruction (no compression)
HR Reconstruction (0.6) HR Reconstruction (0.25)
Depth in m Depth in m Depth in m Depth in m
SLIDE 34
34
Native No compression CS: 0.6 CS: 0.25
SLIDE 35
35
Original LR ToF measurement HR reconstruction (no compression) HR reconstruction (60%) HR reconstruction (25%)
SLIDE 36
36
Native No compression CS: 0.6 CS: 0.25
SLIDE 37
37
Depth in mm
Ground truth depth Depth values with 25dB SNR
Depth resolution
q No depth resolution improvement q Depth resolution is better than bicubic interpolation
SLIDE 38
38
SLIDE 39
39