Performance Bounds for Computational Imaging Oliver Cossairt Assistant Professor Northwestern University Collaborators: Mohit Gupta 1 , Changyin Zhou 1 , Daniel Miau 1 , Shree Nayar 1 , Kaushik Mitra 2 , Ashok Veeraraghavan 2 1 Columbia University 2 Rice University
Computational Imaging: Increased Functionality • Take multiple pictures and computationally combine Panoramic Stitching Light Field Capture HDR Imaging [Wilburn et al. ’04] Image-Based Lighting Digital Holography Others Multiview Stereo • Depth from Focus/Defocus • Tomography • Structured Light • Deconvolution microscopy • etc. • [Greenbum et al. ’ 12] [Debevec et al. ’00]
Computational Imaging: Increased Performance Coded image capture for increased performance Motion Blur Coded Aperture Defocus Blur [Dowski, Cathey ‘96] [Hausler ’72] [Raskar ’ 06] [Levin ‘08] [Mertz ’ 65] [Gottesman ’ 89] [Nagahara ’08] [Cho ’ 10] [Levin et al. ’ 07] [Zhou, Nayar ’ 08] Reflectance Light Field Capture Multi/Hyper-Spectral [Schechner ‘03] [Lanman ’ 08] [Veeraraghavan ’ 07] [Ratner ‘07] [Sloane ’ 79] [Hanley ’ 99] [Liang ‘08] [Ratner ‘ 08] [Baer ‘99] [Wetzstein et al., ’ 12]
Computational Imaging Performance Camera Camera Exposure Exposure Time Time 50 millisec 50 millisec Vs. Short Exposure Long Exposure
Computational Imaging Performance [Raskar ’ 06] Camera Camera Exposure Exposure Time Time 50 millisec 50 millisec Vs. Short Exposure Coded Exposure Deblurred Image When does computational imaging improve performance?
Measuring Computational Imaging Performance
Image Formation Model Image Computational Coded Scene Camera Image Coded Coding Image Matrix Noise No diffraction • Fully determined • Optical Coding Equation Assumption: A) Linear model of incoherent image formation
Affine Noise Model Noise Variance at k th Pixel: Signal dependent / independent noise • Ignore Dark current, fixed pattern • photon noise read noise aperture, electronics, lighting, ADC’s, pixel size quantization Noise PDF: Photon noise modeled as Gaussian • (ok for more than 10 photons) Photon noise spatially averaged • Assumption: B) Affine noise model (photon noise is Gaussian)
Lighting Conditions Signal-level and photon noise depend on illumination Average Illumination Reflectivity Aperture Exposure Quantum Pixel Signal (e - ) (lux) Time (s) Efficiency Size (m) Ex) q=.5, R = .5, F/8, t = 6ms, p=6um Quarter moon Full moon Twilight Indoor lighting Cloudy day Sunny day Illumination I src (lux) 10 -2 10 10 2 10 3 10 4 1 Signal level 0.8 8.1 81.4 814.3 8143 8×10 -3 J (e - ) [Cossairt et al. TIP ‘12] Assumption: C) Naturally occurring light conditions for photography
Measuring Performance For Gaussian noise, Mean-Squared-Error (MSE) can be computed analytically Ex) Coded Motion Deblurring Long Exposure Coded Exposure Observation: 1) Multiplexing performance depends on coding matrix
Multiplexing vs. Impulse Imaging Impulse imaging (identity sampling) Noise variance Coded imaging (multiplexed sampling) Noise variance Increased throughput Observation: 2) Multiplexing increases signal-dependent noise
Multiplexing vs. Impulse Imaging SNR Gain over impulse imaging: Hadamard Multiplexing: Coding Dependent Noise Dependent [Sloane ’ 79] No SNR gain for large signal Decreases with C Increases with C Coded Aperture Astronomy Increasing scene points Fresnel zone plate [Mertz ’ 65] Decreasing contrast Observation: 3) Performance depends on multiplexing and signal prior
Image Prior Models Assume we have a PDF for images, e.g. Power Spectra Prior Other priors • Total Variation (TV) • Wavelet/sparsity prior • Learned priors (K-SVD) Compute the Maximum A Posteriori (MAP) estimate Data term Prior term MSE difficult to express analytically when Assumption: D) Signal prior models naturally occurring images
Image Priors and Noise Denoise Twilight (10 lux) PSNR = 5.5 dB PSNR = 16.4 dB Denoise Daylight 5 (10 lux) PSNR = 35 dB PSNR = 35.9 dB Observation: 4) Signal priors help more at low light levels
Observations: 1) Multiplexing performance depends on coding matrix 2) Multiplexing helps most in low light 3) Performance depends on both multiplexing and signal prior 4) Signal priors help most in low light Assumptions A) Incoherent imaging B) Affine noise model C) Natural lighting conditions D) Natural image prior When does computational imaging improve performance?
Example: Motion Deblurring
Motion Deblurring vs. Impulse Imaging Optical efficiency (C) = total ‘on’ time Camera Exposure Time Time 50 millisec 50 millisec Vs. Impulse Imaging (Short Exposure) Computational Imaging (Coded Exposure) What is the best possible coding performance we can get? [Ratner ‘07]
When Does Motion Deblurring Improve Performance? Upper Bound on SNR Gain: Read Noise Average signal level Performance depends only on lighting conditions! q=.5, R = .5, F/2.1, p = 1um, Motion Invariant Levin et al. Flutter Shutter Raskar et al. Maximum object speed (pixels/sec) [Cossairt et al. TIP ‘12]
Flutter Shutter Simulation q=.5, R = .5, F/2.1, pixel size = 1um, read noise Impulse (4ms) Flutter Shutter (180ms) Deblurred Twilight (10 lux) PSNR = -7.2 dB PSNR = -3.0 dB Cloudy Day 3 (10 lux) PSNR = 12.4 dB PSNR = 10.1 dB
Example: Extended DOF Imaging
Depth of Field and Noise Image Lens F 1.4 F 2.8 F 8.0 F 5.6 Small apertures have large depth of field and low SNR
Focal Sweep Sensor Lens 1200 400 600 900 1500 1700 2000 (depth) Point Spread Function (PSF) [Hausler ‘72 , Nagahara et al. ‘08]
Focal Sweep Sensor Lens 1200 400 600 900 1500 1700 2000 (depth) Integrated PSF + + + + + + = t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7 (1200) (1700) (400) (600) (900) (1500) (2000) [Hausler ‘72 , Nagahara et al. ‘08]
Quasi Depth Invariant PSF Focal Sweep PSF Traditional Camera PSF 0.016 0.016 1200mm 1200mm 1 × 60 0.012 0.012 750mm 0.008 0.008 400mm 750mm 2000mm 0.004 0.004 400mm 2000mm 0 0 -25 0 25 -25 0 25 Extended depth of field with a single deconvolution
Extended Depth of Field Telescope 50 m 75 m Traditional Image 50 m 50 m 75 m 75 m Meade LX200 8’’ Telescope 2000mm FL Focal Sweep: Captured Focal Sweep: Processed
Focal Sweep Simulation Traditional Traditional Focal Sweep Pixel size = 5um Read noise (F/2.0) (F/20.0) (F/2.0) Twilight (10 lux) PSNR = 5.5 db PSNR = 18.5 db Daylight 5 (10 lux) PSNR = 35 db PSNR = 38.5 db
Focal Sweep Simulation (with Prior) Traditional Traditional Focal Sweep Pixel size = 5um Read noise (F/2.0) (F/20.0) (F/2.0) Twilight (10 lux) PSNR = 16.4 dB / 5.5 dB PSNR = 22.8 dB / 18.5 dB Daylight 5 (10 lux) PSNR = 35.9 dB / 35 dB PSNR = 39.6 dB / 38.5 dB BM3D Algorithm: [Dabov et al. ‘06]
A Universal Image Prior State-of-the-art priors are hard to analyze! Gaussian Mixture Model (GMM) prior Number of clusters Cluster Cluster Cluster weight mean covariance GMM parameters learned from database of 30 Million image patches • Analytic expression for MSE depending only on • Coding Noise GMM [Mitra et al., submitted to PAMI ‘13] matrix variance parameters
Focal Sweep Performance with GMM Prior Impulse camera: F/11, Focal Sweep: F/1 q=.5, R = .5, t=6ms, p = 1um, Multiplexing gain with prior SNR gain (in dB) Gain due to prior alone Multiplexing gain without prior Full moon Twilight Indoor lighting Cloudy day Sunny day Illuminance (in lux) [Mitra et al., submitted to PAMI ‘13]
Defocus Deblurring Performance with GMM Prior SNR gain (in dB) Cubic Phase Defocus Deblurring gain as high as 8dB for Cubic Phase Plate [Mitra et al., submitted to TIP ‘13]
Conclusions Results for Motion Deblurring, EDOF also applicable to many • other computational cameras Computational imaging performance should always be • measured relative to impulse imaging Computational imaging performance depends jointly on • multiplexing, noise, and signal priors Important question: “How much performance improvement • from multiplexing above and beyond use of signal priors?”
Depth of Field Small DOF Microscope Tachinid Fly http://en.wikipedia.org/wiki/Focus_stacking
Depth of Field Large DOF Microscope Tachinid Fly http://en.wikipedia.org/wiki/Focus_stacking
Telephoto Focal Sweep with Deformable Optics Canon 800mm EFL Lens Sensor Deformable Lens [Miau et al. ICCP ‘13]
Video Quality Comparison Conventional EDOF (Deformable Lens)
Focal Sweep Performance Mean-Squared Error: Noise Variance: Impulse Camera sensor lens A Focal Sweep Mean-Squared Error: Noise Variance: sensor lens C*A light increase [Cossairt et al. TIP ‘12] diffuser
When Does Defocus Deblurring Improve Performance? Focal sweep multiplexing gain can be expressed analytically Read Noise Average signal level Performance depends only on lighting conditions! q=.5, R = .5, t = 20ms, p = 5um, [Cossairt et al. TIP ‘12]
Recommend
More recommend
