Data-driven Optimal Camera Response Selection and Design for - PowerPoint PPT Presentation

VALSE Data-driven Filter Response Selection and Design Data-driven Optimal Camera Response Selection and Design for Spectral Reconstruction 数据驱动、面向光谱重建的最优相机响应曲线 的自动选择和设计 Yinqiang Zheng ( 郑银强） 1. National Institute of Informatics 2. SOKENDAI （日本国立信息学研究所） （综合研究大学院大学） 2018.07.11

1 Main Contributors Beijing Institute of Technology Prof. Hua Huang Prof. Ying Fu Tao Zhang National Institute of Informatics/Saitama University Prof. Imari Sato Shijie Nie Lin Gu Antony Lam 2018.07.11

2 Color: RGB vs. Spectrum N Channels (N>>3) 3 Channels 2018.07.11

3 Electromagnetic Wave (Light) https://study.com/academy/lesson/electromagnetic-waves-definition-sources-properties-regions.html 2018.07.11

4 Spectra of Ordinary Light Sources HID Lamp ( 高辉度放电灯 ) Xenon Lamp （氙气灯） 2018.07.11

5 Applications of Hyperspectral Imaging Remote Sensing Agriculture Medical Diagnostics Bioscience 2018.07.11

6 Hyperspectral Imaging Devices Spectrometer-Single Point Line-Scan Spectral Camera Filter-Scan Spectral Camera Pixelteq LQT Quantum Hamamatsu Mechanical Electronic 2018.07.11

7 Single-Point Devices （单点分光仪） Spectrometer-Single Point Hamamatsu C10082MD Diffraction Prism/Grating Hamamatsu S11510 ( 分光棱镜、光栅） (2048 x 64 pixels ） Photomultiplier Tube ( 光电倍增管） Hitachi Fluorescence Single-Pixel Camera Spectrometer F-7100 2018.07.11

8 Line Imaging Devices （线分光相机） Line-Scan Spectral Camera LQT Quantum CCD Array DJI Phantom Static Object 2018.07.11

9 Major Drawbacks of Scan-based Spectral Cameras Moving Object ： Noisy Images Rolling Shutter 畸变（果冻效应） High Cost 2018.07.11

10 Computational Reconstruction from Compressed Sensing Measurement Random Sampling in the Spatial Domain Sparse Coding Shallow Network Deep Network Representative Work: 1. Single disperser design for coded aperture snapshot spectral imaging (CASSI), Applied Optics, 2008. 2. Single-shot compressive spectral imaging with a dual- disperser architecture (DD-CASSI), Optical Express, 2007. Coded Aperture 3. Spatial-spectral encoded compressive hyperspectral imaging, TOG, 2014. 2018.07.11

11 Uniformly Downsampled Variants Hybrid-resolution spectral video system using A Prism-Mask System for Multispectral Video low-resolution spectral sensor ， Optical Acquisition ， TPAMI, 2011. Express, 2014. 2018.07.11

12 Low-Resolution Spectral Image and RGB/Gray Fusion 1. High-resolution Hyperspectral Imaging via Matrix Factorization ， CVPR, 2011. 2. Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System ， IJCV, 2013. 3. High speed hyperspectral video with a dual-camera 4. Optics and methods for hybrid resolution architecture, CVPR, 2015. spectral imaging, Applied Optics, 2015. 2018.07.11

13 Challenges in Hybrid Fusion – Image Alignment Algorithmic Alignment – Good if you Physical Alignment – Hard, but know the transformation and the resolution can be done with efforts. is high. Extremely Low Resolution 2018.07.11

14 RGB Camera Color Imaging Mechanism 3 Filters+3 Sensors 1 Filter Arry+1 Sensor 2018.07.11

15 Computational Reconstruction from RGB/Multispectral Images RGB Image Sparse Coding Shallow Network Deep Network Major Advantage: 1. Variation in spectral domain is much less than in space domain . 2. To capture multispectral images is fast. Material Reflectance Spectra Ordinary Illumination Spectra 2018.07.11

16 RGB-to-Spectrum via Manifold based Mapping Jia et al., From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping, ICCV, 2017. 2018.07.11

17 RGB-to-Spectrum via CNN Ying Fu, Tao Zhang, Yinqiang Zheng, Hua Huang, Joint Camera Spectral Sensitivity Selection and Hyperspectral Image Recovery, ECCV, 2018, accepted. 2018.07.11

18 Comparison with State-of-the-Art RBF: Training based spectral reconstruction form a single RGB image, ECCV, 2014. SR: Sparse recovery of hyperspectral images from natural RGB images, ECCV, 2016. MM: From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping, ICCV, 2017. 2018.07.11

19 RGB Spectral Response Database Jiang et al., What is the Space of Spectral Sensitivity Functions for Digital Color Cameras? WACV, 2013 2018.07.11

20 Sensitivity to RGB Response • Using our CNN method for reconstruction • Conducting experiment using all camera responses one by one with the same setting Ref: Filter selection for hyper-spectral estimation, ICCV, 2017. Ref: Sparse recovery of hyperspectral images from natural RGB images, ECCV, 2016. 2018.07.11

21 Camera is Doing Convolution Convolution Kernel Response Function Filter+CCD/CMOS Sensor is actually a convolution operator along the spectral axis, and the stride is 1 (sum of dot product). The convolutional kernel is the filter response function. 2018.07.11

22 Convolutional Neural Network Here is an example of 2D convolutional kernel in the spatial domain. Note that, convolution has been implemented in all existing deep learning toolboxes. So, we can reuse them for our filter designing purpose. 2018.07.11

23 To Select the Best Response via CNN Selection CNN Spectral Reconstruction CNN Response Database        r r r r c c c ... c 1 1 2 2 28 28             g g g g ， c c c ... c ,..., 0,[ ,..., ]sparse. 1 1 2 2 28 28 1 28 1 28        b b b b c c c ... c 1 1 2 2 28 28 2018.07.11

24 Selection Results Sparsity Constraint + Nonnegative Constraint Huge Acceleration: 28 -> 1 2018.07.11

25 Go beyond Existing RGB Responses Channel #1 Channel #2 Channel #3 Smooth Shape Nonnegative Spiky Shape Negative 2018.07.11

26 To Design Optimal Response Shijie Nie, Ling Gu, Yinqiang Zheng, Antony Lam, Nobutaka Ono and Imari Sato, Deeply Learned Filter Response Functions for Hyperspectral Reconstruction, CVPR ， 2018. 2018.07.11

27 Synthetic Experiment Results Training Loss on CAVE Designed Response Curves on CAVE Comparison of Recovered Spectra from Our Method and Existing Ones 2018.07.11

28 Synthetic Experiment Results 2018.07.11

29 Realized Filters Blue: Filter 1 Dotted: Designed/Solid: Measured. Red: Filter 2 Dotted: Designed/Solid: Measured. 2018.07.11

30 Prototype Two-Band Camera Grayscale Image 1 Grayscale Image 2 2018.07.11

31 Remaining Challenges 1: Limited Datasets • Common Datasets: 1. ICVL, 201 images, Natural Illumination 2. NUS, 66 images, Mixed Illuminations 3. Harvard, 50 images, Outdoor 4. CAVE, 32 images, D65 (but normalized) • The Metamerism Issue: 2018.07.11

32 Remaining Challenges 2: Are Hyper- Spectral Images Really Necessary? RGB Multi-Spectral Hyper-Spectral Object Classification GT RGB (97.07%) Recon.(98.36%) HIS (99.12%) 2018.07.11

33 Remaining Challenges 3: From Spectral Reconstruction to High-Level Tasks RGB Image Spectral Sensing: Low-Level Vision Task Classification Face Detection Detection/Recognition/Classification: High-Level Vision Task 2018.07.11

34 Open Question: Is Human Eye Perception Optimal? vs. For spectral reconstruction, and on our limited datasets, the answer is negative. 2018.07.11

35 Thank you very much! 2018.07.11