xun cao
play

Xun Cao ( ) School of Electronic Science & Engineering Nanjing - PowerPoint PPT Presentation

Mar 23, 2023 •357 likes •1.02k views

High Resolution Spectral Video Capture & Computational Photography Xun Cao ( ) School of Electronic Science & Engineering Nanjing University caoxun@nju.edu.cn Dec 30th, 2015 Computational Photography Computational Photography

  1. High Resolution Spectral Video Capture & Computational Photography Xun Cao ( 曹汛 ) School of Electronic Science & Engineering Nanjing University caoxun@nju.edu.cn Dec 30th, 2015

  2. Computational Photography Computational Photography Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. (Wikipedia) Computational photography is an emerging new field created by the convergence of computer graphics, computer vision and photography. Its role is to overcome the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world. (CMU Course Introduction) 2

  3. CP for various imaging dimensions Spectral （ Color ） Depth & View （ 3D ） Spatial Hyperspectral Light Field Gigapixel Temporal Multiview Multispectral Stereo RGB Ps.Fs UHD 3840*2160 120Hz Gray 2D 60Hz scale HD 1920*1080 30Hz Dynamic 10Hz SD<720p Range 24 bit 10/12 bit 8 bit

  4. Computational Imaging Technology & Engineering The lab focuses on 3 kinds of computational cameras • Spectral Camera – High Resolution Spectral Video Camera: PMIS • Multiview Stereo Camera Array – High Accuracy 3D Reconstruction • Super-Resolution Camera – Nano-Scale Pixel Camera – Gigapixel on Single Chip

  5. Grayscale Imaging light source K I  ( )       K R ( ) ( ) S d S  ( )  R  ( ) scene sensor 

  6. Color Imaging light source （ ） R G B , , I  ( )       R R ( ) S ( ) d R      G R ( ) S ( ) d G      B R ( ) S ( ) d B S  ( ) R   ( ) scene sensor 

  7. Spectral Imaging light source R  ( ) I  ( )   ？ ？ R  ( ) scene Imaging system 

  8. Related Work (1)  Filtered Camera based Spectrum Imaging [Kidono07] [Gat00][Yamaguchi06][Schechner02 ]… Filter wheel Color Filter Array Programmable Filter Filter Scanning Key idea: Trade time for spectrum Shortcomings:  Incapable of capturing dynamic scenes  Low spectrum resolution Prof. S.Nayar Spatially varying filter Columbia PAMI ’02

  9. Related Work (2)  Coded Aperture Snapshot Spectral Imager ( CASSI ) Key Idea: Coded Aperture [Brady’06] [Willett’07] [Gehm’07] [Wagadarikar’08] 2D Imaging + reconstruction Spectrum resolution: 6 nm Spatial resolution: 256 x 248 Limited spatial resolution Limited accuracy Time-consuming reconstructing (20min / frame) Prof. D.Brady (Duke Univ.) CASSI: Applied Optics SPIE , JOSA’06 -09

  10. Related Work (3)  Computed Tomography Imaging Spectrometer [Descour95] [Descour01] [Vandervlugt07 ] [Hagen08]… Key Ideas: CT Projections + Reconstruction Shortcomings: Low Resolution Difficult to Calibrate Computed Tomographic Prof. E. Dereniak CTIS High Computational Cost Arizona Imaging Spectrometer different linear “projections” of the spectral data cube E. Dereniak Applied Optics SPIE, JOSA’95 -08 [JOSA’08 ]

  11. Our Spectral Video Camera - PMIS 2008~2010: Prism-Mask Imaging Spectrometer (PMIS 1 ) – Directly capture multispectral video – High spectra-resolution – Low cost – Easy setup and calibration 2011~2014: Hybrid-Camera PMIS 2 – Both high spectral and spatial resolution – Real-time hyperspectral video capture 2014~now Scene-Adaptive PMIS 3 – Space-time coded modulation – Spectral video capture with improved accuracy and efficiency

  12. A glance at PMIS 1 Pointgrey grayscale camera 2248x2048 @15fps mask capturing system

  13. System Principle Grayscale Camera Occlusion Mask Prism GIF source: Wiki Camera System Re-generated RGB Video

  14. A Typical Camera lens sensor array camera

  15. Camera & Prism Spectra Overlap! lens sensor array prism camera

  16. Camera & Mask lens sensor array mask camera

  17. Camera & Mask & Prism Spectra Overlap! lens sensor array mask camera

  18. Spectral Resolution W S ( ) a  R  spec CCD cell size grayscale camera f        e  W S ( ) s mask prism aperture image plane     sin ( ) sin ( ( ))             n  W S ( ) f (tan( ( ) a ) tan( ( ) a ))      e s sin ( ( )) sin ( ( ))

  19. Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane           W S ( ) f (tan( ( ) a ) tan( ( ) a )) e s

  20. Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane   W S ( )          W S ( ) f (tan( ( ) a ) tan( ( ) a )) R  spec e s

  21. Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane   W S ( )          W S ( ) f (tan( ( ) a ) tan( ( ) a )) R  spec e s

  22. Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane   W S ( )          W S ( ) f (tan( ( ) a ) tan( ( ) a )) R  spec e s

  23. Spatial Resolution • Small Mask Hole Distance Spectra Overlap! mask prism aperture image plane

  24. Spatial Resolution • Large Mask Hole Distance Unused Pixels mask prism aperture image plane

  25. Spatial Resolution In practice, we can use a uniform mask  Design Mask Hole Distance d  Perfect D Alignment mask prism aperture image plane           D d (tan( ( )) tan( ( ))) e s

  26. Device Calibration

  27. Calibration Overview Mapping Position to Wavelength Spectrum Calibration Geometry Calibration Geometry Distortion caused by the prism Radiance (Smile Distortion) Non-constant Calibration CCDSensitivity

  28. Spectrum Calibration Spectrum Calibration Geometry Calibration Radiance Calibration

  29. Spectrum Calibration captured spectra Ground truth fluorescent spectra

  30. Spectrum Calibration target spectra Warp Ground truth fluorescent spectra

  31. Spectrum Calibration • Mapping Function : Wavelength <-> Position a  f x  aperture mask prism image plane    sin      Non linear , but   x ( ) f tan a arcsin( n sin( arcsin( )))    n smooth curve ! 

  32. Geometry Calibration Spectrum Calibration Geometry Calibration Radiance Calibration

  33. Geometry Calibration Predefined mask pattern captured image geometry calibrated image

  34. Radiance Calibration Spectrum captured radiance genuine radiance Calibration Geometry Calibration Radiance Calibration

  35. Radiance Calibration captured radiance genuine radiance   assuming c ( ), ( ) l locally constant   z ( )     b I z ( ) g ( c ( ) ( ) l d )  z ( ) a sensitivity intensity  1 ( ( )) g I z   l ( ) light input     wavelength c ( )( ) b a

  36. Application 1: Human Skin Detection • The ‘W’ pattern in human skin reflectance • [Angelopoulou01]

  37. Application 1: Human Skin Detection

  38. Application 2: Material Discrimination RGB Image IR Image Our measurement The differences in IR

  39. PMIS 1 Conclusions • Compared to Traditional Spectrometers • Passive Multispectral Video Capture • High spectral resolution • Tradeoff spectral and spatial resolution • Easy setup and calibration • Applications • Skin Detection • Material Recognition • Illumination Identification PMIS 1 ： A Prism-Mask System for Multispectral Video Acquisition, IEEE Intl’ Conf. Computer Vision (ICCV), 2009 , Oral IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 2011 High Resolution Multispectral Image Capture ， US Patent.20140085502

  40. PMIS 1 : limitations • Light throughput is limited by – occlusion mask – relatively small aperture • Can NOT achieve both high spatial and spectral resolution – Limited CCD resolution – Spatial resolution (1000 pixels)

  41. Low-Spatial High-Spectral Gray Camera Scene or Object Resolution Video Occlusion Mask Prism High-Spatial RGB Camera Low-Spectral PMIS 2 : Hybrid Camera System Resolution Video

  42. PMIS 2 : System Pipeline Propagation

  43. PMIS 2 : System Implementation

  44. Propagation Algorithm Gray Camera RGB Camera High Spatial Low Spatial Resolution Resolution RGB Video Multispectral Video

  45. Propagation Algorithm   RGB xy c G ( d ) G ( d )  ms     k k k k k r s  ms ij RGB xy G ( d ) G ( d )    c R G B , ,  k k k r s   e.g. for red channel R / R , k ij k

  46. Propagation Algorithm Ground Truth Data Evaluation (11 datasets, .aix, .mat) - Spectral Database, University of Joensuu Color Group

  47. Propagation Algorithm Temporal Enhancement Error 7.8% 4.3%

  48. PMIS 2 : Results & Applications

Recommend

More recommend