Dermatological Imaging ISIC Skin Image Analysis Workshop @ CVPR 2019 - PowerPoint PPT Presentation

A Customized Camera Imaging Pipeline for Dermatological Imaging ISIC Skin Image Analysis Workshop @ CVPR 2019 Hakki Can Karaimer 1 Iman Khodadad 2 Farnoud Kazemzadeh 2 Michael S. Brown 1 1 York University, Toronto 2 Elucid Labs 1

Talk's topic A customize camera for dermatological analysis Prototype Schematic drawing of the device Machine vision LED light ring camera Broadband visible light and selected non-visible spectral bands Device housing 19mm Macro opening lens ISIC Skin Image Analysis Workshop @ CVPR 2019 2

Challenges 1. Machine vision camera vs. consumer camera 2. How to use the visible image with the narrow band spectral image? Output from Consumer camera Enhanced RGB image machine vision camera's API output using a spectral band ISIC Skin Image Analysis Workshop @ CVPR 2019 3

In-camera processing pipeline • There are a number of steps onboard a camera that convert the light falling on the camera’s sensor image (raw image) to the final R,G,B image output • These steps are collectively called the “in -camera image processing pipeline” ISIC Skin Image Analysis Workshop @ CVPR 2019 4

In-camera processing pipeline # # # # # # # # # # # # f f 6- White-balancing & Gamma applied for visualization 2- Black light subtraction, 3- Lens correction 4- Demosaicing color space transform 5- Noise reduction 1- Reading raw Image linearization [2D Array(s)] [Func] to CIE XYZ [MATs] [Func] [Values or 1D LUT] # # # # # # # # # 12- Gamma curve 11- Final color-space 10- Tone curve 9- Color mani- 8- Exposure curve 7- Hue/Sat map application [1D LUT] conversion [Mat] pulation [3D LUT] application [1D LUT] [EV value or 1D LUT] [3D LUT] Intermediate images for each stage Sensor 1 2 3 4 5 6 Output 12 11 9 8 7 10 ISIC Skin Image Analysis Workshop @ CVPR 2019 5

In-camera processing pipeline Color space transform # # # # # # # # # # # # f f 6- White-balancing & Gamma applied for visualization 2- Black light subtraction, 3- Lens correction 4- Demosaicing color space transform 5- Noise reduction 1- Reading raw Image linearization [2D Array(s)] [Func] to CIE XYZ [MATs] [Func] [Values or 1D LUT] # # # # # # # # # 12- Gamma curve 11- Final color-space 10- Tone curve 9- Color mani- 8- Exposure curve 7- Hue/Sat map application [1D LUT] conversion [Mat] pulation [3D LUT] application [1D LUT] [EV value or 1D LUT] [3D LUT] Photofinishing Intermediate images for each stage Sensor 1 2 3 4 5 6 User 12 11 9 8 7 10 ISIC Skin Image Analysis Workshop @ CVPR 2019 6

Machine Vision vs. Consumer Camera Pipelines • Why does a machine vision camera's image appear different from consumer camera? Typical pipeline for machine vision cameras 2 5 1 3 4 6 RAW RAW Pre- Pre- White- White- Gamma Gamma Demosaic Demosaic Output Output Image Image processing processing balance balance Black level offset, ( Optional ) ( Optional ) normalization ISIC Skin Image Analysis Workshop @ CVPR 2019 7

Machine Vision vs. Consumer Camera Pipelines • Why does a machine vision camera's image appear different from consumer camera? Typical pipeline for machine vision cameras 2 5 1 3 4 6 RAW RAW Pre- Pre- White- White- Gamma Gamma Demosaic Demosaic Output Output Image Image processing processing balance balance Black level offset, ( Optional ) ( Optional ) normalization Typical pipeline for consumer cameras 1 2 4 6 7 9 3 5 8 RAW RAW Pre- Pre- Flat-field Flat-field Noise Noise White- White- Color Color Photo Photo JPEG JPEG Demosaic Demosaic image image processing processing Balance Balance transform transform correction correction reduction reduction finishing finishing compression compression Black level offset, Tone-mapping, normalization sRGB gamma, 3D color LUT ISIC Skin Image Analysis Workshop @ CVPR 2019 7

Customized imaging pipeline Visualization/ • Our customized camera live preview Photo-finished Classification Colorimetric sRGB output Pipeline for visible images module linear-sRGB Pipeline for non-visible images 2 16 1.0 1 R G R G R R R R R R G G G G G G B G B G R B G B G # # # B B B B B ⋅⋅⋅ G B G B G R G R G R R G R G R B B B B B B # # # G G R G R G B G B G R B G B G B B B B B G B G B G R G R G R R G R G R # # # 0.0 Black level B B B B B 00 ⋅⋅⋅ G G R G R R G 1 B B B B B 4- Lens 5- Color space 1- Sensor's raw 2- Black light 6- Photo-finishing 3- Demosaicing 7- Tone curve correction transform image subtraction, 3D LUT (3x3, or 3x11) [2D Array] linearization (OPTIONAL) 2 16 1.0 Enhanced non-spectral R R R R R R G R G R visualization G G G G G R B G B G G B G B G B B B B B ⋅⋅⋅ G B G B G R G R G R R G R G R ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ B B B B B G G R G R R B G B G G B G B G Show Real B B B B B G B G B G R G R G R R G R G R 0.0 Black level Image B B B B B G G R G R ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ B B B B B 1- Sensor's raw 2- Black light 3- Selection on 4- Lens image subtraction, Bayer pattern correction Enhanced sRGB image linearization Image fusion ISIC Skin Image Analysis Workshop @ CVPR 2019 8

LED flat-field correction • A flat-field correction for each LED LED 1 LED 2 LED 3 LED 4 LED 5 LED 6 LED 7 LED 8 LED 9 ISIC Skin Image Analysis Workshop @ CVPR 2019 9

In-camera processing pipeline • Sensor needs to be colorimetrically calibrated • Color space transform (CST) to map raw-RGB values to the CIE XYZ color space Classification Colorimetric module XYZ 2 16 1.0 R G R G R R R R R R G G G G G G B G B G R B G B G # # # B B B B B ⋅⋅⋅ G B G B G R G R G R R G R G R B B B B B # # # G G R G R G B G B G R B G B G B B B B B G B G B G # # # R G R G R R G R G R 0.0 Black level B B B B B ⋅⋅⋅ G G R G R B B B B B 4- Lens 5- Color space 1- Sensor's raw 2- Black light 3- Demosaicing correction transform image subtraction, (3x3, or 3x11) [2D Array] linearization ISIC Skin Image Analysis Workshop @ CVPR 2019 10

In-camera processing pipeline • Errors with and without sensor calibration Off-the-shelf Visualization of patches WB CST 3 × 3 CST 3 × 11 machine vision camera CC ° Mean angular error: 22.04 ° Mean angular error: 9.97 ° Mean angular error: 2.75 ° Mean angular error: 2.70 ° SC Mean angular error: 22.23 ° Mean angular error: 7.44 ° Mean angular error: 2.80 ° Mean angular error: 2.66 ° CC: Macbeth color checker chart SC: Skin colors from the Munsell Book of Color ISIC Skin Image Analysis Workshop @ CVPR 2019 11

Photo-finishing • Photo-finishing to make the images look visually- pleasing 2 16 1.0 1 R G R G R R R R R R G G G G G G B G B G R B G B G # # # B B B B B ⋅⋅⋅ G B G B G R G R G R R G R G R B B B B B B # # # G G R G R G B G B G R B G B G B B B B B G B G B G R G R G R R G R G R # # # 0.0 Black level B B B B B 00 ⋅⋅⋅ G G R G R R G 1 B B B B B 4- Lens 5- Color space 1- Sensor's raw 2- Black light 6- Photo-finishing 3- Demosaicing 7- Tone curve correction transform image subtraction, 3D LUT (3x3, or 3x11) [2D Array] linearization (OPTIONAL) ISIC Skin Image Analysis Workshop @ CVPR 2019 12

Photo-finishing • Photo-finishing to make the images look visually- pleasing • We can mimic different consumer cameras A B C D E F Raw-RGB + gamma Linear-sRGB (B) Photo finished w/ (B) Photo finished w/ (B) Photo finished w/ (B) Photo finished w/ Adobe tone-curve Nikon's Vivid mode Canon's Portrait Olympus's Natural mode mode ISIC Skin Image Analysis Workshop @ CVPR 2019 13

Non-visible spectral images Visualization/ • Our customized camera live preview Photo-finished Classification Colorimetric sRGB output Pipeline for visible images module linear-sRGB Pipeline for non-visible images 2 16 1.0 1 R G R G R R R R R R G G G G G G B G B G R B G B G # # # B B B B B ⋅⋅⋅ G B G B G R G R G R R G R G R B B B B B B # # # G G R G R G B G B G R B G B G B B B B B G B G B G R G R G R R G R G R # # # 0.0 Black level B B B B B 00 ⋅⋅⋅ G G R G R R G 1 B B B B B 4- Lens 5- Color space 1- Sensor's raw 2- Black light 6- Photo-finishing 3- Demosaicing 7- Tone curve correction transform image subtraction, 3D LUT (3x3, or 3x11) [2D Array] linearization (OPTIONAL) 2 16 1.0 Enhanced non-spectral R R R R R R G R G R visualization G G G G G R B G B G G B G B G B B B B B ⋅⋅⋅ G B G B G R G R G R R G R G R ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ B B B B B G G R G R R B G B G G B G B G Show Real B B B B B G B G B G R G R G R R G R G R 0.0 Black level Image B B B B B G G R G R ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ ⋅⋅⋅ B B B B B 1- Sensor's raw 2- Black light 3- Selection on 4- Lens image subtraction, Bayer pattern correction Enhanced sRGB image linearization ISIC Skin Image Analysis Workshop @ CVPR 2019 14