Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) University of Toronto Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology Image Sharpness Metric Based on MaxPol Convolution Kernels Mahdi S. Hosseini and Konstantinos N. Plataniotis mahdi.hosseini@mail.utoronto.ca kostas@ece.utoronto.ca Multimedia Laboratory The Edward S. Rogers Dept. of Electrical and Computer Engineering University of Toronto, Ontario, Canada 2018 IEEE International Conference on Image Processing (ICIP) Paper#2842, Session: MQ.L3: Visual Quality Assessment I Monday, 17:40-18:00, October 8, 2018, Athens, Greece Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 1 / 18
Objective and Contribution Main objective: Propose a computational model to Human Visual System (HVS) response to assess natural image blur 1 Synthesize visual sensitivity response by a convolutional filter 2 Use HVS convolution filter to perceive image blur features 3 Implement algorithmic workflow to quantize image blur Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 2 / 18
Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) University of Toronto Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology Outline Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 3 / 18
Frequency Response of Natural Images • Natural images follow a decay response ∝ 1 /ω γ • ω is spatial frequency, γ > 1 is energy tuning factor • Amplitude response of high-frequency is lower than low-frequency Natural Image Frequency Spectrum Amplitude Spectrum Radial Freq. Binning 3 Amplitude Spectrum 2.5 2 1.5 1 0.5 0 0 1 2 3 Frequency ( ) | ˆ | ˆ I 2D ( ω x , ω y ) | I 2D ( ω r ) | I 2D ( x, y ) Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 4 / 18
Visual Sensitivity in Human Visual System (HVS) • HVS analyzes visual inputs in frequency domain • Energy of all amplitude frequencies are perceives equally in HVS • HVS introduces a sensitivity response to compensate the energy-loss of high frequency information • Neurones in visual cortex automatically tune the frequency amplitudes to balance out the falloff of high-frequency range 1 Natural Image Perception in Human Vision System (HVS) 1 [Field-OSA1987], [FieldBrady-Elsevier1995], [FieldBrady-Elsevier1997] Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 5 / 18
Modelling HVS as a Linear Operator • Visual sensitivity response boosts high frequencies to balance out wide spectrum of input visuals • Model HVS as a linear convolution process ¯ I ≈ I Input ∗ h HVS ¯ I - Output image signal perceived by human visual cortex 1 2 I Input - Input image signal 3 h HVS - Convolution filter emulating visual sensitivity response • Goal: synthesize a convolution filter h HVS ( x ) to boost high-frequency amplitudes such that h falloff ( x ) ∗ h HVS ( x ) = δ ( x ) • h falloff ( x ) simulates falloff frequency of input image | ˆ I 2D ( ω r ) | • What is the main merit? If all frequencies are balanced, the features corresponding to different edge types can be visually compared in a meaningful way Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 6 / 18
Design of HVS Convolution Filter − 1 • HVS filter response should satisfy ˆ h HVS ( ω ) = ˆ h falloff ( ω ) • Define HVS as a linear combination of even-derivative operators h HVS ( x ) ≡ c 1 d 2 ( x ) + c 2 d 4 ( x ) + . . . + c N d 2 N ( x ) where d 2 n ( x ) = d 2 n /dx 2 n • Fourier transform of even derivatives is F{ d 2 n ( x ) } = ( jω ) 2 n • So, Fourier transform of HVS filter gives N N ˆ c n ˆ � � ( − 1) n c n ω 2 n h HVS ( ω ) ≡ d 2 n ( ω ) = n =1 n =1 • Unknown coefficients c n are inferred by fitting the model into the inverse falloff response N − 1 ( − 1) n c n ω 2 n ≡ ˆ � h falloff ( ω ) n =1 Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 7 / 18
Numerical Approximation via MaxPol Convolution Kernels • HVS attenuates frequencies close to Nyquist band • Once coefficients c n are obtained, we design lowpass filter N ( − 1) n c n ω 2 n , � 0 ≤ ω ≤ ω c ˆ h HVS ( ω ) = n =1 0 , ω ≥ ω c • ω c is cutoff frequency and is tuned for optimum performance • MaxPol 2 library is used for numerical implementation of lowpass derivative filters ω 2 n ˆ ˆ h falloff ( x ) h HV S ( x ) h falloff ( ω ) h HVS ( ω ) 1 1.5 6 Amplitude response 0.25 Amplitude spectrum 0.8 1 Amplitude 0.2 4 0.5 0.6 0.15 0 0.1 0.4 2 -0.5 0.05 0.2 -1 0 -30 -20 -10 0 10 20 30 -32-28-24-20-16-12 -8 -4 0 4 8 12 16 20 24 28 32 -3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3 Discrete node (x) Discrete node (x) 2 [MaxPol Package] [HosseiniPlataniotis-IEEE2017] [HosseiniPlataniotis-SIAM2017] Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 8 / 18
Natural Image Frequency Falloff Modeling The falloff frequency ˆ h falloff ( ω ) is related to imaging application Synthetic Imaging Blur [HosseiniPlataniotis-ICIP2018] • h falloff ( x ) = 1 /ω p , blur is dominant in p ∈ { 1 , 3 } Natural Imaging Blur [HosseiniPlataniotis-arXive2018] • Using generalized Gaussian (GG) as a frequency falloff distribution A ( β,α ) | β , Scale α = 1 . 7 , Shape β = 1 . 4 x • h falloff ( x ) = c exp −| Microscopic Out-of-Focus Blur [HosseiniPlataniotis-2018] • Encode out-of-focus blur in digital microscopy � 1 2 � � n xρ ) e − 1 2 ikρ 2 z ( NA n ) 2 ρdρ 0 J 0 ( k NA • h falloff ( x ) = � C � � � Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 9 / 18
No-Reference Sharpness Metric Development Images can now be convolved with HVS filter to identify balanced features for NR-FQA metric development I F x F y Algorithm for Sharpness Scoring 1 Exclude background pixels 2 Decompose image using HVS filter F y = I ∗ h HVS T F x = I ∗ h HVS , 3 Activate features by ReLu R ( x ) = max( x, 0) 4 Construct sparse feature map in ℓ 1 / 2 -norm | R ( F x ) | 1 / 2 + | R ( F y ) | 1 / 2 � 2 M HVS R ( F x ) R ( F y ) � M HVS = . 5 Keep a subset Ω of feature pixels M HVS = sort d ( M HVS ) k , k ∈ Ω , 6 Measure the m th central moment ( M HVS − µ 0 ) m � � µ m = E 7 Record the final score Sharpness Score = − log µ m Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 10 / 18
Experiment-I: Synthetic Blur Imaging • Images are synthetically blurred for quality assessment (IQA) • Images are subjectively evaluated for mean opinion score (MOS) • Database examples: LIVE, CSIQ, TID2008, and TID2013 • Terms of evaluation 1 Pearson linear correlation coefficient (PLCC) 2 Spearman rank order correlation (SRCC) Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 11 / 18
Overall Performance • Developed metrics based on MaxPol meet both 1 High correlation accuracy 2 Fast speed calculation CPU time vs image size PLCC vs CPU Time: Synthetic PLCC vs CPU Time: Natural ARISM 10 2 SPARISH RISE MaxPol 0.8 S3 0.96 Computation time (sec) MLV MaxPol HVS MaxPol-2 RISE GPC 0.95 0.7 RISE HVS MaxPol-2 HVS MaxPol-1 GPC S3 HVS MaxPol-1 0.94 HVS MaxPol-1 0.6 HVS MaxPol-2 MaxPol 10 0 SPARISH 0.93 plcc plcc ARISM 0.5 MLV SPARISH 0.92 0.4 MLV 0.91 S3 ARISM 0.3 0.9 GPC 10 -2 0.89 0.2 10 -8 10 -7 10 -6 10 -5 10 -4 10 -8 10 -7 10 -6 10 -5 10 -4 CPU Time/Pixel (sec) CPU Time/Pixel (sec) 4 8 6 2 4 8 6 2 5 1 2 4 x 1 2 5 0 0 4 1 2 x x x 6 8 6 2 x x 2 5 1 4 8 1 2 5 2 4 0 0 1 2 Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 12 / 18
Experiment-II: FocusPath Natural Blur Database • Out-of-focus is common problem in whole slide imaging (WSI) • FocusPath 3 is 864 digital pathology image patches from 9 WSIs • FocusPath images are scanned by Huron TissueScope LE1.2 • 16 Z-stack scans collected from each slide to cover all focus levels 3 download from https://sites.google.com/view/focuspathuoft/home Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 13 / 18
Experiment-II: Natural Blur Imaging • Images are natural blurred for quality assessment (IQA) 1 BID (586 images) 2 CID2013 (474 images) 3 FocusPath (864 images) Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 14 / 18
Recommend
More recommend
