The EXQUIRES (EXtensible QUantitative Image RESampling) Test Suite: Impact of the Downsampler, Difference Metric, Test Image, Resampling Ratio and Colour Space on Upsampler Rank Adam Turcotte Department of Mathematics and Computer Science Laurentian University Sudbury, October 12, 2012 EXQUIRES (EXtensible QUantitative Image RESampling) 1
Image Resampling • Image resampling is performed when: • zooming in on a digital photograph on a tablet or computer screen • watching a classic sitcom in HD (high definition) • registering (aligning) two medical images taken at different times with slightly different positioning and state of the patient in order to compare them • using the software zoom in a digital camera • warping a portrait for artistic purposes • assembling multiple satellite views of the Earth or the Sun • producing image thumbnails • making an image fit a browser window • processing one’s desktop so that it is shown within a computer monitor taking into account the type and configuration of the screen’s physical pixels • etc. EXQUIRES (EXtensible QUantitative Image RESampling) 2
Overview • What is Image Resampling? • Combining known pixel values in order to obtain: • pixel values at locations where none are provided (between the original pixel locations, as in image enlargement) • a pixel value suitable for different viewing conditions (as in image reduction) EXQUIRES (EXtensible QUantitative Image RESampling) 3
Overview • Geometric Conventions: • align (image) corners [ImageMagick] • align (corner pixel) centres [old versions of VIPS] • linear algebra (define a zero-coordinate and stretch) • align top-left corner pixel centre [current VIPS] • others... EXQUIRES (EXtensible QUantitative Image RESampling) 4
Overview • Image Quality: • Subjective (Perceptual) VS Objective (Physical) EXQUIRES (EXtensible QUantitative Image RESampling) 5
Overview • Re-Enlargement (Reconstruction) Test: • An objective test to evaluate image upsampling methods EXQUIRES (EXtensible QUantitative Image RESampling) 6
Re-Enlargement Tests • We need small versions of the test image so we can upsample them • In theory, take a step back and take the picture from further away • In practice: • changes in lighting, conditions, focus, alignment, etc. EXQUIRES (EXtensible QUantitative Image RESampling) 7
Re-Enlargement Tests • Box filtering: • Reasonable approximation of "stepping back" • Introduces artifacts detectable by professional photographers • The specific artifacts identify the image as being box filtered • Lesson: use several different downsamplers EXQUIRES (EXtensible QUantitative Image RESampling) 8
STEP 1: Downsample the original image Original Reduced EXQUIRES (EXtensible QUantitative Image RESampling) 9
STEP 2: Upsample the reduced image Original Re-Enlarged Reduced EXQUIRES (EXtensible QUantitative Image RESampling) 10
STEP 3: Compare the re-enlargement and original Original Re-Enlarged Difference Reduced EXQUIRES (EXtensible QUantitative Image RESampling) 11
Image Comparison Metrics • Standard physical (pixel-by-pixel) comparison: N � • ℓ 1 ( x , y ) = AAE ( x , y ) = 1 | x i − y i | N i = 1 � N � � � � 1 ( x i − y i ) 2 • ℓ 2 ( x , y ) = RMSE ( x , y ) = N i = 1 • PSNR = 20 log 10 ( MAX RMSE ) • Non-standard physical (pixel-by-pixel) comparison: � N � � � ( x i − y i ) 4 � 1 • ℓ 4 ( x , y ) = 4 N i = 1 • ℓ ∞ ( x , y ) = MAE ( x , y ) = max 1 ≤ i ≤ N | x i − y i | EXQUIRES (EXtensible QUantitative Image RESampling) 12
Image Comparison Metrics • Perceptual: ( 2 µ x µ y + C 1 )( 2 σ xy + C 2 ) • SSIM ( x , y ) = ( µ 2 x + µ 2 y + C 1 )( σ 2 x + σ 2 y + C 2 ) • µ x and µ y are the sample means • σ x and σ y are the standard deviations • σ xy is the correlation coefficient between images x and y • Typically: • µ x = Gaussian _ blur ( x ) • µ y = Gaussian _ blur ( y ) • σ 2 x = Gaussian _ blur ( x 2 ) − µ 2 x • σ 2 y = Gaussian _ blur ( y 2 ) − µ 2 y • σ xy = Gaussian _ blur ( xy ) − µ x µ y EXQUIRES (EXtensible QUantitative Image RESampling) 13
The Issue of Colour Spaces • sRGB is a perceptual colour space • Approximates the frequency response of the HVS • RMSE is a physical measurement • So why use straight sRGB values? • They aren’t in linear light... • Solution: convert to a linear colour space EXQUIRES (EXtensible QUantitative Image RESampling) 14
Should we always use linear light? • Charles Poynton (chose size of HDTVs) • Eric Brasseur’s Dalai Lama example (gamma correction): Original sRGB Linear EXQUIRES (EXtensible QUantitative Image RESampling) 15
Implemented Image Comparison Metrics • Pixel-by-pixel comparison in different colour spaces: • sRGB (perceptual; approximates frequency response of HVS) • XYZ (physical; linear) EXQUIRES (EXtensible QUantitative Image RESampling) 16
Implemented Image Comparison Metrics • Delta-E CMC(1:1) • Defined by the Colour Measurement Committee of the Society of Dyers and Colourists • Color tolerancing system • Based on the CIE LCH (lightness, chroma, hue) colour space EXQUIRES (EXtensible QUantitative Image RESampling) 17
Implemented Image Comparison Metrics • Mean SSIM (MSSIM) • Refactored computation requires 20% fewer FLOPs • Removed a Gaussian blur • MATLAB version handles images 4x larger than the official release • Additional metric (called "blur") • SSIM-inspired pixel-by-pixel comparison (grayscale) • Apply Gaussian blur to both images • Compute ℓ 1 , ℓ 2 , ℓ 4 , and ℓ ∞ EXQUIRES (EXtensible QUantitative Image RESampling) 18
Aggregation • Aggregate in a way that is consistent with the metric • Average of ℓ 1 errors • Square root of sum of squares of ℓ 2 errors • Fourth root of sum of quads of ℓ 4 errors • Maximum of ℓ ∞ errors • Average of MSSIM values EXQUIRES (EXtensible QUantitative Image RESampling) 19
Rank Correlation • Standard Competition Ranking (1, 2, 2, 4) • Fractional Ranks (1, 2.5, 2.5, 4): • Ties are resolved by assigning to all tied upsamplers the average rank they would obtain if the ties were resolved arbitrarily without affecting the ranks of the other upsamplers. • Spearman’s Rank Correlation: • Measures how uncorrelated rankings are • Not a very accurate measure of how strongly correlated rankings are • It shows, just not quite as well as one would want • With lots of data, probably not so different from Kendall • Kendall’s Rank Correlation: • Better choice for measuring rank correlation strength • More complex and difficult to implement EXQUIRES (EXtensible QUantitative Image RESampling) 20
EXQUIRES Overview • 5 main programs • exquires-new • exquires-run • exquires-update • exquires-report • exquires-correlate EXQUIRES (EXtensible QUantitative Image RESampling) 21
EXQUIRES Overview • 2 support programs • exquires-compare • exquires-aggregate EXQUIRES (EXtensible QUantitative Image RESampling) 22
EXQUIRES Overview • Sphinx documentation • Can compile into many formats (HTML, PDF, L A TEX, etc...) • HTML version is online at http://exquires.ca EXQUIRES (EXtensible QUantitative Image RESampling) 23
Natural Images • We used natural images: • different cameras • different photographers • different demosaicing • different subject matter • different conversion from RAW • different denoising EXQUIRES (EXtensible QUantitative Image RESampling) 24
Test Images • Images are high quality: • Taken with reasonably recent high quality DSLRs • 840x840 colour images • TIFF file format • 16 bits per sample (48 bits per pixel) • sRGB_IEC61966-2-1_black_scaled.icc ICC profile • ICC = International Color Consortium wave EXQUIRES (EXtensible QUantitative Image RESampling) 25
Test Images apartments baby boy cabins cat curios dragon footbridge EXQUIRES (EXtensible QUantitative Image RESampling) 26
Test Images man frog garland horse paint shed tower wreck EXQUIRES (EXtensible QUantitative Image RESampling) 27
Dumic, Grgic & Grgic “Hidden influences on image quality when comparing interpolation methods” • Published in 2008 • Performed a re-enlargement test • Did not specify the colour space(s) they used (except that they used one greyscale image) • Looked at upsampler ranks and compared the results "by hand" • Didn’t use correlations • Determined that the choice of downsampler had the largest impact on upsampler rankings EXQUIRES (EXtensible QUantitative Image RESampling) 28
Improving Upon Dumic, Grgic & Grgic • They used the following components: • 4 test images • 3 downsamplers • 3 resampling ratios (2, 4, 8) • 3 upsamplers • 4 metrics (SNR, PSNR, PQS, SSIM) • 1 colour space = 1 toolchain • TOTAL DATA POINTS = 432 EXQUIRES (EXtensible QUantitative Image RESampling) 29
Recommend
More recommend
