EE 6882 Visual Search Engine Prof. Shih Fu Chang, Feb. 6 th 2012 - PDF document

2/6/2012 EE 6882 Visual Search Engine Prof. Shih ‐ Fu Chang, Feb. 6 th 2012 Lecture #3  Evaluation Metrics  Local Features: Corner Detector, SIFT  Local Feature Matching (Many slides from A. Efors, W. Freeman, C. Kambhamettu, L. Xie, and likely others) (Slides preparation assisted by Rong ‐ Rong Ji) Performance evaluation Sample results by Zhou Sha Given a search result list, how to measure performance? 2 1

2/6/2012 Evaluation search Ground truth  V n 1 " Relevant" DB  n  0 " Irrelevant " 0 N - 1 N Images K Results   K  1 Detection A V  n  n 0   K  1  False Alarms B ( 1 V )   n n 0    N 1  Misses C V A  ( )  n n 0    N 1   Correct Dismissals D V B ( ( 1 ) )   n n 0   R A A C /( ) Recall  “Relevant   P A A B Results” /( ) “Returned” Precision    F B A B /( ) D False C A  B  P R Combined   F  1 P R ( ) / 2 Evaluation Measures Precision at depth K    k 1 P V K ( ) /  k n 0 n Precision Recall Curve P R Receiver Operating Characteristic (ROC Curve) R (Recall) F (false) 2

2/6/2012 Average Precision AP approximates areas under PR curve K 1    AP ( P V ) R : total # of relevant data j j min( K , R )  j 1 Ranked list of results Example: D D D D ... ... D 15 8 63 21 s Ground truth 1 0 1 1 0 0 0 Precision 1/1 1/2 2/3 3/4 3/5 3/6 3/7 Precision  P  i AP 1.0 3 j 0 1 2 3 4 5 6 7 Evaluation Metric: Average Precision  Observations (AP)  AP depends on the rankings of relevant data and the size of the relevant data set. E.g., R= 10 + + + + + - - - - - - - - - - Case I: + + + + + Affected - + - + - + - + - + - + - + - + - + - + by ranks Case II: - - - - - - - - - - + + + + + + + + + Case III: Same top … + + + - - - - - - - - - - + + + Case IV: ranks, … different + + + + + + - - - - - - - - - - + + + Case V: recall 3

2/6/2012 Texture What is texture?  “stylized subelements, repeated in meaningful ways”  May have quasi ‐ stochastic macro structure (e.g. bricks), each with stochastic  micro structure Why texture?  Application to satellite images, medical images  Useful for describing and reproducing contents of real world images, i.e., clouds,  fabrics, surfaces, wood, stone Challenging issues  Rotation and scale variance (3D)  Segmentation/extraction of texture regions from images  Texture in noise  A wide range of filters for textures Randen, T. and Husøy, J. H. 1999. Filtering for Texture Classification: A Comparative Study. IEEE Trans. Pattern Anal. Mach. Intell. 21, 4 (Apr. 1999), 291 ‐ 310. Tamura Texture Quadrature mirror filters   Zernike moments Discrete cosine transform   Steerable filters Co-occurrence matrices   Ring/wedge filters  Gabor filter banks  wavelet transforms  4

2/6/2012 Filter approaches for texture description  Fourier Transform F(u,v) Energy Distribution  y  Angular features (directionality)    2 V F ( u , v ) dudv   1 2 where ,  x   v      1 tan   1   2 u  Radial features (coarseness)  y   2 V r F u v dudv ( , ) r 1 2 r where ,    2 2  r u v r x 1 2 Gabor filters  Gaussian windowed Fourier Transform  Dyadic partitions of the spatio ‐ frequency space  Basis filters are product/conv of Fourier basis images and Gaussians x = Gabor filters       1 /( 2 ); 1 /( 2 ) Different Frequency u x v y 5

2/6/2012 Example: Filter Responses Input Filter image bank from Forsyth & Ponce Local Features  What are good local features?  Distinct, interesting content  Repeatable (invariant)  Precise locations – sensitive to position shift  Aperture Problem – information within a small window often insufficient for determining true motion or matching The barber pole illusion Multiple motion Corners help shift up shift left hypotheses exist Images of Elizabeth Johnson 6

2/6/2012 Corner Detection  Types of local image windows  Flat : Little or no brightness change  Edge : Strong brightness change in single direction  Flow : Parallel stripes  Corner/spot : Strong brightness changes in orthogonal directions  Basic idea  Find points where two edges meet  Look at the gradient behavior over a small window (Slide of A. Efros) Harris Detector: Basic Idea “flat” region: “edge”: “corner”: no change in all no change along the significant change in directions edge direction all directions (Slide of A. Efros) 7

2/6/2012 Harris Detector: Mathematics Change of intensity for the shift [u,v]:        2 E u v ( , ) w x y ( , ) I x ( u y , v ) I x y ( , ) x y , Window Shifted Intensity function intensity Window function w(x,y) = or 1 in window, 0 outside Gaussian Harris Corner Detector  Taylor’s expansion      2 E u v w x y I x u y v I x y ( , ) ( , )[ ( , ) ( , )] x y ,     2 2 2 w x y I u ( , )[ I v O u ( , v )] x y x y , 2 2    E u v Au Cuv Bv ( , ) 2   2 A w x y I ( , ) ( , ) x y x x y ,   2 B w x y I ( , ) ( , ) x y y x y ,   C w x y I ( , ) ( , ) x y I ( , ) x y x y x y , 8

2/6/2012 Harris Detector: Mathematics Taylor’s Expansion: For small shifts [u,v ] we have a bilinear approximation:   u    E u v u v M ( , ) ,     v where M is a 2  2 matrix computed from image derivatives:   2 I I I   x x y   M w x y ( , ) 2   I I I   x y , x y y Harris Detector: Mathematics Intensity change in shifting window: eigenvalue analysis   u    1 >  2 – eigenvalues of M  E u v u v M ( , ) ,     v If we try every possible shift, the direction of fastest change is  1 Ellipse E(u,v) = const (  1 ) -1/2 (  2 ) -1/2 (Slide of K. Efros) 9

2/6/2012 Harris Detector: Mathematics  2 Classification of image “Edge”  2 >>  1 points using eigenvalues of “Corner” M:  1 and  2 are large,  1 ~  2 ; E increases in all directions  1 and  2 are small; E is almost constant “Edge” “Flat”  1 >>  2 in all directions region  1 (Slide of A. Efros) Harris Detector: Another Interpretation- Optical Flow Model image sequence as a spatiotemporal function I(x,y,t)     dI I dx I dy I       dt x dt y dt t  Assume no change at displaced point, similar optical flow in local area dI    dt  I u I v I 0 0 x y t dx dy where  ( , ) u v ( , ) Opt ical Flow dt dt 10

2/6/2012 Harris Detector: Interpretation based on Tracking  Lucas-Kanade Tracking: local consistency of optical flow – (u,v) remain constant within a local patch     2 Min Min I u I v I E(u,v)= ( ) x y t ( , ) u v ( , ) u v  x y ,            I I I I u I I x x x y x t          I I I I   v I I     x x y y y t M  Optimal (u, v) satisfies Lucas-Kanade equation (Slide of W. Freeman ) Conditions for solvability  Optimal (u, v) satisfies Lucas-Kanade equation  When is this Solvable? M should be invertible  eigenvalues  1 ,  2 of M should not be too small  M should be well-conditioned   1 /  2 should not be too large (  1 = larger eigenvalue)  (Slide of Khurram Hassan-Shafique ) 11

2/6/2012 Harris Detector: Mathematics Measure of corner response:   M det   2  R det M k trace M R M Trace Or    M det    M det 1 2 1 2     M trace     M trace 1 2 1 2 (k – empirical constant, k = 0.04-0.06) Harris Detector: Mathematics  2 “Edge” “Corner” • R depends only on eigenvalues of R < 0 M • R is large for a corner R > 0 • R is negative with large magnitude for an edge • |R| is small for a flat region “Flat” “Edge” |R| small R < 0  1 (Slide of K. Efros) 12

2/6/2012 Edge – large gradients, all the same – large  1 , small  2 Low texture region – gradients have small magnitude – small  1 , small  2 13

2/6/2012 High textured region – gradients are different, large magnitudes – large  1 , large  2 Harris Detector: Summary Average intensity change in direction [ u,v ] can be expressed as a  bilinear form:   u    E u v u v M   ( , ) ,   v   2 I I I Describe a point in terms of eigenvalues of M :    x x y   M w x y ( , ) measure of corner response 2  I I I    x y , x y y A good (corner) point should have a large intensity change in all  directions , i.e. R should be large positive          2 R k 1 2 1 2 (k – empirical constant, k = 0.04-0.06) (Slide of K. Efros) 14