Object Recognition Showcase Nicu Sebe University of Amsterdam, The Netherlands Allan Hanbury, Julian Stoettinger TU Wien – PRIP Jaume Amores INRIA - IMEDIA
luminance-based points color-based points? Salient Points
Salient Points • Capture visual “interesting” parts of an image • All points should summarize the image content — Multiple scales: coarse … fine Fine scale Interesting? No not really… Interesting? Hmm… Yes! Coarse scale
Salient Points - Usage • Matching them! — Compare detected salient points � Detect points in different images � Describe these points and compare using a similarity measure � Derive relations between images: — i.e.: S ame scene with different viewpoint; common obj ect(s); etc. • For example: � Obj ect recognition — Different scales: Hierarchical obj ect model � Obj ect tracking � Content Based Image Retrieval (CBIR)
Existing Research • Finding visual “interesting” points is not easy S alient — Mathematical definition? points Corners • Local Image Descriptors – Harris [Harris88], Multi-scale point detection [Mikolaj czyk01], Local gray value invariants [S chmid97], Edge-based region detect or [Tuytelaars04], S US AN [S mith97], Wavelets [S ebe 03], etc. • Local Region Descriptors — S IFT [Lowe04], shape context [Belongie02], moment invariants [Gool96], N-j et [Koenderink87], etc.
Existing Research – Issues • Images are mostly color — Why are the existing salient point techniques luminance-based? — They typically focus on shape saliency rather than color saliency — They cannot distinguish between black-and-white corners (low salient) and red-green corners (high-salient) • Few existing salient point algorithms that use color [Montesions98][Itti98][Heidenman04] — Their results do not differ greatly from the intensity-based methods — Difficulties in combining the information available from the color channels — Many possible color spaces
Research – Affine invariance • Detect regions under common transformations � Translation � Rotation Affine invariance ! � S caling � Viewpoint Viewpoint 1 Related by rotation Viewpoint 2 Detected regions Normalized detected regions
Research - Framework • Existing method by Mikolajczyk — Iterative affine invariant point detector � Multi-scale Harris corner detector � Laplacian characteristic scale selection � S econd moment matrix shape determination Initial region based Iteratively adjust final region on initial scale and scale, position and location shape of region
Research – Framework • Characteristic scale � Convolve with multiple Laplacian of Gaussian kernels: scale trace. � S elect maximum
Research – Framework • Affine deformation — S econd moment matrix � S uppress noise without suppressing the anisotropic shape of a structure. � Eigenvalues represent two principal curvatures of a point: shape normalization! � Calculated using (affine) Gaussian kernels Uniform kernel — Affine invariance � Detect regions that comply to: Affine kernel
Color-based salient points • Color Harris (Weijer04) — Extend calculation of second moment matrix to color � S um gradients of the channels color-based points? luminance-based points What’s the problem?
Evaluation Criteria [Schmid98] • Repeatability — S alient point detection should be stable under varying viewing conditions • Distinctiveness — S alient points should focus on events with a low probability of occurrence Idea: Incorporate color distinctiveness into the design of salient point detectors!!!!!!
Color-based salient points • The efficiency of the salient point detection depends on distinctiveness of the extracted points • At the salient points’ positions, local neighborhoods are extracted and described by local image descriptors • The distinctiveness of a descriptor describes the conciseness of the representation and the discriminative power of the salient points • The distinctiveness is measured from the information content • the information content of an event, v, is equal to : ( ) ( ( ) ) = − I v log p v
Color-based salient points • For luminance-based descriptors the information content is measured by the local two-j et of the local structure [S chmid00] • Due to extra information available in color images, the local one-j et is sufficient ( ) v = R G B R G B R G B x x x y y y Assuming independent probabilities of the 0 th order signal and the 1 st • derivatives, the information content is: ( ( ) ( ) ( ) ) ( ) ( ( ) ) = − = − = f f f f I v log p v log p p p ( R , G , B ) x y • By adapting the saliency map to focus on rare color derivatives, the color distinctiveness of the detector is improved!!!!
Color-based salient points Saliency boosting — Image derivatives that occur equally often should contribute equally to the saliency measure — Vectors with equal information content should have equal influence on the saliency map — Find a transformation g for which it holds: ( ) ( ) ( ) ( ) = ↔ = p f p f ' g f g f ' Color Boosting Saliency: x x x x
Color-based salient points Invariance � distinctiveness — The channels of f x are correlated!!! S hadows, shading, and specularities will have a great influence — There is a need to use different color spaces which will eliminate the influence of these perturbations shadows shading highlights ill. intensity ill. Colour I - - - - - RGB - - - - - rgb + + - + - Ratios + + - + +
Statistics of color images f • The statistics of are computed by looking at the 40.000 images of x the Corel database. opponent colorspace (I,RG,BY) RGB • Isosalient surfaces can be approximated by aligned ellipsoids in decorrelated color spaces.
Statistics of color images ( ) ( ) ( ) ( ) = ↔ = f f f f p p ' g g ' Color Boosting Saliency: x x x x opponent colorspace RGB λ ⎛ ⎞ 0 0 ( ) = ⎜ ⎟ ( ) 1 λ g f ⎜ ⎟ h f 0 0 Color Boosting function: 2 x x ⎜ ⎟ λ ⎝ ⎠ 0 0 3
Statistics of color images spherical opponent HSI λ 0.85 0.85 0.86 1 λ 0.52 0.52 0.51 2 λ 0.10 0.065 0.066 3 • Opponent color space was to perform best [vdWeijer04] — One of the components is still the intensity (although, with a very low weight, i.e., 0.065) • Investigate a more invariant color space which has no intensity information anymore: color ratios • The goal is to analyse the tradeoff between invariance and distinctiveness
Color constancy: Color Ratios x x x x x x R G R B G B 1 2 1 2 1 2 = = = m , m , m 1 2 3 x x x x x x R G R B G B 2 1 2 1 2 1 Taking the natural logarithm of both sides results for m in : 1 ⎛ ⎞ x x R G 1 2 ⎜ ⎟ = = + − − = x x x x ln m ln ln R ln G ln R ln G ⎜ ⎟ 1 2 2 2 1 x x ⎝ ⎠ R G 2 1 + − + = x x x x ln R ln G (ln R ln G ) 1 2 2 2 x x ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ∂ 1 2 x x R R R R R 1 2 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ − = − = ln ln ln ln ln ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ∂ x x ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ G G G G x G 1 2
Color constancy: Derivatives Funt and Finlayson Gevers and Smeulders (Mondrian-world) (3D world) ⎛ ⎞ ⎛ ⎞ ∂ ∂ R ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ln ⎜ ⎟ ⎜ ⎟ ln R ∂ ⎜ ⎟ ⎝ ⎠ x G ∂ ⎜ ⎟ x ⎜ ⎟ ⎜ ⎟ ∂ ⎛ ⎞ ∂ ⎜ ⎟ R ⎜ ⎟ ⎜ ⎟ ln G ln ⎜ ⎟ ⎜ ⎟ ∂ ∂ x ⎝ ⎠ ⎜ ⎟ x B ⎜ ⎟ ∂ ⎜ ⎟ ⎜ ⎟ ⎛ ⎞ ∂ ⎜ ⎟ ln B G ⎜ ⎟ ∂ ⎜ ⎟ ⎝ ⎠ ln x ⎜ ⎟ ⎜ ⎟ ∂ ⎝ ⎠ ⎝ x B ⎠
Saliency boosted points input car-image RGB-based (first 20 points) saliency boosting (first 4 points)
saliency boosting RGB-based Saliency boosted points
Research - Approach • Use different corner detectors in the framework — Intensity: Harris, S US AN — Color: 2 colorHarris variants (colOppHarris, colRatHarris) • Evaluation — Repeatability under common transformations (invariance) � Test sets for different common variations in imaging conditions — Blur, Lighting, Rotation/ S caling, viewing angle, JPEG compression — Information content of the detected regions (distinctiveness) � Detect lots of regions, estimate entropy from them. — Complexity
Intensity-based corner detectors • Harris corner detector — S econd moment matrix (S MM) at certain scale — Eigenvalues of S MM represent principal curvatures � Detect regions with high gradient in different directions • Discrete low-level corner detector [Smith 97] — Fundamentally different from Harris detector — Circular mask — Determine the area of the mask with a similar value as the center � Derive cornerness measure from it
Recommend
More recommend
