Multiscale Conditional Random Fields for Image Labeling Xuming He, - PowerPoint PPT Presentation

Multiscale Conditional Random Fields for Image Labeling Xuming He, Richard Zemel and Miguel A. Carreira-Perpinan Department of Computer Science University of Toronto

Introduction • Image labeling – Classifying every image patch into a finite set of classes • Typical issues – Using local image features – Capturing structures in labels

Random Field Framework • Generative Markov Random Fields (MRFs) Label ( L ) l l j i x i Image ( X ) 1 ∏ ∏ = • Modeling: ( ) ( | ) ( , ) P L , X P x l q l l i i i j Z , i i j = • Labeling: ( | ) ( , ) / ( ) P L X P L X P X

Motivation • Capturing context in different scales – Local constraints vs. Global configurations – Representing context dependency • Building conditional model – Saving modeling resources – Training model discriminatively • Learning context from data – Adapting context representation to images and labels

Overview • Multiscale Conditional Random Fields (mCRF) ( L ) ( | ) ( L ) P P L X P Regional Global G C R label features label features 1 = ( | ) ( | ) ( ) ( ) P L X P L X P L P L C R G Z

Examples of Global Features Rhino/hippo Polar bear Water Snow Vegetation Ground Sky

Examples of Local Features Sky Vegetation Marking Road Building Street Obj. Car

Labeling Images • Inferring Labels L from Image X • Mode of Posterior Marginals Criterion = * arg max ( | ) l P l X i i l i • Approximate Inference – Block Gibbs sampling – Comparable to other methods: Loopy BP, Mean Field.

Parameter Estimation { } = • Supervised Learning with Dataset t t K ( , ), 1 , , L X t N • Conditional Maximum Likelihood ) ∑ θ = θ t t max log( ( | ; )) P L X θ t • Gradient Ascent using MCMC ∂ θ ∂ θ t t log ( | ; ) log ( | ; ) h L X h L X ∑ ∆ θ ∝ − k k k k ∂ θ ∂ θ k t t k t k ( | ) ( | ) P L X P L X θ 0 • Supervised Contrastive Divergence

Experiment Setup • Data set: – Subset of Corel database: African and Arctic wildlife natural scenes. (Training: 60, Testing: 40, 7 label classes) – Sowerby database: Rural and suburban scenes. (Training: 60, Testing: 44, 7 label classes) • Performance comparison: – Generative MRF – Local classifier

Results on Corel Image Data Image Classifier MRF mCRF mCRF confidence Rhino/hippo Polar bear Water Snow Vegetation Ground Sky Classification rate 66.9% 66.2% 80.0% -

Results on Sowerby Image Data Image Classifier MRF mCRF mCRF confidence Sky Vegetation Marking Road Building Street Obj. Car Classification rate 82.4% 81.8% 89.5% -

Summary and Future Work • Summary – Multiscale representation of contextual information – Learning features from image data – Conditional RF model trained discriminatively • Future Work – Comparison with tree-structured models – Invariance in contextual features – Contextual features with adaptive position and scale