Hierarchical Probabilistic Models for Object Segmentation S. M. Ali - PowerPoint PPT Presentation

Hierarchical Probabilistic Models for Object Segmentation S. M. Ali Eslami Christopher K. I. Williams Institute for Adaptive and Neural Computation School of Informatics The University of Edinburgh August 8, 2010

Classification

Localisation

Segmentation

Chicken and egg problem Localisation Segmentation Classification Ali Eslami (Edinburgh) 6 of 41

Chicken and egg problem (Panoramio/nicho593) What is this? Ali Eslami (Edinburgh) 7 of 41

Chicken and egg problem (Panoramio/nicho593) Segment this Ali Eslami (Edinburgh) 8 of 41

Outline 1. The task 2. Related research 3. The approach 4. Current progress 5. Discussion Ali Eslami (Edinburgh) 9 of 41

The Segmentation Task (Pascal VOC, Everingham et al., 2010) Ali Eslami (Edinburgh) 10 of 41

The segmentation task Object class labelling Ali Eslami (Edinburgh) 11 of 41

The segmentation task Foreground/background labelling Ali Eslami (Edinburgh) 12 of 41

The segmentation task The image X The segmentation S 2 1 2 1 1 1 2 1 2 1 1 1 2 1 1 1 1 1 1 2 2 1 1 1 1 Ali Eslami (Edinburgh) 13 of 41

Outline 1. The task 2. Related research 3. The approach 4. Current progress 5. Discussion Ali Eslami (Edinburgh) 14 of 41

Related research ◮ Continuity-based methods binary potentials S unary potentials X p ( S | X )= 1 p ( X , S ) or Z exp {− E ( X , S ) } ◮ Shape-based methods ◮ Global models of shape ◮ Parts-based models of shape Ali Eslami (Edinburgh) 15 of 41

Related research ◮ Continuity-based methods ◮ Shape-based methods ◮ Global models of shape Active Shape and Appearance Models (Cootes et al., 1995) ◮ Parts-based models of shape Ali Eslami (Edinburgh) 15 of 41

Related research ◮ Continuity-based methods ◮ Shape-based methods ◮ Global models of shape ◮ Parts-based models of shape Layered Pictorial Structures (Kumar et al., 2005) Ali Eslami (Edinburgh) 15 of 41

Related research ◮ Continuity-based methods ◮ Shape-based methods ◮ Global models of shape ◮ Parts-based models of shape Multiple Cause Vector Quantization (Ross and Zemel, 2006) Ali Eslami (Edinburgh) 15 of 41

Related research ◮ Continuity-based methods ◮ Shape-based methods ◮ Global models of shape ◮ Parts-based models of shape Fragment CRF (Levin and Weiss, 2009) Ali Eslami (Edinburgh) 15 of 41

Related research Summary Model Continuity Shape Parts Part shape LSM (Frey et al., 2003) � – FA ISM (Leibe et al., 2004) � – fragments � ∼ – exemplars GrabCut (Rother et al., 2004) � OBJCUT (Kumar et al., 2005) � � – PS � LOCUS (Winn and Jojic, 2005) � � – mask LHRF (Kapoor and Winn, 2006) � � – part biases � ∼ – CRF LCRF (Winn and Shotton, 2006) � SPCRF (Fulkerson et al., 2009) � FCRF (Levin and Weiss, 2009) � � – fragments � ∼ – exemplars DPMCRF (Larlus et al., 2009) � – DPM � Ali Eslami (Edinburgh) 16 of 41

Related research Summary Model Continuity Shape Parts Part shape LSM (Frey et al., 2003) � – FA ISM (Leibe et al., 2004) � – fragments � ∼ – exemplars GrabCut (Rother et al., 2004) � OBJCUT (Kumar et al., 2005) � � – PS � LOCUS (Winn and Jojic, 2005) � � – mask LHRF (Kapoor and Winn, 2006) � � – part biases � ∼ – CRF LCRF (Winn and Shotton, 2006) � SPCRF (Fulkerson et al., 2009) � FCRF (Levin and Weiss, 2009) � � – fragments � ∼ – exemplars DPMCRF (Larlus et al., 2009) � – DPM � Ali Eslami (Edinburgh) 16 of 41

Outline 1. The task 2. Related research 3. The approach 4. Current progress 5. Discussion Ali Eslami (Edinburgh) 17 of 41

Approach Shape model type Three dimensional Two dimensional Concerned with tractability Ali Eslami (Edinburgh) 18 of 41

Approach Part shape variability Need to model part shape variability Ali Eslami (Edinburgh) 19 of 41

Approach Aspect variability Rectangular Circular Same object, different outlines Ali Eslami (Edinburgh) 20 of 41

Approach Summary Model overview 1. Capture the object’s shape using a number of deformable parts, 2. Combine models of different viewpoints in a mixture, 3. Use this as prior on a random field. Goal Learning of dense object class shape and parts from variable, realistic datasets of images. ◮ Useful for both object segmentation and object parsing . ◮ More expressive power. Ali Eslami (Edinburgh) 21 of 41

Current progress 1. The task 2. Related research 3. The approach 4. Current progress 5. Discussion Ali Eslami (Edinburgh) 22 of 41

Multiple Transformed Masks and Appearances Task To learn the shapes of the parts and infer their positions and appearances. Ali Eslami (Edinburgh) 23 of 41

Multiple Transformed Masks and Appearances Schematic diagram 0 1 2 M 0 1 2 0 2 T S 0 1 2 A X Ali Eslami (Edinburgh) 24 of 41

Multiple Transformed Masks and Appearances 0 1 2 M 0 1 2 0 2 T S 0 1 2 A X ( T ℓ m ℓ ) d p ( s ℓ d = 1 | T , θ ) = � L k =0 ( T k m k ) d L � N ( x d ; ( Wa ℓ + µ ) d , Ψ d ) s ℓ d p ( x d | A , s d ) = l =0 Ali Eslami (Edinburgh) 25 of 41

Multiple Transformed Masks and Appearances Learning Z i = { A i , S i , T i } θ = { M } Use Expectation Maximisation algorithm to find a setting of the masks that approximately maximises the likelihood of the parameters given the data p ( D | θ ): 1. Expectation: Evaluate p ( Z i | X i , θ old ), 2. Maximisation: Find arg max θ Q ( θ , θ old ) where n Q ( θ , θ old ) = � � p ( Z i | X i , θ old ) ln p ( X i , Z i | θ ) . i =1 Z i Ali Eslami (Edinburgh) 26 of 41

Multiple Transformed Masks and Appearances Inference Goal Wish to find p ( Z | X , θ ) = p ( A , S , T | X , θ ). Approximate Instead approximate p ( A , S , T | X , θ ) by sampling in two steps: 1. Approximate p ( T | X , θ ) and draw K T | X samples of T , 2. For each sample T ( k ) , draw from K A , S | T samples from p ( S | A , T , X , θ ) and p ( A | S , T , X , θ ). K T | X K A , S | T 1 1 � � δ ( A ( k 2 ) , S ( k 2 ) , T ( k 1 ) ) p ( A , S , T | X , θ ) ≃ K T | X K A , S | T k 1 =1 k 2 =1 Ali Eslami (Edinburgh) 27 of 41

Multiple Transformed Masks and Appearances Inference Goal Wish to find p ( Z | X , θ ) = p ( A , S , T | X , θ ). Approximate Instead approximate p ( A , S , T | X , θ ) by sampling in two steps: 1. Approximate p ( T | X , θ ) and draw K T | X samples of T , ◮ Na¨ ıve implementation exponential in L , use greedy algorithm (Williams and Titsias, 2004) instead. 2. For each sample T ( k ) , draw from K A , S | T samples from p ( S | A , T , X , θ ) and p ( A | S , T , X , θ ). K T | X K A , S | T 1 1 δ ( A ( k 2 ) , S ( k 2 ) , T ( k 1 ) ) � � p ( A , S , T | X , θ ) ≃ K T | X K A , S | T k 1 =1 k 2 =1 Ali Eslami (Edinburgh) 27 of 41

Multiple Transformed Masks and Appearances Results ◮ Dataset of 30 images: n = 30. ◮ Transformations discretised into 3 vertical translations: J = 3. ◮ Running time ∼ 3 minutes: 10 EM iterations. 2 2 1.5 1.5 1 1 0.5 0.5 0 0 Mask for layer 1, m 1 Mask for layer 2, m 2 Ali Eslami (Edinburgh) 28 of 41

Multiple Transformed Masks and Appearances Results Ali Eslami (Edinburgh) 29 of 41

Future work 1. Learning inter-part relationships. 2. Incorporating richer part shape models. 3. Determining the number of parts. 4. Incorporating low-level image features. 5. Modelling aspect variability. Ali Eslami (Edinburgh) 30 of 41

Future work 1. Learning inter-part relationships. 2. Incorporating richer part shape models. 3. Determining the number of parts. 4. Incorporating low-level image features. 5. Modelling aspect variability. Ali Eslami (Edinburgh) 30 of 41

Future work 1. Learning inter-part relationships. 2. Incorporating richer part shape models. 3. Determining the number of parts. 4. Incorporating low-level image features. 5. Modelling aspect variability. Ali Eslami (Edinburgh) 30 of 41

Future work 1. Learning inter-part relationships. 2. Incorporating richer part shape models. 3. Determining the number of parts. 4. Incorporating low-level image features. 5. Modelling aspect variability. Ali Eslami (Edinburgh) 30 of 41

Future work 1. Learning inter-part relationships. 2. Incorporating richer part shape models. 3. Determining the number of parts. 4. Incorporating low-level image features. 5. Modelling aspect variability. Ali Eslami (Edinburgh) 30 of 41

Questions

Bibliography I Cootes, T., Taylor, C., Cooper, D. H., and Graham, J. (1995). Active shape models—their training and application. Computer Vision and Image Understanding , 61:38–59. Everingham, M., Gool, L. V., Williams, C. K. I., Winn, J., and Zisserman, A. (2010). The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision , 88:303–338. Frey, B. J., Jojic, N., and Kannan, A. (2003). Learning appearance and transparency manifolds of occluded objects in layers. In IEEE Conference on Computer Vision and Pattern Recognition 2003 , pages 45–52. Fulkerson, B., Vedaldi, A., and Soatto, S. (2009). Class Segmentation and Object Localization with Superpixel Neighborhoods. In International Conference on Computer Vision 2009 , pages 670–677. Ali Eslami (Edinburgh) 32 of 41