Spatial Pattern Templates for Recognition of Objects with Regular Structure GCPR 2013 cek and Radim ˇ Radim Tyleˇ S´ ara Center for Machine Perception Czech Technical University in Prague cmp.felk.cvut.cz cek, Radim ˇ 1 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Introduction Observation: Some objects in images have regular layout. cek, Radim ˇ 2 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Introduction Observation: Some objects in images have regular layout. Regularity: Repetition of elements according to simple rules (symmetry). cek, Radim ˇ 2 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Introduction Observation: Some objects in images have regular layout. Regularity: Repetition of elements according to simple rules (symmetry). Task: Incorporate regular contextual cues as a structure prior for recognition. cek, Radim ˇ 2 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Introduction Observation: Some objects in images have regular layout. Regularity: Repetition of elements according to simple rules (symmetry). Task: Incorporate regular contextual cues as a structure prior for recognition. Problem: How to specify a language for complex relations between many object instances of many classes? cek, Radim ˇ 2 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Related Work Shape Grammars Sparse Graphical Models [Gould 2008] [Simon 2011] set of production rules restrictive split layout adjacency, associative potentials grammar complex relations not captured specification Dense Graphical Models [Schmidt 2010] complete, high order cliques learning weights jointly in large graphs intractable [Simon 2011] Simon, Teboul, Koutsourakis, Paragios: Random exploration of the procedural space. IJCV (2011) [Gould 2008] Gould et al.: Multi-class segmentation with relative location prior. IJCV (2008) [Schmidt 2010] Schmidt, Murphy: Convex structure learning: Beyond pairwise potentials. AISTATS (2010) cek, Radim ˇ 3 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Our Approach Annotated Train Set Core Relation Pool Principal Idea: Identify which dense structural relations are important for recognition Learning Segmented Train Set Effective Cliques Spatial Pattern Templates Parsing Input Test Image Output Test Labels Dense Graphical Model cek, Radim ˇ 4 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Problem Formulation Input X – rectified image data S – unsupervised segmentation Output X S L L – class labels for segments in S Probability Model: Conditional Random Field [CRF] find labeling L : S → C that maximizes l 3 l 1 p ( L | X , S ) = 1 �� � l 2 � exp θ j ϕ j ( l q , x q , s q ) Z x 3 j ∈ Φ q x 1 x 2 q ∈ Q ( S ) General CRF topology given by cliques (factors) in Q ( S ) [CRF] Lafferty et al.: CRF: Probabilistic models for segmenting and labeling sequence data. ICML (2001) cek, Radim ˇ 5 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Spatial Pattern Templates SPT = representation for learning dense structural relations 1. Specify core attribute relation functions relations act on attributes of segments tuples cek, Radim ˇ 6 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Spatial Pattern Templates SPT = representation for learning dense structural relations 1. Specify core attribute relation functions relations act on attributes of segments tuples 2. Discretize relation functions value range is split into discrete intervals cek, Radim ˇ 6 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Spatial Pattern Templates SPT = representation for learning dense structural relations 1. Specify core attribute relation functions relations act on attributes of segments tuples 2. Discretize relation functions value range is split into discrete intervals cek, Radim ˇ 6 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Spatial Pattern Templates SPT = representation for learning dense structural relations 1. Specify core attribute relation functions relations act on attributes of segments tuples 2. Discretize relation functions value range is split into discrete intervals 3. Create composite relations A N subsets in Cartesian product of relations A N cek, Radim ˇ 6 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Spatial Pattern Templates SPT = representation for learning dense structural relations 1. Specify core attribute relation functions relations act on attributes of segments tuples 2. Discretize relation functions value range is split into discrete intervals 3. Create composite relations A N subsets in Cartesian product of relations A 4. Template domain indicate allowed combinations N cek, Radim ˇ 6 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Spatial Pattern Templates SPT = representation for learning dense structural relations 1. Specify core attribute relation functions relations act on attributes of segments tuples 2. Discretize relation functions value range is split into discrete intervals 3. Create composite relations A N subsets in Cartesian product of relations A - - 0 AA 0 AN 4. Template domain indicate allowed combinations - 0 NA N 5. Learn weights for each allowed combination cek, Radim ˇ 6 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Aligned Pairs capture pair-wise alignment connect two segments aligned horizontally or vertically relative position relations d 1 , d 2 nesting, overlap possible statistical potential [Tighe11] factor graph ϕ 2 ( u , v ) = θ d 1 d 2 f c ( l u , l v , d 1 , d 2 ) A W N - - - 0 AA 0 AW 0 AN u u u window .64 .14 .05 .03 .07 .02 .02 .04 wall .06 .74 .06 .03 .04 .02 .06 - 0 WW balcony .05 .21 .47 .01 .14 .05 .02 .06 - - 0 WA 0 WN door .04 .02 .02 .60 .32 v v v roof .03 .05 .12 .02 .53 .16 .09 .04 .02 .09 .01 .17 .58 .08 chimney core relation function d sky .10 .10 .80 - - - 0 NN 0 NA shop .09 .05 .06 .29 .01 .51 0 NW window wall balcony door roof chimney sky shop SPT domain label co-occurence [Tighe 2011] Tighe, Lazebnik: Understanding scenes on many levels. ICCV (2011) cek, Radim ˇ 7 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Regular Triplets capture basis for repetitive structure (rows, columns) connect three equally spaced segments relative position relation similarity in segment size generalized associative potential [Kohli 2009] θ c if l u = l v = l w , factor graph ϕ 3 ( u , v , w ) = θ 0 if different , 0 if irregular enforcing same labels in a triplet [Kohli 2009] Kohli, Ladicky, Torr: Robust higher order potentials for enforcing label consistency. IJCV (2009) cek, Radim ˇ 8 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Inference Learning Algorithm piece-wise estimation of parameters θ unary SVM classifier CRF pseudo-likelihood maximization (50 instances) 1 2 600 3 4 5 6 500 7 8 9 10 400 11 12 13 14 300 15 16 17 200 18 19 20 21 100 22 23 24 25 0 123456789 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Aligned Pairs weights θ d 1 d 2 cek, Radim ˇ 9 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Inference Learning Algorithm Decoding Algorithm piece-wise estimation of recognition of labels L parameters θ max-product tree-reweighted unary SVM classifier message passing CRF pseudo-likelihood voting scheme maximization (50 instances) 1 2 600 3 4 5 6 500 7 8 9 10 400 11 12 13 14 300 15 16 17 200 18 19 20 21 100 22 23 24 25 0 123456789 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Aligned Pairs weights θ d 1 d 2 cek, Radim ˇ 9 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Inference Learning Algorithm Decoding Algorithm piece-wise estimation of recognition of labels L parameters θ max-product tree-reweighted message passing unary SVM classifier CRF pseudo-likelihood voting scheme maximization (50 instances) 1 2 600 3 4 5 6 500 7 8 9 10 400 11 12 13 14 300 15 16 17 200 18 19 20 21 100 22 23 24 25 0 123456789 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Aligned Pairs weights θ d 1 d 2 cek, Radim ˇ 9 / 12 Radim Tyleˇ S´ ara, CMP CTU Prague Spatial Pattern Templates for Regular Structure Recognition
Recommend
More recommend
