Higher-order Segmentation Functionals: Entropy, Color Consistency, Curvature, etc. Yuri Boykov jointly with Andrew Delong M. Tang I. Ben Ayed O. Veksler H. Isack L. Gorelick C. Nieuwenhuis E. Toppe C. Olsson A. Osokin A. Delong
Different surface representations continuous combinatorial mixed optimization optimization optimization s s {0,1} s Z p p p p point cloud graph labeling level-sets mesh labeling on complex on grid
this talk combinatorial optimization s {0,1} p graph labeling on grid Implicit surfaces/bondary
Image segmentation Basics s {0,1} p E(S) E(S) f, f, S S B(S) E(S) f p s p p S Pr(I | Fg) Pr(I | Bg) Pr(I | fg) p f ln p Pr(I | bg) p 4 I
Linear (modular) appearance of region , R ( S ) f S f p s p p Examples of potential functions • Log-likelihoods ln Pr f ( I ) p p • Chan-Vese 2 f ( I c ) p p • Ballooning f 1 p
Basic boundary regularization for B ( S ) w [ s s ] p q s {0,1} pq N p pair-wise discontinuities
Basic boundary regularization for B ( S ) w [ s s ] p q s {0,1} pq N p second-order terms 1 1 [ s s ] s ( s ) ( s ) s p q p q p q quadratic
Basic boundary regularization for B ( S ) w [ s s ] pq p q s {0,1} pq N p second-order terms Examples of discontinuity penalties • Boundary length 1 w pq • Image-weighted 2 exp w ( I I ) pq p q boundary length
Basic boundary regularization for B ( S ) w [ s s ] pq p q s {0,1} pq N p second-order terms • corresponds to boundary length | | a cut – grids [B&K, 2003], via integral geometry t n-links – complexes [Sullivan 1994] • submodular second-order energy w – can be minimized exactly via graph cuts pq [Greig et al.’91, Sullivan’94, Boykov - Jolly’01] s
Submodular set functions Ω any (binary) segmentation energy E(S) S is a set function E: 2
Submodular set functions E : 2 S , T Set function is submodular if for any E ( S T ) E ( S T ) E ( S ) E ( T ) Ω S T Significance : any submodular set function can be globally optimized in polynomial time 9 O ( | | ) [Grotschel et al.1981,88, Schrijver 2000]
Submodular set functions an alternative equivalent definition providing intuitive interpretation: “ diminishing returns ” E : 2 S T Set function is submodular if for any E ( T { v } ) E ( T ) E ( S { v } ) E ( S ) Ω v S T v S S T S T E ( T { v } ) E ( S ) E ( S { v } ) E(T) Easily follows from the previous definition: Significance : any submodular set function can be globally optimized in polynomial time 9 O ( | | ) [Grotschel et al.1981,88, Schrijver 2000]
Graph cuts for minimization of submodular set functions Assume set Ω and 2nd -order (quadratic) function , E ( s ) E ( s s ) , 0 , s p s { 1 } pq p q q ( pq ) N Indicator variables ( , p q ) N Function E(S) is submodular if for any , , , , E ( 0 0 ) E ( 1 1 ) E ( 1 0 ) E ( 0 1 ) pq pq pq pq Significance : submodular 2 nd -order boolean (set) function can be globally optimized in polynomial time by graph cuts 2 O (| N | | | ) [Hammer 1968, Pickard&Ratliff 1973] [Boros&Hammer 2000, Kolmogorov&Zabih2003]
Global Optimization Combinatorial Continuous optimization optimization ? submodularity convexity
Graph cuts for minimization of posterior energy (MRF) 0 , s { 1 } Assume Gibbs distribution over binary random variables p Pr ( s ,..., s ) exp ( E ( S )) S { p | s 1 } for p 1 n Theorem [ Boykov, Delong, Kolmogorov, Veksler in unpublished book 2014?] All random variables s p are positively correlated iff set function E ( S ) is submodular That is, submodularity implies MRF with “ smoothness ” prior
Basic segmentation energy f s w [ s s ] p p pq p q p pq N segment region/appearance boundary smoothness
Higher-order binary segmentation segment region/appearance boundary smoothness this talk Curvature (3-rd order) Appearance Entropy (N-th order) Color consistency (N-th order) Convexity (3-rd order) Cardinality potentials (N-th order) Distribution consistency (N-th order) Connectivity (N-th order) Shape priors (N-th order)
Overview of this talk high-order functionals optimization block-coordinate descent • From likelihoods to entropy [Zhu&Yuille 96, GrabCut 04] • From entropy to color consistency global minimum [our work: One Cut 2014] • Convex cardinality potentials submodular approximations • Distribution consistency [our work: Trust Region 13, Auxiliary Cuts 13] • From length to curvature other extensions [arXiv13]
Given likelihood models unary (linear) term pair-wise (quadratic) term ( | , ) ln | E S Pr( I ) w [ s s ] s { 1 0 , } p s pq p q 0 1 p p p pq N assuming known guaranteed globally optimal S • parametric models – e.g. Gaussian or GMM • non-parametric models - histograms I p RGB image segmentation, graph cut [Boykov&Jolly, ICCV2001]
Beyond fixed likelihood models mixed optimization term pair-wise (quadratic) term ( , , ) ln | E S Pr( I ) w [ s s ] s { 1 0 , } p s pq p q 0 1 p p p pq N extra variables NP hard mixed optimization! • parametric models – e.g. Gaussian or GMM [Vesente et al., ICCV’09] • non-parametric models - histograms I p RGB Models 0 , 1 are iteratively re-estimated (from initial box) iterative image segmentation, Grabcut 0 , S (block coordinate descent ) 1 [Rother , et al. SIGGRAPH’2004]
Block-coordinate descent for , 0 , E ( S ) 1 • Minimize over segmentation S for fixed 0 , 1 , , ln | E ( S ) Pr( I ) w [ s s ] p S pq p q 0 1 p p pq N optimal S is computed using graph cuts, as in [BJ 2001] • Minimize over 0 , 1 for fixed labeling S fixed for S=const E ( S , , ) ln Pr( I | ) ln Pr( I | ) w [ s s ] 0 1 p 0 p 1 pq p q p : s p : s pq N 0 1 p p ˆ ˆ S S p p 1 0 distribution of intensities in distribution of intensities in current bkg. segment = {p:S p =0} S current obj. segment S= {p:S p =1}
Iterative learning of color models 0 , s { 1 } (binary case ) p • GrabCut: iterated graph cuts [Rother et al., SIGGRAPH 04] , , ln | E ( S ) Pr( I ) w [ s s ] 0 1 p S pq p q p p pq N start from models 0 , 1 iterate graph cuts and model re-estimation inside and outside some given box until convergence to a local minimum solution is sensitive to initial box
Iterative learning of color models 0 , s { 1 } (binary case ) p E=1.410×10 6 E=2.41×10 6 E=2.37×10 6 E=1.39×10 6 , 0 , E ( S ) BCD minimization of converges to a local minimum 1 (interactivity a la “snakes”)
Iterative learning of color models , 2 , ,... s { } 0 1 (could be used for more than 2 labels ) p • Unsupervised segmentation [Zhu&Yuille, 1996] using level sets + merging heuristic , , , ... ln | E ( S ) Pr( I ) w [ s s ] | labels | 0 1 2 p S pq p q p p pq N iterate segmentation initialize models 0 , 1 , 2 , and model re-estimation from many randomly sampled boxes until convergence models compete, stable result if sufficiently many
Iterative learning of color models , 2 , ,... s { } 0 1 (could be used for more than 2 labels ) p • Unsupervised segmentation [Delong et al., 2012] using a-expansion (graph-cuts) , , , ... ln | E ( S ) Pr( I ) w [ s s ] | labels | 0 1 2 p S pq p q p p pq N iterate segmentation initialize models 0 , 1 , 2 , and model re-estimation from many randomly sampled boxes until convergence models compete, stable result if sufficiently many
Iterative learning of other models , 2 , ,... s { } 0 1 (could be used for more than 2 labels ) p • Geometric multi-model fitting [Isack et al., 2012] using a-expansion (graph-cuts) , , , ... E ( S ) p - p w [ s s ] | labels | S pq p q 0 1 2 p p pq N initialize plane models 0 , 1 , 2 , from many randomly sampled SIFT matches iterate segmentation in 2 images of the same scene and model re-estimation until convergence models compete, stable result if sufficiently many
Iterative learning of other models , 2 , ,... s { } 0 1 (could be used for more than 2 labels ) p • Geometric multi-model fitting [Isack et al., 2012] using a-expansion (graph-cuts) , , , ... E ( S ) p - p w [ s s ] | labels | S pq p q 0 1 2 p p pq N VIDEO initialize Fundamental matrices 0 , 1 , 2 , from many randomly sampled SIFT matches iterate segmentation in 2 consecutive frames in video and model re-estimation until convergence models compete, stable result if sufficiently many
From color model estimation to entropy and color consistency global optimization in One Cut [Tang et al. ICCV 2013]
Recommend
More recommend
