5/22/2006 S I GGRAPH 2004 I nte rac tive I mag e Cuto ut � Se pa ra te a n o b je c t fro m its b a c kg ro und Yin Li Jian Sun L azy Snapping � Co mpo se the o b je c t o n a no the r ima g e Chi-Keung Tang Heung-Yeung Shum Microsoft Research Asia Microsoft Research Asia Hong Kong University Hong Kong University Carsten Rother Gr abCut Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge- Microsoft Research Cambridge -UK UK I nte rac tive Graph Cut I nte rac tive Graph Cut � (Bo yko v & Jo lly. I CCV’ 01) � Optimize d b y s-t min-c ut a lg o rithm and background Draw foreground and background Draw foreground Graph Cut Graph Cut Image Segmentation Segmentation � (Bo yko v & Jo lly. I CCV’ 01) I nte rac tive Graph Cut Hard Co nstraints � X : Se g me nta tio n. x i ∈ {" obj " , " bkg " } � Ha rd Co nstra int: ∀ ∈ = i O x " obj " i ∀ ∈ = i B x " bkg " i � (Bo yko v e t a l. I CCV’ 01) 1
5/22/2006 I mag e as a We ig hte d Graph S o ft Co nstraints Foreground Foreground (source S) (source S) Image Image Min Cut Min Cut � Minimize the E ne rg y: ∑ ∑ = + λ E ( X ) E ( x ) E ( x , x ) 1 i 2 i j ∈ ∈ i V i , j E ≠ x x i j � E 1 : Re g io n: Co lo r diffe re nc e to use r ma rks Background Background (sink T) (sink T) � E 2 : Bo unda ry: Co lo r simila rity b e twe e n pixe ls Graph: s ource & sink, n-links & t-links Cut=Segmentation: Separate ‘source‘ & ‘sink‘ Energy of cut: sum wieghts of edges Min-Cut Max-Flow: Global minimal enegry in polynomial time We ig hts L azy t-links ∈ ⇒ ∞ i B { i , T } : S napping { i , S } : 0 ∈ ⇒ = i U E ( x ) h ( I ) 1 i x i i L i e t al. S I GGRAPH’ 04 n-links ∝ 2 E ( x , x ) exp( - (I – I ) ) 2 i j 1 2 Min Cut = Minimize Soft Constr aints ke e ping Har d Constr aints L azy S napping L azy S napping � L a zy Sna pping fo r 2. F ine Ste p: L a zy Use rs a . Bo rde r Brush � 2 Ste ps UI : b . Pixe l E diting => Gra ph-Cut 1. Co a rse Ste p: Ob j/ Bkg Ma rking o n b o rde r => Gra ph Cut 2
5/22/2006 We ig hts Pe r-Pix Graph Cut � E 1 : Co lo r diffe re nc e to use r ma rks Inte nsitie s -> Co lo rs Histo g ra m -> “K -me a ns” c luste ring = obj ∝ E ( x " " ) RGB _ dist to c lo se st c luste r c e ntro id 1 i � E 2 : Co lo r simila rity b e twe e n pixe ls F o r ne ig hb o ring pixe ls o f diffe re nt x i 1 = E ( x i x , ) 2 j 2 + 1 C - C i j Pre -S e g me ntatio n Graph Cut o n Re g io ns Graph Cut o n Re g io ns Graph Cut o n Re g io ns 3
5/22/2006 Graph Cut Alg o rithm Re g io n-base d Graph Cut � Adva nta g e s Pe r-pixe l me tho d Re g io n b a se d me tho d � Mo re tha n 10 time s fe we r no de s Pixe ls Sma ll re g io ns � I nsta nt fe e db a c k o f c uto ut re sult Ne ig hb o rs Re g io n c o nne c tio n � Pre -pro c e ssing o ve rhe a d Pixe l c o lo r Re g io n me a n c o lo r � 2~3 se c o nds b a c kg ro und pro c e ssing Co lo r diffe re nc e Re g io n c o lo r diffe re nc e Po lyg o n F itting Divide and Co nque r � F irst ve rte x – b o rde r pixe l with hig he st c urvature � Ne xt ve rtic e s: furthe st b o unda ry pixe l First Step: Object Marking Second Steps: Boundary Editing � Sto p whe n dista nc e < thre sh Input Input Coarse Coarse Refined Refined Image Image Boundary Boundary Boundary Boundary Bo rde r E diting � Brush - Re pla c e po lyg o n se g me nt � Ve rte x E diting : Mo ve / Ad d/ De le te Quickly identify the object Quickly identify the object Control the detail boundary Control the detail boundary => Gra ph Cut o n b o rde r pixe ls Band o f Unc e rtainty Optimizatio n in the Band Pixel Based Graph Cut Segmentation 4
5/22/2006 E dit the Po lyg o n Ve rtic e s E dit the Po lyg o n Ve rtic e s L o w Co ntrast E xample Bo undary E diting Bo undary E diting Vide o De mo (L e ft boy) � F o r L o w Co ntra st c a se : � I n E 2 - Add a te rm to re fle c t dista nc e fro m po lyg o n � Ha rd Ve rte x c o nstra int � Adjust g ra ph so c ut pa sse s thro ug h ve rte x 5
5/22/2006 S ummary: T wo S te ps Vide o De mo (R ight Boy) First Step: Object Marking Second Steps: Boundary Editing Input Small Coarse Editable Refined Input Small Coarse Editable Refined Image Image Regions Regions Boundary Boundary Polygon Polygon Boundary Boundary Region Based Band Pixels Region Based Band Pixels Pre- -Segmentation Segmentation Polygon Fitting Pre Polygon Fitting Graph Cut Graph Cut Graph Cut Graph Cut Pho to mo ntag e Pho to mo ntag e GrabCut GrabCut I nte rac tive F o re gro und E xtrac tio n I nte rac tive F o re gro und E xtrac tio n using I using I te rate d Graph Cuts te rate d Graph Cuts I I te rate d Graph Cut te rate d Graph Cut Gaussian Mixture Mo de ls (GMMs GMMs) ) Gaussian Mixture Mo de ls ( � GMM inste a d o f Histo g ra m (Co lo r mo de l) � Assume distrib utio n is a mixture o f Ga ussia ns ? User Initialization − G ( x ) Gaussian μ Σ , K ∑ = GMM(x) w G ( x ) μ Σ k , k k = k 1 ∑ = w 1 k Graph cuts to GMM estimation infer the for learning , μ Σ � E w , M a lg o rithm – find b e st k k k segmentation colour distributions fo r the g ive n se t o f sa mple s � Gra b Cut – Diffe re nt a ppro a c h 6
5/22/2006 I nc o mple te L abe ling I I te rate d Graph Cuts te rate d Graph Cuts = = φ � Use r spe c ifie s b o rde r => B , U B , F � E 1 – GMMs ( E 2 – No c ha ng e ) � F po pula te s thro ug h ite ra tio ns � Alg o rithm: � So me F pixe ls c a n b e re tra c te d. B c a nno t = = φ 1. I nitia lize B , U B , F , μ Σ I nitia lize GMMs w , k k k 2. Re pe a t (until c o nsta nt e ne rg y) ∀ p ∈ a . U a ssig n b e st G k => 2 K c luste rs E diting (I n c a se o f e rro r): , μ Σ w , b . F o r e a c h c luste r c a lc ula te => 2 GMMs k k k � Use r a dds F, B (b rush) c . F ind Min Cut => U d e c re a se s 3. Apply b o rde r ma tting � Re -c o mpute 4. E na b le use r e diting & re pe a t � Gra ph Cut c a n b e re use d. I te rate d Graph Cuts I te rate d Graph Cuts Gaussian S Gaussian S e paratio n e paratio n Guaranteed to converge R R Iterated Foreground & Background Foreground graph cut Background G Background G 1 2 3 4 Result Energy after each Iteration Ga ussia n Mixture Mo de l (typic a lly K =5) Mo de rate ly straig htfo rward Mo de rate ly straig htfo rward Diffic ult E Diffic ult E xample s xample s e xample s e xample s Camouflage & F ine str uc tur e No te le pathy L ow Contr ast Initial Rectangle Initial Result 7
5/22/2006 Co mpariso n E E valuatio n – valuatio n – L L abe lle d abe lle d Co mpariso n Database Database Boykov and Jolly (2001) GrabCut User Input Result Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.32% Error Rate: 1.25% Error Rate: 0.72% Error Rate: 0.72% Bo rde r Matting Bo rde r Matting Baye s Matting - Chuang e t. al. (2001) xtra c t α -va lue s E ± a lo ng b o rde r � Cre a te U b a nd w � L o c a l re c ta ng le � E stima te G F , G B � μ = α μ + − α μ F α U : ( 1 ) α F B α = μ Σ � G ( ) G( , ) α α U U ind α tha t ma ximize s G U � F B with re spe c t to pixe ls in U Hard Segmentation Band of Uncertainty Soft Segmentation Bo rde r Matting Bo rde r Matting - - GrabCut GrabCut Dynamic Pro g ramming Dynamic Pro g ramming Foreground Noisy alpha-profile 1 Mix t+1 σ t Back- ground DP 0 Δ Result using DP Border Matting Foreground Background Mix T ∑ μ Σ Δ − Δ + σ − σ 2 2 Max : G ( , ) Min : ( ) ( ) α α − − t t 1 t t 1 = Δ σ t 1 , Fit a smooth alpha-profile with parameters Regularisation Noisy alpha-profile 8
5/22/2006 S ummary Matting Re sults S ummary Matting Re sults � ( α sho uld ma tc h U pixe ls G ) U � α sho uld c ha ng e like a so ft ste p func tio n � Ste p func tio n sho uld c ha ng e smo o thly a lo ng c o nto ur Input Bayes Matting GrabCut (no regularization) (with regularization) L azy S napping vs. Grab Cut L azy S napping vs. Grab Cut Thank You L azy Sna pping Gr a bCut Use r Inte r fac e Mar king br ush – F G + BG R e c tangle / lasso – BG only Ove r riding br ush Mar king br ush - [optiona l] Ve r te x e diting Algor ithm R e gion- ba se d Gr a ph Cut Ite r ative Gr aph Cut Bor de r pixe l Graph Cut Pe r for manc e F ully inte r a c tive F ast Inc lude s Pr e - Pr oc e ssing Bor de r Bor de r E diting Bor de r Matting 9
Recommend
More recommend
