Submodularity beyond submodular energies: Coupling edges in graph - PowerPoint PPT Presentation

Graph Cuts Cooperative Cuts Optimization Applications Submodularity beyond submodular energies: Coupling edges in graph cuts Stefanie Jegelka and Jeff Bilmes Max Planck Institute for Intelligent Systems T¨ ubingen, Germany University of Washington Seattle, USA 1 / 18

Graph Cuts Cooperative Cuts Optimization Applications local pairwise random fields . . . 2 / 18

Graph Cuts Cooperative Cuts Optimization Applications 3 / 18

Graph Cuts Cooperative Cuts Optimization Applications Random Walker Curvature reg. Graph Cut 3 / 18

Graph Cuts Cooperative Cuts Optimization Applications Random Walker Curvature reg. Graph Cut 3 / 18

Graph Cuts Cooperative Cuts Optimization Applications Markov Random Fields and Energies p ( x | z ) ∝ exp( − E Ψ ( x ; z )) s x ∗ = arg min MAP x E Ψ ( x ; z ) t 4 / 18

Graph Cuts Cooperative Cuts Optimization Applications Markov Random Fields and Energies p ( x | z ) ∝ exp( − E Ψ ( x ; z )) s x ∗ = arg min MAP x E Ψ ( x ; z ) � � E ( x ; z ) = Ψ i ( x i ) + Ψ ij ( x i , x j ) i ( i , j ) ∈N t 4 / 18

Graph Cuts Cooperative Cuts Optimization Applications Markov Random Fields and Energies p ( x | z ) ∝ exp( − E Ψ ( x ; z )) s x ∗ = arg min MAP x E Ψ ( x ; z ) � � E ( x ; z ) = Ψ i ( x i ) + Ψ ij ( x i , x j ) i ( i , j ) ∈N � � E ( x ; z ) = w e + w e t e ∈ Γ x ∩E t e ∈ Γ x ∩E n 4 / 18

Graph Cuts Cooperative Cuts Optimization Applications Markov Random Fields and Energies p ( x | z ) ∝ exp( − E Ψ ( x ; z )) s x ∗ = arg min MAP x E Ψ ( x ; z ) 1 1 1 � � E ( x ; z ) = Ψ i ( x i ) + Ψ ij ( x i , x j ) 1 1 i ( i , j ) ∈N 1 � � E ( x ; z ) = w e + w e t e ∈ Γ x ∩E t e ∈ Γ x ∩E n 4 / 18

Graph Cuts Cooperative Cuts Optimization Applications s 1 1 1 1 1 1 t 5 / 18

Graph Cuts Cooperative Cuts Optimization Applications s 1 1 1 1 1 1 t 5 / 18

Graph Cuts Cooperative Cuts Optimization Applications s 1 1 1 1 1 1 t 5 / 18

Graph Cuts Cooperative Cuts Optimization Applications s 1 1 1 1 1 1 t Couple edges globally 5 / 18

Graph Cuts Cooperative Cuts Optimization Applications Richer Cuts: Cooperative Cuts � E ( x ) = w ( e ) e ∈ Γ x s = w (Γ x ) 1 1 1 1 1 1 t 6 / 18

Graph Cuts Cooperative Cuts Optimization Applications Richer Cuts: Cooperative Cuts � E ( x ) = w ( e ) e ∈ Γ x s = w (Γ x ) 1 1 1 E f ( x ) = f (Γ x ) 1 1 1 submodular function on edges t 6 / 18

Graph Cuts Cooperative Cuts Optimization Applications Richer Cuts: Cooperative Cuts � E ( x ) = w ( e ) e ∈ Γ x s = w (Γ x ) 1 1 1 E f ( x ) = f (Γ x ) 1 1 1 submodular function on edges t non-submodular & global energy 6 / 18

Graph Cuts Cooperative Cuts Optimization Applications Coupling via Submodularity t s 7 / 18

Graph Cuts Cooperative Cuts Optimization Applications Coupling via Submodularity e e A A B B f ( A ∪ e ) − f ( A ) ≥ f ( A ∪ B ∪ e ) − f ( A ∪ B ) Graph Cuts: LHS = RHS “it does not matter which other edges are cut” t s 8 / 18

Graph Cuts Cooperative Cuts Optimization Applications Coupling via Submodularity e e A A B B f ( A ∪ e ) − f ( A ) ≥ f ( A ∪ B ∪ e ) − f ( A ∪ B ) Graph Cuts: LHS = RHS “it does not matter which other edges are cut” t submodularity: s reward co-occurrence structure 8 / 18

Graph Cuts Cooperative Cuts Optimization Applications Generality Special cases of cooperative cuts: labels features (robust) P n potentials s (Kohli et al. ’07,’09) ... label costs 1 1 1 (Delong et al. ’11) 1 1 boundary discrete versions of norm-based 1 cuts (Sinop & Grady ’07) t . . . 9 / 18

Graph Cuts Cooperative Cuts Optimization Applications Optimization? 10 / 18

Graph Cuts Cooperative Cuts Optimization Applications Optimization? ( s , t )-cut Γ ⊆ E with min cost f (Γ). Theorem Minimum Cooperative Cut is NP-hard. 10 / 18

Graph Cuts Cooperative Cuts Optimization Applications Optimization Γ 0 = ∅ ; repeat compute upper bound ˆ f i ≥ f based on Γ i − 1 ; until convergence ; ˆ f i (Γ i − 1 ) = f (Γ i − 1 ) 11 / 18

Graph Cuts Cooperative Cuts Optimization Applications Optimization Γ 0 = ∅ ; repeat compute upper bound ˆ f i ≥ f based on Γ i − 1 ; Γ i ∈ argmin { ˆ f i (Γ) | Γ a cut } ; // Min-cut! i = i + 1; until convergence ; ˆ f i (Γ i − 1 ) = f (Γ i − 1 ) 11 / 18

Graph Cuts Cooperative Cuts Optimization Applications Optimization Γ 0 = ∅ ; repeat compute upper bound ˆ f i ≥ f based on Γ i − 1 ; Γ i ∈ argmin { ˆ f i (Γ) | Γ a cut } ; // Min-cut! i = i + 1; until convergence ; Worst-case approximation bound: | Γ ∗ | for ν = min e ∈ Γ ∗ ρ e ( E\ e ) 1+( | Γ ∗ |− 1) ν E f ( x ∗ ) ≤ E f ( x ) max e ∈ C ∗ f ( e ) 11 / 18

Graph Cuts Cooperative Cuts Optimization Applications Image Segmentation Random Walker Curvature reg. Graph Cut 12 / 18

Graph Cuts Cooperative Cuts Optimization Applications Image Segmentation Random Walker Curvature reg. Graph Cut prefer congruous boundaries 12 / 18

Graph Cuts Cooperative Cuts Optimization Applications Selective Discount for Congruous Boundaries s � � E w ( x ) = w e + λ w e e ∈ Γ ∩E t e ∈ Γ ∩E n � f (Γ ∩ E n ) E f ( x ) = w e + λ e ∈ Γ ∩E t t 13 / 18

Graph Cuts Cooperative Cuts Optimization Applications Selective Discount for Congruous Boundaries s � � E w ( x ) = w e + λ w e e ∈ Γ ∩E t e ∈ Γ ∩E n � f (Γ ∩ E n ) E f ( x ) = w e + λ e ∈ Γ ∩E t t discount for co-occurring similar edges no discount for dissimilar edges 13 / 18

Graph Cuts Cooperative Cuts Optimization Applications Structured Discounts groups S i of edges � f (Γ) = i f i (Γ ∩ S i ) 150 100 50 0 0 100 200 300 400 14 / 18

Graph Cuts Cooperative Cuts Optimization Applications Structured Discounts groups S i of edges � f (Γ) = i f i (Γ ∩ S i ) 150 100 50 0 0 100 200 300 400 14 / 18

Graph Cuts Cooperative Cuts Optimization Applications Structured Discounts groups S i of edges � f (Γ) = i f i (Γ ∩ S i ) 150 100 50 0 0 100 200 300 400 14 / 18

Graph Cuts Cooperative Cuts Optimization Applications Some Results: Shading Graph Cut CoopCut 7 . 39% 2 . 23% 7 . 65% 3 . 50% 15 / 18

Graph Cuts Cooperative Cuts Optimization Applications Some Results: Shading gray color high-freq Graph Cut: no discount 14.03 3.41 2.56 CoopCut (1 group): discount 11.58 2.95 1.49 structured CoopCut (15 groups): 3.63 1.69 1.27 discount Graph Cut CoopCut 5 . 08% 0 . 64% 16 / 18

Graph Cuts Cooperative Cuts Optimization Applications Shrinking bias � � 50 ψ i ( x i ) + λ w e GC twig 14 CoopC twig GC total e ∈ Γ x i 12 CoopC total 40 total error (%) twig error (%) 10 30 8 20 6 4 10 2 2.5 0 0 0 0.5 0.5 1 1 1.5 1.5 2 2 2.5 Graph Cut λ 17 / 18

Graph Cuts Cooperative Cuts Optimization Applications Shrinking bias � � 50 ψ i ( x i ) + λ w e GC twig 14 CoopC twig GC total e ∈ Γ x i 12 CoopC total 40 total error (%) twig error (%) 10 30 8 20 6 4 10 2 2.5 0 0 0 0.5 0.5 1 1 1.5 1.5 2 2 2.5 Graph Cut λ 17 / 18

Graph Cuts Cooperative Cuts Optimization Applications Shrinking bias � 50 ψ i ( x i ) + λ f (Γ x ) GC twig 14 CoopC twig GC total i 12 CoopC total 40 total error (%) twig error (%) CoopCut 10 30 8 20 6 4 10 2 2.5 0 0 0 0.5 0.5 1 1 1.5 1.5 2 2 2.5 Graph Cut λ 17 / 18

Graph Cuts Cooperative Cuts Optimization Applications Shrinking bias � 50 ψ i ( x i ) + λ f (Γ x ) GC twig 14 CoopC twig GC total i 12 CoopC total 40 total error (%) twig error (%) CoopCut 10 30 8 20 6 4 10 2 2.5 0 0 0 0.5 0.5 1 1 1.5 1.5 2 2 2.5 Graph Cut λ 17 / 18

Graph Cuts Cooperative Cuts Optimization Applications Summary: Coupling Edges in Graph Cuts global, non-submodular family of energies NP-hard, but . . . graph structure indirect submodularity → efficient approximation algorithm applications guide segmentations via edge coupling 18 / 18