“Convex Formulation of Continuous Multi-label Problems” by Pock, Schoenemann, Graber, Bischof, Cremers (2008) Pascal Getreuer Department of Mathematics University of California, Los Angeles getreuer@math.ucla.edu Pascal Getreuer (UCLA) Functional Lifting 1 / 28
A General Problem Let G = ( V , E ) be a graph with nodes x ∈ V and edges ( x , y ) ∈ E , and let L be a node label set, and consider � � � � � � u : V→ L E ( u ) = min R u ( x ) − u ( y ) + u ( x ) , x (1) ρ x ∈V ( x , y ) ∈E where R is convex and ρ is arbitrary. Applications include segmentation, stereo estimation, and denoising. Ishikawa showed that even though E is nonconvex , a global minimizer of (1) can be found with graph cuts. Pascal Getreuer (UCLA) Functional Lifting 2 / 28
Applications � � � � � � min R u ( x ) − u ( y ) + ρ u ( x ) , x u : V→ L ( x , y ) ∈E x ∈V Multiphase segmentation: c ℓ 1 , c ℓ 2 , . . . are given segment intensities ρ ( u , x ) = | I ( x ) − c u | 2 u is the segment label Stereo estimation: I L and I R are a stereo pair of left and right images � x + ( u �� � ρ ( u , x ) = � I L ( x ) − I R 0 ) u is the displacement � Multiplicative noise removal: f = n · u exact with n ∼ Gamma ρ ( u , x ) = log u + f ( x ) u is the denoised pixel value u Pascal Getreuer (UCLA) Functional Lifting 3 / 28
Ishikawa’s Method T Labels L Q cut value = E S 2 2 3 0 1 Pixels V Each pixel corresponds to a column of nodes. The horizontal edges encode R and the vertical edges encode ρ . Pascal Getreuer (UCLA) Functional Lifting 4 / 28
Ishikawa’s Method T Labels L Q cut value = ∞ S 2 2 ? 0 1 Pixels V To prevent cutting any column more than once, the red edges are given infinite weight. Pascal Getreuer (UCLA) Functional Lifting 5 / 28
Ishikawa’s Method Problems Memory: many edges, all represented explicitly Parallelization: currently no fast parallel algorithm for graph cuts Metrification (grid bias) artifacts Pascal Getreuer (UCLA) Functional Lifting 6 / 28
Continuous Problem Pock et al. consider the variational problem � � � � u :Ω → Γ E ( u ) = min |∇ u ( x ) | dx + ρ u ( x ) , x dx (2) Ω Ω � � where Γ = [ γ min , γ max ] and ρ u ( x ) , x is any nonnegative function. The authors show how to obtain a global minimizer of this nonconvex problem by reinterpreting Ishikawa’s method. Pascal Getreuer (UCLA) Functional Lifting 7 / 28
Functional Lifting Define the super level sets of u � 1 if u ( x ) > γ , . ϕ ( x , γ ) = 1 1 { u >γ } ( x ) = 0 otherwise. Then u is recovered from ϕ as � γ max u ( x ) = γ min + ϕ ( x , γ ) d γ. γ min For notational convenience, let Σ = Ω × Γ and D ′ = � � ϕ : Σ → { 0 , 1 } | ϕ ( x , γ min ) = 1 , ϕ ( x , γ max ) = 0 Pascal Getreuer (UCLA) Functional Lifting 8 / 28
Functional Lifting Theorem The minimization for u (2) is equivalent to � min |∇ ϕ ( x , γ ) | + ρ ( x , γ ) | ∂ γ ϕ ( x , γ ) | d Σ (3) ϕ ∈ D ′ Σ Proof: By the co-area formula, we have for the TV term � � . |∇ u ( x ) | dx = perimeter(1 1 { u >γ } ) d γ Ω Γ � � = |∇ ϕ | dx d γ. Γ Ω Pascal Getreuer (UCLA) Functional Lifting 9 / 28
Functional Lifting Observe that | ∂ γ ϕ ( x , γ ) | = δ ( u ( x ) − γ ) . So for the fidelity term, � � � ρ u ( x ) , x dx Ω � � = ρ ( γ, x ) δ ( u ( x ) − γ ) d γ dx Ω Γ � � = ρ ( γ, x ) | ∂ γ ϕ ( x , γ ) | d γ dx . Ω Γ Pascal Getreuer (UCLA) Functional Lifting 10 / 28
Functional Lifting Thus we have that the problem in u � � � � u :Ω → Γ E ( u ) = min |∇ u ( x ) | dx + ρ u ( x ) , x dx Ω Ω is equivalent to the lifted problem in ϕ � ϕ ∈ D ′ E ( ϕ ) = min |∇ ϕ ( x , γ ) | + ρ ( x , γ ) | ∂ γ ϕ ( x , γ ) | d Σ . Σ Pascal Getreuer (UCLA) Functional Lifting 11 / 28
Convex Relaxation Still, at this point, the lifted problem � ϕ ∈ D ′ E ( ϕ ) = min |∇ ϕ | + ρ | ∂ γ ϕ | d Σ Σ is nonconvex because D ′ is nonconvex: D ′ = � � ϕ : Σ → { 0 , 1 } | ϕ ( x , γ min ) = 1 , ϕ ( x , γ max ) = 0 . Pascal Getreuer (UCLA) Functional Lifting 12 / 28
Convex Relaxation To make the problem convex, define the relaxed set � � D = ϕ : Σ → [0 , 1] | ϕ ( x , γ min ) = 1 , ϕ ( x , γ max ) = 0 . Then the problem � min ϕ ∈ D E ( ϕ ) = |∇ ϕ | + ρ | ∂ γ ϕ | d Σ (4) Σ is convex. We can find a minimizer ϕ ∗ ∈ D of (4) and then threshold it, . 1 { ϕ ∗ ≥ µ } ∈ D ′ . 1 Pascal Getreuer (UCLA) Functional Lifting 13 / 28
Convex Relaxation Theorem Let ϕ ∗ ∈ D be a minimizer of the relaxed problem (4). Then for a.e. µ ∈ [0 , 1] , the thresholded solution . 1 { ϕ ∗ ≥ µ } ∈ D ′ 1 is a minimizer of the unrelaxed problem (3). Proof: Again using the co-area formula. Pascal Getreuer (UCLA) Functional Lifting 14 / 28
Convex Relaxation (Proof by contradiction) By the co-area formula, � E ( ϕ ) = |∇ ϕ ( x , γ ) | + ρ ( x , γ ) | ∂ γ ϕ ( x , γ ) | d Σ Σ � 1 � . . � + ρ ( x , γ ) � d Σ d µ � � � � = � ∇ 1 1 { ϕ ≥ µ } � ∂ γ 1 1 { ϕ ≥ µ } 0 Σ � 1 . = E (1 1 { ϕ ≥ µ } ) d µ. 0 . Suppose there exists ϕ ′ ∈ D ′ such that E ( ϕ ′ ) < E (1 1 { ϕ ∗ ≥ µ } ) for all µ in a measurable subset of [0 , 1] of nonzero measure, then � 1 � 1 . E ( ϕ ′ ) = E ( ϕ ′ ) d µ < 1 { ϕ ∗ ≥ µ } ) d µ = E ( ϕ ∗ ) . E (1 0 0 But this contradicts that ϕ ∗ is a minimizer of (4). Pascal Getreuer (UCLA) Functional Lifting 15 / 28
Convex Relaxation So, we can find a minimizer ϕ ∗ of the relaxed convex problem � min |∇ ϕ | + ρ | ∂ γ ϕ | d Σ , ϕ ∈ D Σ . then thresholding it 1 1 { ϕ ∗ ≥ µ } gives a minimizer of the unrelaxed problem � min |∇ ϕ | + ρ | ∂ γ ϕ | d Σ . ϕ ∈ D ′ Σ Pascal Getreuer (UCLA) Functional Lifting 16 / 28
Convex Relaxation Then a solution u ∗ is recovered by � γ max . u ∗ = γ min + 1 1 { ϕ ∗ ≥ µ } d γ, γ min and it is a minimizer of the original problem � � � � min |∇ u ( x ) | dx + ρ u ( x ) , x dx . u :Ω → Γ Ω Ω Pascal Getreuer (UCLA) Functional Lifting 17 / 28
Minimization Algorithm Now that we have established a convex formulation of the problem, we wish to solve it. To find the minimizer of � min ϕ ∈ D E ( ϕ ) = |∇ ϕ | + ρ | ∂ γ ϕ | d Σ , Σ one could attempt to solve the associated Euler-Lagrange equations � ∇ ϕ � � ρ ∂ γ ϕ � − div − ∂ γ = 0 , s.t. ϕ ∈ D . |∇ ϕ | | ∂ γ ϕ | But this is hard because of the singularities as |∇ ϕ | or | ∂ γ ϕ | → 0. Pascal Getreuer (UCLA) Functional Lifting 18 / 28
Minimization Algorithm Instead, write E ( ϕ ) in a dual formulation: observe that � p 2 1 + p 2 |∇ ϕ | + ρ | ∂ γ ϕ | = max p { p · ∇ 3 ϕ } , s.t. 2 ≤ 1 , | p 3 | ≤ ρ, where p = ( p 1 , p 2 , p 3 ) is the dual variable and ∇ 3 := ( ∂ x 1 , ∂ x 2 , ∂ γ ) T . This leads to the primal-dual formulation � � � min max p · ∇ 3 ϕ d Σ , (5) ϕ ∈ D p ∈ C Σ p : Σ → R 3 | p 1 ( x , γ ) 2 + p 2 ( x , γ ) 2 ≤ 1 , � � where C = � | p 3 ( x , γ ) | ≤ ρ ( γ, x ) . Pascal Getreuer (UCLA) Functional Lifting 19 / 28
Minimization Algorithm � � � The authors solve min ϕ ∈ D max p ∈ C Σ p · ∇ 3 ϕ d Σ with a primal-dual proximal point method: Primal Step: Solve for ϕ k +1 as the minimizer of � � p k · ∇ 3 ϕ + 1 ( ϕ − ϕ k ) 2 min 2 τ p ϕ ∈ D Σ Σ ϕ k +1 = P D ( ϕ k + τ c div 3 p k ) = ⇒ Dual Step: Solve for p k +1 as the maximizer of � � p · ∇ 3 ϕ k +1 − 1 ( p − p k ) 2 max 2 τ d p ∈ C Σ Σ p k +1 = P C ( p k + τ d ∇ 3 ϕ k +1 ) = ⇒ Pascal Getreuer (UCLA) Functional Lifting 20 / 28
Numerical Results Pock et al. compare their method with Ishikawa’s for color stereo estimation. The left image is Pascal Getreuer (UCLA) Functional Lifting 21 / 28
Numerical Results Ishikawa 4-neighbor Pock et al. Pascal Getreuer (UCLA) Functional Lifting 22 / 28
Numerical Results Ishikawa 8-neighbor Pock et al. Pascal Getreuer (UCLA) Functional Lifting 23 / 28
Numerical Results Ishikawa 16-neighbor Pock et al. Pascal Getreuer (UCLA) Functional Lifting 24 / 28
Numerical Results Method Error (%) Runtime (s) Memory (MB) Ishikawa 4-neighbor 2.90 2.9 450 Ishikawa 8-neighbor 2.63 4.9 630 Ishikawa 16-neighbor 2.71 14.9 1500 Pock et al., CPU 2.57 25 54 Pock et al., GPU 2.57 0.75 54 The authors tested both CPU and a GPU implementations of their method (on a fancy NVidia GeForce GTX 280). Ishikawa’s method is only on CPU as there is currently no parallel algorithm for graph cuts. Pascal Getreuer (UCLA) Functional Lifting 25 / 28
Summary Pock et al. consider nonconvex problems of the form � � � � min |∇ u ( x ) | dx + ρ u ( x ) , x dx . u :Ω → Γ Ω Ω Functional lifting is used to obtain a convex formulation � min |∇ ϕ | + ρ | ∂ γ ϕ | d Σ . ϕ ∈ D Σ The convex formulation is solved by a proximal primal-dual method on � � � min max p · ∇ 3 ϕ d Σ . ϕ ∈ D p ∈ C Σ Pascal Getreuer (UCLA) Functional Lifting 26 / 28
Recommend
More recommend
