iPiano: Inertial Proximal Algorithm for Non-Convex Optimization - PowerPoint PPT Presentation

iPiano: Inertial Proximal Algorithm for Non-Convex Optimization David Stutz June 2, 2016 David Stutz | June 2, 2016 David Stutz | June 2, 2016 0/34 1/34

Table of Contents 1 Problem Related Work 2 Algorithm 3 Convergence 4 Implementation 5 6 Applications Conclusion 7 David Stutz | June 2, 2016 2/34

Problem Problem. Minimize composite function x ∈ R d h ( x ) = min min n ∈ R d ( f ( x ) + g ( x )) (1) where – f : R d → R ∈ C 1 with L -Lipschitz continuous gradient; – g : dom ( g ) ⊂ R d → R ∞ is proper closed convex and lower semicontinuous; – and h coercive and bounded below by −∞ < h min := inf x ∈ R d h ( x ) . Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term to solve Equation (1) iteratively. David Stutz | June 2, 2016 3/34

Problem Problem. Minimize composite function x ∈ R d h ( x ) = min min n ∈ R d ( f ( x ) + g ( x )) (1) where – f : R d → R ∈ C 1 with L -Lipschitz continuous gradient; – g : dom ( g ) ⊂ R d → R ∞ is proper closed convex and lower semicontinuous; – and h coercive and bounded below by −∞ < h min := inf x ∈ R d h ( x ) . Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term to solve Equation (1) iteratively. David Stutz | June 2, 2016 3/34

Problem Problem. Minimize composite function x ∈ R d h ( x ) = min min n ∈ R d ( f ( x ) + g ( x )) (1) where – f : R d → R ∈ C 1 with L -Lipschitz continuous gradient; – g : dom ( g ) ⊂ R d → R ∞ is proper closed convex and lower semicontinuous; – and h coercive and bounded below by −∞ < h min := inf x ∈ R d h ( x ) . Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term to solve Equation (1) iteratively. David Stutz | June 2, 2016 3/34

Problem Problem. Minimize composite function x ∈ R d h ( x ) = min min n ∈ R d ( f ( x ) + g ( x )) (1) where – f : R d → R ∈ C 1 with L -Lipschitz continuous gradient; – g : dom ( g ) ⊂ R d → R ∞ is proper closed convex and lower semicontinuous; – and h coercive and bounded below by −∞ < h min := inf x ∈ R d h ( x ) . Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term to solve Equation (1) iteratively. David Stutz | June 2, 2016 3/34

Problem Problem. Minimize composite function x ∈ R d h ( x ) = min min n ∈ R d ( f ( x ) + g ( x )) (1) where – f : R d → R ∈ C 1 with L -Lipschitz continuous gradient; – g : dom ( g ) ⊂ R d → R ∞ is proper closed convex and lower semicontinuous; – and h coercive and bounded below by −∞ < h min := inf x ∈ R d h ( x ) . Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term to solve Equation (1) iteratively. David Stutz | June 2, 2016 3/34

Table of Contents 1 Problem Related Work 2 Algorithm 3 Convergence 4 Implementation 5 6 Applications Conclusion 7 David Stutz | June 2, 2016 4/34

Related Work Gradient descent for h ∈ C 1 : x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) . Gradient descent with inertial force/momentum term: x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) . Proximal point for h being proper closed convex: x ( n +1) = prox α n h ( x ( n ) ) . Forward-backward splitting for h = f + g with f ∈ C 1 and f , g being proper closed convex: x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) )) . David Stutz | June 2, 2016 5/34

Related Work Gradient descent for h ∈ C 1 : x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) . Gradient descent with inertial force/momentum term: x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) . Proximal point for h being proper closed convex: x ( n +1) = prox α n h ( x ( n ) ) . Forward-backward splitting for h = f + g with f ∈ C 1 and f , g being proper closed convex: x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) )) . David Stutz | June 2, 2016 5/34

Related Work Gradient descent for h ∈ C 1 : x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) . Gradient descent with inertial force/momentum term: x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) . Proximal point for h being proper closed convex: x ( n +1) = prox α n h ( x ( n ) ) . Forward-backward splitting for h = f + g with f ∈ C 1 and f , g being proper closed convex: x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) )) . David Stutz | June 2, 2016 5/34

Related Work Gradient descent for h ∈ C 1 : x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) . Gradient descent with inertial force/momentum term: x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) . Proximal point for h being proper closed convex: x ( n +1) = prox α n h ( x ( n ) ) . Forward-backward splitting for h = f + g with f ∈ C 1 and f , g being proper closed convex: x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) )) . David Stutz | June 2, 2016 5/34

Related Work Gradient descent for h ∈ C 1 : x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) . Gradient descent with inertial force/momentum term: x ( n +1) = x ( n ) − α n ∇ h ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) . Proximal point for h being proper closed convex: x ( n +1) = prox α n h ( x ( n ) ) . Forward-backward splitting for h = f + g with f ∈ C 1 and f , g being proper closed convex: x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) )) . David Stutz | June 2, 2016 5/34

Table of Contents 1 Problem Related Work 2 Algorithm 3 Convergence 4 Implementation 5 6 Applications Conclusion 7 David Stutz | June 2, 2016 6/34

Algorithm – Iterates and Backtracking Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term. x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) )) (2) with step size parameters ( α n ) n ∈ N and momentum parameters ( β n ) n ∈ N . Backtracking to estimate the local Lipschitz constant L n such that f ( x ( n +1) ) ≤ f ( x ( n ) )+ ∇ f ( x ( n ) ) T ( x ( n +1) − x ( n ) ) (3) + L n 2 � x ( n +1) − x ( n ) � 2 2 David Stutz | June 2, 2016 7/34

Algorithm – Iterates and Backtracking Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term: x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) )) (2) with step size parameters ( α n ) n ∈ N and momentum parameters ( β n ) n ∈ N . Backtracking to estimate the local Lipschitz constant L n such that f ( x ( n +1) ) ≤ f ( x ( n ) )+ ∇ f ( x ( n ) ) T ( x ( n +1) − x ( n ) ) (3) + L n 2 � x ( n +1) − x ( n ) � 2 2 David Stutz | June 2, 2016 7/34

Algorithm – Iterates and Backtracking Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) )) (2) with step size parameters ( α n ) n ∈ N and momentum parameters ( β n ) n ∈ N . Backtracking to estimate the local Lipschitz constant L n such that f ( x ( n +1) ) ≤ f ( x ( n ) )+ ∇ f ( x ( n ) ) T ( x ( n +1) − x ( n ) ) (3) + L n 2 � x ( n +1) − x ( n ) � 2 2 David Stutz | June 2, 2016 7/34

Algorithm – Iterates and Backtracking Ochs et al. [OCBP14] combine forward-backward splitting with an inertial force/momentum term x ( n +1) = prox α n g ( x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) )) (2) with step size parameters ( α n ) n ∈ N and momentum parameters ( β n ) n ∈ N . Backtracking to estimate the local Lipschitz constant L n such that f ( x ( n +1) ) ≤ f ( x ( n ) )+ ∇ f ( x ( n ) ) T ( x ( n +1) − x ( n ) ) (3) + L n 2 � x ( n +1) − x ( n ) � 2 2 David Stutz | June 2, 2016 7/34

Algorithm – iPiano Algorithm iPiano. 1: choose c 1 , c 2 > 0 close to zero, L − 1 > 0 , η > 1 , x (0) 2: x ( − 1) := x (0) 3: for n = 1 , . . . do 4: 5: 6: 7: choose α n ≥ c 1 , β n ≥ 0 8: 9: � � x ( n +1) = prox α n g x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) 10: 11: 12: end for David Stutz | June 2, 2016 8/34

Algorithm – iPiano Algorithm iPiano. 1: choose c 1 , c 2 > 0 close to zero, L − 1 > 0 , η > 1 , x (0) 2: x ( − 1) := x (0) 3: for n = 1 , . . . do 4: 5: 6: repeat 7: choose α n ≥ c 1 , β n ≥ 0 8: 2 − β n 2 − β n α n − L n 1 α n − L n 1 until δ n := 2 α n ≥ γ n := α n ≥ c 2 9: � � x ( n +1) = prox α n g x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) 10: 11: 12: end for David Stutz | June 2, 2016 8/34

Algorithm – iPiano Algorithm iPiano. 1: choose c 1 , c 2 > 0 close to zero, L − 1 > 0 , η > 1 , x (0) 2: x ( − 1) := x (0) 3: for n = 1 , . . . do L n := 1 η L n − 1 4: repeat 5: L n := ηL n 6: repeat 7: choose α n ≥ c 1 , β n ≥ 0 8: 2 − β n 2 − β n α n − L n 1 α n − L n 1 until δ n := 2 α n ≥ γ n := α n ≥ c 2 9: x ( n +1) = prox α n g � x ( n ) − α n ∇ f ( x ( n ) ) + β n ( x ( n ) − x ( n − 1) ) � 10: until (3) is satisifed for x ( n +1) 11: 12: end for David Stutz | June 2, 2016 8/34