Complexity of a quadratic penalty accelerated inexact proximal point - PowerPoint PPT Presentation

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Complexity of a quadratic penalty accelerated inexact proximal point method W. Kong 1 J.G. Melo 2 R.D.C. Monteiro 1 1 School of Industrial and SystemsEngineering Georgia Institute of Technology 2 Institute of Mathematics and Statistics Federal University of Goias ICERM 2019 - April 30th , Providence

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty The Main Problem 1 The Penalty Approach 2 AIPP Method For Solving the Penalty Subproblem(s) 3 Special Structure of Penalty Subproblem Previous Works AIPP = Inexact Proximal Point + Acceleration AIPP Method and its Complexity Complexity of the Penalty AIPP 4 Computational Results 5 Additional Results and Concluding Remarks 6

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty The main problem: φ ∗ : = min { φ ( z ) : = f ( z ) + h ( z ) : Az = b , z ∈ R n } ( P ) where A : R n → R l is linear and b ∈ R l h : R n → ( − ∞ , ∞ ] closed proper convex with bounded domain; f is differentiable (not necessarily convex) on dom h and, for some L f > 0, ∀ z , z ′ ∈ dom h �∇ f ( z ) − ∇ f ( z ′ ) � ≤ L f � z − z ′ � ,

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty The main problem (continued): φ ∗ : = min { φ ( z ) : = f ( z ) + h ( z ) : Az = b , z ∈ R n } ( P ) Our goal: Given ( ¯ ρ , ¯ η ) > 0, find a ( ¯ ρ , ¯ η ) -approximate solution of ( P ) , i.e., a triple ( ¯ v ) such that z , ¯ w ; ¯ z ) + A ∗ ¯ v ∈ ∇ f ( ¯ ¯ z ) + ∂ h ( ¯ w , � ¯ v � ≤ ¯ ρ , � A ¯ z − b � ≤ ¯ η It will be achieved via a penalty approach.

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty For c > 0, consider φ ∗ ( P c ) c : = min φ c ( z ) : = f c ( z ) + h ( z ) z where f c ( z ) : = f ( z ) + c 2 � Az − b � 2 Quadratic Penalty Approach: 0. choose initial c > 0 1. obtain a ¯ ρ -approximate solution ( ¯ z ; ¯ v ) of ( P c ) , i.e., satisfying v ∈ ∇ f c ( ¯ z ) + ∂ h ( ¯ z ) , � ¯ v � ≤ ¯ ¯ ρ 2. if � A ¯ z − b � ≤ ¯ η then stop and output ¯ z ; otherwise, set c ← 2 c and go to step 1

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty For c > 0, consider φ ∗ ( P c ) c : = min φ c ( z ) : = f c ( z ) + h ( z ) z where f c ( z ) : = f ( z ) + c 2 � Az − b � 2 Quadratic Penalty Approach: 0. choose initial c > 0 1. obtain a ¯ ρ -approximate solution ( ¯ z ; ¯ v ) of ( P c ) , i.e., satisfying v ∈ ∇ f c ( ¯ z ) + ∂ h ( ¯ z ) , � ¯ v � ≤ ¯ ¯ ρ 2. if � A ¯ z − b � ≤ ¯ η then stop and output ¯ z ; otherwise, set c ← 2 c and go to step 1

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Theorem Let ( ¯ η ) > 0 be given. Assume that ( ¯ v ) is a ¯ ρ , ¯ z ; ¯ ρ -approximate solution of ( P c ) and define R : = 2 ∆ ∗ ρ D h + L f D 2 w : = c ( A ¯ z − b ) , φ + 2 ¯ ¯ h where D h : = sup {� z − z ′ � : z , z ′ ∈ dom h } , φ : = φ ∗ − φ ∗ , ∆ ∗ z { ( f + h )( z ) : z ∈ R n } φ ∗ : = inf Then, ( ¯ z , ¯ w ; ¯ v ) is ( ¯ ρ , ¯ η ) -approximate solution of ( P ) whenever c ≥ R η 2 ¯

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Special Structure of Penalty Subproblem The Main Problem 1 The Penalty Approach 2 AIPP Method For Solving the Penalty Subproblem(s) 3 Special Structure of Penalty Subproblem Previous Works AIPP = Inexact Proximal Point + Acceleration AIPP Method and its Complexity Complexity of the Penalty AIPP 4 Computational Results 5 Additional Results and Concluding Remarks 6

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Special Structure of Penalty Subproblem Recall that the objective function of ( P c ) is φ c = f c + h where f c ( z ) : = f ( z ) + c � Az − b � 2 / 2 For every z , z ′ ∈ dom h , − m ≤ f c ( z ′ ) − [ f c ( z ) + �∇ f c ( z ) , z ′ − z � ] ≤ M c � z ′ − z � 2 / 2 where M c : = L f + c � A � 2 m : = L f , The complexity of the composite gradient meth for solving ( P c ) is mD 2 � � h O M c ρ 2 ¯ which is high for large c , or when M c >> m .

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Special Structure of Penalty Subproblem Recall that the objective function of ( P c ) is φ c = f c + h where f c ( z ) : = f ( z ) + c � Az − b � 2 / 2 For every z , z ′ ∈ dom h , − m ≤ f c ( z ′ ) − [ f c ( z ) + �∇ f c ( z ) , z ′ − z � ] ≤ M c � z ′ − z � 2 / 2 where M c : = L f + c � A � 2 m : = L f , The complexity of the composite gradient meth for solving ( P c ) is mD 2 � � h O M c ρ 2 ¯ which is high for large c , or when M c >> m .

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Previous Works The Main Problem 1 The Penalty Approach 2 AIPP Method For Solving the Penalty Subproblem(s) 3 Special Structure of Penalty Subproblem Previous Works AIPP = Inexact Proximal Point + Acceleration AIPP Method and its Complexity Complexity of the Penalty AIPP 4 Computational Results 5 Additional Results and Concluding Remarks 6

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty Previous Works S. Ghadimi and G. Lan "Accelerated gradient methods for nonconvex nonlinear and stochastic programming", published 2016 Complexity: � � 2 / 3 � M c mD 2 � M c d 0 h O + ρ 2 ¯ ρ ¯ The dominant term (i.e., the blue one) is O ( M c ) . Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford "Accelerated methods for non-convex optimization", arXiv 2017 obtained a O ( √ M c log M c ) complexity bound under the assumption that h = 0. Our AIPP approach removes the log M c from the above bound and the assumption that h = 0

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty AIPP = Inexact Proximal Point + Acceleration The Main Problem 1 The Penalty Approach 2 AIPP Method For Solving the Penalty Subproblem(s) 3 Special Structure of Penalty Subproblem Previous Works AIPP = Inexact Proximal Point + Acceleration AIPP Method and its Complexity Complexity of the Penalty AIPP 4 Computational Results 5 Additional Results and Concluding Remarks 6

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty AIPP = Inexact Proximal Point + Acceleration AIPP for solving ( P c ) is based on an IPP scheme whose k -th iteration is as follows. Given z k − 1 , it chooses λ k > 0 and approximately solves the ‘prox’ subproblem � � λ k ( f c + h )( z ) + 1 ( P k 2 � z − z k − 1 � 2 c ) min i.e., for some σ ∈ ( 0 , 1 ) , it computes a point z k and a residual pair ( v k , ε k ) ∈ R n × R + such that � � λ k ( f c + h ) + 1 2 � · − z k − 1 � 2 v k ∈ ∂ ε k ( z k ) � v k � 2 + 2 ε k ≤ σ � z k − 1 − z k + v k � 2

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty AIPP = Inexact Proximal Point + Acceleration AIPP for solving ( P c ) is based on an IPP scheme whose k -th iteration is as follows. Given z k − 1 , it chooses λ k > 0 and approximately solves the ‘prox’ subproblem � � λ k ( f c + h )( z ) + 1 ( P k 2 � z − z k − 1 � 2 c ) min i.e., for some σ ∈ ( 0 , 1 ) , it computes a point z k and a residual pair ( v k , ε k ) ∈ R n × R + such that � � λ k ( f c + h ) + 1 2 � · − z k − 1 � 2 v k ∈ ∂ ε k ( z k ) � v k � 2 + 2 ε k ≤ σ � z k − 1 − z k + v k � 2

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty AIPP = Inexact Proximal Point + Acceleration AIPP method: It is an accelerated instance of the above IPP scheme in which for all k : λ k = 1 / ( 2 m ) , and hence ( P k c ) is a strongly convex problem z k and ( v k , ε k ) are computed by an accelerated composite gradient (ACG) method applied to ( P k c ) in at most ��� �� M c O iterations m Obs: Each ACG iteration requires one or two evaluations of the resolvent of h , i.e., exact solution of min { a T z + h ( z ) + θ � z � 2 }

The Main Problem Penalty Problem and Approach AIPP Method For Solving the Penalty Subproblem(s) Complexity of the Penalty AIPP Method and its Complexity The Main Problem 1 The Penalty Approach 2 AIPP Method For Solving the Penalty Subproblem(s) 3 Special Structure of Penalty Subproblem Previous Works AIPP = Inexact Proximal Point + Acceleration AIPP Method and its Complexity Complexity of the Penalty AIPP 4 Computational Results 5 Additional Results and Concluding Remarks 6