Convex Optimization 11. Interior-point methods Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2018 SJTU Ying Cui 1 / 42
Outline Inequality constrained minimization problems Logarithmic barrier function and central path Barrier method Feasibility and phase I methods Complexity analysis via self-concordance Problems with generalized inequalities Primal-dual interior-point methods SJTU Ying Cui 2 / 42
Inequality constrained minimization min f 0 ( x ) x s.t. f i ( x ) ≤ 0 , i = 1 , ..., m Ax = b assumptions: ◮ f i : R n → R are convex and twice continuously differentiable ◮ A ∈ R p × n with rank A = p < n ◮ there exists an optimal point x ∗ , i.e., optimal value p ∗ = inf { f ( x ) | Ax = b } is attained and finite ◮ problem is strictly feasible, i.e., there exists x ∈ D with f i ( x ) < 0 , i = 1 , ..., m, Ax = b ◮ means that Slater’s constraint qualification holds hence, strong duality holds and there exist dual optimal λ ∗ ∈ R m and ν ∗ ∈ R p which together with x ∗ satisfy KKT conditions SJTU Ying Cui 3 / 42
Inequality constrained minimization interior-point methods solve convex optimization problems with inequality constraints ◮ barrier method: solve the problem by applying Newton’s method to a sequence of equality constrained problems ◮ primal-dual interior-point method: solve the problem by applying Newton’s method to a sequence of modified versions of KKT conditions SJTU Ying Cui 4 / 42
Examples ◮ LP, QP, QCQP, GP ◮ entropy maximization with linear inequality constraints ( dom f 0 = R n ++ ) n � max x i log x i x i =1 s.t. Fx � g Ax = b ◮ differentiability may require reformulation, e.g., piecewise-linear minimization or ℓ ∞ -norm approximation via LP ◮ SDPs and SOCPs are not in the required form, but can be handled by extensions of interior-point methods to problems with generalized inequalities SJTU Ying Cui 5 / 42
Logarithmic barrier function and central path goal is to approximately formulate the inequality constrained problem as an equality constrained problem to which Newton’s method can be applied SJTU Ying Cui 6 / 42
Logarithmic barrier reformulation via indicator function : m � min f 0 ( x ) + I − ( f i ( x )) x i =1 s.t. Ax = b where I − : R → R is indicator function for R − : � 0 , u ≤ 0 I − ( u ) = ∞ , otherwise ◮ an equality constrained problem ◮ objective function is not (in general) differentiable, so Newton’s method cannot be applied SJTU Ying Cui 7 / 42
Logarithmic barrier approximation via logarithmic barrier : m � min f 0 ( x ) − (1 /t ) log( − f i ( x )) x i =1 s.t. Ax = b where t > 0 is a parameter that sets the accuracy of approximation ◮ − (1 /t ) log( − u ) , t > 0 is a smooth approximation of I − and the approximation becomes more accurate as t increases ◮ an equality constrained problem which is an approximation of the original problem 10 5 0 − 5 − 3 − 2 − 1 0 1 u Figure 11.1 The dashed lines show the function I − ( u ), and the solid curves show � I − ( u ) = − (1 /t ) log( − u ), for t = 0 . 5 , 1 , 2. The curve for t = 2 gives the best approximation. SJTU Ying Cui 8 / 42
Logarithmic barrier logarithmic barrier function : m � φ ( x ) = − log( − f i ( x )) , dom φ = { x | f 1 ( x ) < 0 , ..., f m ( x ) < 0 } i =1 ◮ convex (follows from composition rules) ◮ twice continuously differentiable, with derivatives m 1 � ∇ φ ( x ) = − f i ( x ) ∇ f i ( x ) i =1 m m 1 1 f i ( x ) 2 ∇ f i ( x ) ∇ f i ( x ) T + ∇ 2 φ ( x ) = � � − f i ( x ) ∇ 2 f i ( x ) i =1 i =1 SJTU Ying Cui 9 / 42
Central path equivalent problem of approximation via logarithmic barrier : min tf 0 ( x ) + φ ( x ) x s.t. Ax = b assume existence and uniqueness of solution x ∗ ( t ) for each t > 0 ◮ central path: { x ∗ ( t ) | t > 0 } ◮ central path conditions: x ∗ ( t ) satisfies Ax ∗ ( t ) = b, f i ( x ∗ ( t )) < 0 , i = 1 , · · · , m and there exists a w ∈ R p such that 0 = t ∇ f 0 ( x ∗ ( t )) + ∇ φ ( x ∗ ( t )) + A T w m 1 − f i ( x ∗ ( t )) ∇ f i ( x ∗ ( t )) + A T w � = t ∇ f 0 ( x ∗ ( t )) + i =1 SJTU Ying Cui 10 / 42
Central path example : central path for an LP c T x min x a T s.t. i x ≤ b i , i = 1 , ..., 6 hyperplane c T x = c T x ∗ ( t ) is tangent to level curve of φ through x ∗ ( t ) c x ⋆ (10) x ⋆ Figure 11.2 Central path for an LP with n = 2 and m = 6. The dashed curves show three contour lines of the logarithmic barrier function φ . The central path converges to the optimal point x ⋆ as t → ∞ . Also shown is the point on the central path with t = 10. The optimality condition (11.9) at this point can be verified geometrically: The line c T x = c T x ⋆ (10) is tangent to the contour line of φ through x ⋆ (10). SJTU Ying Cui 11 / 42
Dual points from central path property of central path: every central point yields a dual feasible point, and hence a lower bound on optimal value p ∗ ◮ define λ ∗ i ( t ) = 1 / ( − tf i ( x ∗ ( t )) > 0 and ν ∗ ( t ) = w/t ◮ express optimality condition as i ( t ) ∇ f i ( x ∗ ( t )) + A T ν ∗ ( t ) = 0 , implying ∇ f 0 ( x ∗ ( t )) + � m i =1 λ ∗ x ∗ ( t ) = arg min x L ( x, λ ∗ ( t ) , ν ∗ ( t )) ◮ dual function is g ( λ ∗ ( t ) , ν ∗ ( t )) = L ( x ∗ ( t ) , λ ∗ ( t ) , ν ∗ ( t )) = f 0 ( x ∗ ( t )) − m/t ◮ weak duality g ( λ ∗ ( t ) , ν ∗ ( t )) ≤ p ∗ = ⇒ f 0 ( x ∗ ( t )) − p ∗ ≤ m/t x ∗ ( t ) is no more than m/t -suboptimal and converges to an optimal point as t → ∞ SJTU Ying Cui 12 / 42
Interpretation via KKT conditions interpret central path conditions as a continuous deformation of KKT optimality conditions x = x ∗ ( t ) , λ = λ ∗ ( t ) , ν = ν ∗ ( t ) satisfy ◮ primal constraints: f i ( x ) ≤ 0 , i = 1 , ..., m, Ax = b ◮ dual constraints: λ � 0 ◮ approximate complementary slackness: − λ i f i ( x ) = 1 /t, i = 1 , ..., m ◮ replaces complementary slackness λ i f i ( x ) = 0 in KKT conditions ◮ gradient of Lagrangian with respect to x vanishes: m λ i ∇ f i ( x ) + A T ν = 0 � ∇ f 0 ( x ) + i =1 SJTU Ying Cui 13 / 42
Force field interpretation centering problem (for problem with no equality constraints) m � min tf 0 ( x ) − log( − f i ( x )) x i =1 force field interpretation ◮ tf 0 ( x ) is potential of force field F 0 ( x ) = − t ∇ f 0 ( x ) ◮ − log( − f i ( x )) is potential of force field F i ( x ) = (1 /f i ( x )) ∇ f i ( x ) the forces balance at x ∗ ( t ) : m � F 0 ( x ∗ ( t )) + F i ( x ∗ ( t )) = 0 i =1 SJTU Ying Cui 14 / 42
Force field interpretation example c T x min x a T s.t. i x ≤ b i , i = 1 , ..., m ◮ objective force field is constant: F 0 ( x ) = − tc ◮ constraint force field decays as inverse distance to constraint hyperplane: − a i 1 F i ( x ) = i x, || F i ( x ) || 2 = b i − a T dist ( x, H i ) where H i = { x | a T i x = b i } − c − 3 c Figure 11.3 Force field interpretation of central path . The central path is shown as the dashed curve. The two points x ⋆ (1) and x ⋆ (3) are shown as dots in the left and right plots, respectively. The objective force, which is equal to − c and − 3 c , respectively, is shown as a heavy arrow. The other arrows represent the constraint forces, which are given by an inverse-distance law. As the strength of the objective force varies, the equilibrium position of the particle traces out the central path. SJTU Ying Cui 15 / 42
Barrier method Barrier method . given strictly feasible x, t := t (0) > 0 , µ > 1 , tolerance ǫ > 0 . repeat 1. Centering step . Compute x ∗ ( t ) by minimizing tf 0 + φ , subject to Ax = b , starting at x . 2. Update . x := x ∗ ( t ) . 3. Stopping criterion . quit if m/t < ǫ 4. Increase t . t := µt . ◮ solve a sequence of unconstrained or linearly constrained minimization problems ◮ at each iteration, start from previously computed central point and perform Newton’s method ◮ terminate at ǫ -suboptimal solution ( f 0 ( x ∗ ( t )) − p ∗ ≤ m/t < ǫ ) ◮ choice of µ involves a trade-off: large µ means fewer outer iterations, more inner (Newton) iterations ◮ typical values: µ = 10 - 20 ◮ several heuristics for choice of t (0) SJTU Ying Cui 16 / 42
Variation of barrier method ◮ initialize with a point x not necessarily satisfy Ax = b ◮ an infeasible start Newton method is used for first centering step ◮ a full Newton step is taken at some point during first centering step and thereafter the iterates are all primal feasible ◮ algorithm coincides with standard barrier method from second centering step SJTU Ying Cui 17 / 42
Convergence analysis number of outer (centering) iterations: exactly � � log( m/ ( ǫt (0) )) log µ plus initial centering step to achieve desired accuracy ǫ ◮ duality gap after initial centering step and k additional ones is m/ ( µ k t (0) ) convergence of Newton’s method for centering problem : min tf 0 ( x ) + φ ( x ) x s.t. Ax = b ◮ tf 0 + φ must have closed sublevel sets for t ≥ t (0) ◮ classical analysis requires strong convexity of tf 0 + φ and Lipschitz condition for Hessian of tf 0 + φ ◮ analysis via self-concordance requires self-concordance of tf 0 + φ SJTU Ying Cui 18 / 42
Recommend
More recommend
