Constrained Nonlinear Optimization Moritz Diehl & S ebastien - PowerPoint PPT Presentation

Constrained Nonlinear Optimization Moritz Diehl & S´ ebastien Gros S. Gros, M. Diehl 1 / 12

Outline KKT conditions 1 Some intuitions on the KKT conditions 2 Second Order Sufficient Conditions (SOSC) 3 S. Gros, M. Diehl 2 / 12

Outline KKT conditions 1 Some intuitions on the KKT conditions 2 Second Order Sufficient Conditions (SOSC) 3 S. Gros, M. Diehl 3 / 12

Algebraic Characterization of Unconstrained Local Optima Consider the unconstrained problem: min w Φ ( w ) 1st-Order Necessary Condition of Optimality (FONC) w ∗ local optimum ∇ Φ( w ∗ ) = 0 , w ∗ stationary point ⇒ 2nd-Order Sufficient Conditions of Optimality (SOSC) NLP: x ∗ strict local minimum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≻ 0 ⇒ x ∗ strict local maximum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≺ 0 ⇒ No conclusion can be drawn in the case ∇ 2 Φ( w ∗ ) is indefinite! S. Gros, M. Diehl 4 / 12

Algebraic Characterization of Unconstrained Local Optima Consider the unconstrained problem: min w Φ ( w ) 1st-Order Necessary Condition of Optimality (FONC) w ∗ local optimum ∇ Φ( w ∗ ) = 0 , w ∗ stationary point ⇒ 2nd-Order Sufficient Conditions of Optimality (SOSC) NLP: x ∗ strict local minimum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≻ 0 ⇒ x ∗ strict local maximum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≺ 0 ⇒ No conclusion can be drawn in the case ∇ 2 Φ( w ∗ ) is indefinite! S. Gros, M. Diehl 4 / 12

Algebraic Characterization of Unconstrained Local Optima Consider the unconstrained problem: min w Φ ( w ) 1st-Order Necessary Condition of Optimality (FONC) w ∗ local optimum ∇ Φ( w ∗ ) = 0 , w ∗ stationary point ⇒ 2nd-Order Sufficient Conditions of Optimality (SOSC) NLP: x ∗ strict local minimum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≻ 0 ⇒ x ∗ strict local maximum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≺ 0 ⇒ No conclusion can be drawn in the case ∇ 2 Φ( w ∗ ) is indefinite! Note: ∇ Φ( w ∗ ) = 0 then ∄ d such that ∇ Φ( w ∗ ) T d < 0 ∇ 2 Φ ≻ 0 then ∀ d � = 0, d T ∇ 2 Φ( w ∗ ) d > 0 S. Gros, M. Diehl 4 / 12

Algebraic Characterization of Unconstrained Local Optima Consider the unconstrained problem: min w Φ ( w ) 1st-Order Necessary Condition of Optimality (FONC) w ∗ local optimum ∇ Φ( w ∗ ) = 0 , w ∗ stationary point ⇒ 2nd-Order Sufficient Conditions of Optimality (SOSC) NLP: x ∗ strict local minimum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≻ 0 ⇒ x ∗ strict local maximum ∇ Φ( w ∗ ) = 0 and ∇ 2 Φ( w ∗ ) ≺ 0 ⇒ No conclusion can be drawn in the case ∇ 2 Φ( w ∗ ) is indefinite! Note: ∇ Φ( w ∗ ) = 0 then ∄ d such that ∇ Φ( w ∗ ) T d < 0 ∇ 2 Φ ≻ 0 then ∀ d � = 0, d T ∇ 2 Φ( w ∗ ) d > 0 Local optimum: ”No direction d can improve the cost (locally)” S. Gros, M. Diehl 4 / 12

FONC for equality constraints Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 S. Gros, M. Diehl 5 / 12

FONC for equality constraints Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 Definition: a point w satisfies LICQ a iff ∇ g ( w ) is full column rank a Linear Independence Constraint Qualification S. Gros, M. Diehl 5 / 12

FONC for equality constraints Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 Definition: a point w satisfies LICQ a iff ∇ g ( w ) is full column rank a Linear Independence Constraint Qualification First-order Necessary Conditions Let Φ , g in C 1 . If w ∗ is a (local) optimum, and w ∗ satisfies LICQ, then there is a unique vector λ such that: ∇ Φ( w ∗ ) + ∇ g ( w ∗ ) λ Dual feasibility: = 0 g ( w ∗ ) Primal feasibility: = 0 S. Gros, M. Diehl 5 / 12

FONC for equality constraints Consider the NLP problem: Square system: ( n + m ) conditions in min Φ ( w ) ( n + m ) variables ( w , λ ) w s.t. g ( w ) = 0 Lagrange multipliers: λ i ↔ g i Dual feasibility ≡ Lagrangian stationarity: ∇L ( w ∗ , λ ∗ ) = 0 Definition: a point w satisfies LICQ a iff ∇ g ( w ) is full column rank ∆ = Φ( w ) + λ T g ( w ) is the where L ( w , λ ) Lagrangian a Linear Independence Constraint Qualification First-order Necessary Conditions Let Φ , g in C 1 . If w ∗ is a (local) optimum, and w ∗ satisfies LICQ, then there is a unique vector λ such that: ∇ Φ( w ∗ ) + ∇ g ( w ∗ ) λ Dual feasibility: = 0 g ( w ∗ ) Primal feasibility: = 0 S. Gros, M. Diehl 5 / 12

KKT point Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 S. Gros, M. Diehl 6 / 12

KKT point Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 A point ( w ∗ , µ ∗ , λ ∗ ) is called a KKT point if it satisfies: where L = Φ ( w ) + λ T g ( w ) + µ T h ( w ) S. Gros, M. Diehl 6 / 12

KKT point Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 A point ( w ∗ , µ ∗ , λ ∗ ) is called a KKT point if it satisfies: ∇ w L ( w ∗ , µ ∗ , λ ∗ ) = 0 , µ ∗ ≥ 0 , Dual Feasibility: where L = Φ ( w ) + λ T g ( w ) + µ T h ( w ) S. Gros, M. Diehl 6 / 12

KKT point Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 A point ( w ∗ , µ ∗ , λ ∗ ) is called a KKT point if it satisfies: ∇ w L ( w ∗ , µ ∗ , λ ∗ ) = 0 , µ ∗ ≥ 0 , Dual Feasibility: g ( w ∗ ) = 0 , h ( w ∗ ) ≤ 0 , Primal Feasibility: where L = Φ ( w ) + λ T g ( w ) + µ T h ( w ) S. Gros, M. Diehl 6 / 12

KKT point Consider the NLP problem: min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 A point ( w ∗ , µ ∗ , λ ∗ ) is called a KKT point if it satisfies: ∇ w L ( w ∗ , µ ∗ , λ ∗ ) = 0 , µ ∗ ≥ 0 , Dual Feasibility: g ( w ∗ ) = 0 , h ( w ∗ ) ≤ 0 , Primal Feasibility: µ ∗ i h i ( w ∗ ) = 0 , Complementary Slackness: ∀ i where L = Φ ( w ) + λ T g ( w ) + µ T h ( w ) S. Gros, M. Diehl 6 / 12

First-Order Necessary Conditions (FONC) min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 First-Order Necessary Conditions Let Φ , g , h be C 1 . If w ∗ is a (local) optimum and satisfies LICQ, then there is a unique vector λ ∗ and µ ∗ such that ( w ∗ , λ ∗ , ν ∗ ) is a KKT point. S. Gros, M. Diehl 7 / 12

First-Order Necessary Conditions (FONC) min Φ ( w ) w s.t. g ( w ) = 0 h ( w ) ≤ 0 First-Order Necessary Conditions Let Φ , g , h be C 1 . If w ∗ is a (local) optimum and satisfies LICQ, then there is a unique vector λ ∗ and µ ∗ such that ( w ∗ , λ ∗ , ν ∗ ) is a KKT point. Active constraints: h i ( w ) < 0 then µ ∗ i = 0, and h i is inactive µ ∗ i > 0 and h i ( w ) = 0 then h i ( w ) is strictly active µ ∗ i = 0 and h i ( w ) = 0 then then h i ( w ) is weakly active We define the active set A ∗ as the set of indices i of the active constraints S. Gros, M. Diehl 7 / 12

First-Order Necessary Conditions (FONC) Definition: a point w satisfies LICQ iff min Φ ( w ) w ∇ h A ∗ ( w )] [ ∇ g ( w ) , s.t. g ( w ) = 0 h ( w ) ≤ 0 is full column rank First-Order Necessary Conditions Let Φ , g , h be C 1 . If w ∗ is a (local) optimum and satisfies LICQ, then there is a unique vector λ ∗ and µ ∗ such that ( w ∗ , λ ∗ , ν ∗ ) is a KKT point. Active constraints: h i ( w ) < 0 then µ ∗ i = 0, and h i is inactive µ ∗ i > 0 and h i ( w ) = 0 then h i ( w ) is strictly active µ ∗ i = 0 and h i ( w ) = 0 then then h i ( w ) is weakly active We define the active set A ∗ as the set of indices i of the active constraints S. Gros, M. Diehl 7 / 12

Outline KKT conditions 1 Some intuitions on the KKT conditions 2 Second Order Sufficient Conditions (SOSC) 3 S. Gros, M. Diehl 8 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence S. Gros, M. Diehl 9 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence w 2 w 1 S. Gros, M. Diehl 9 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence w 2 w 1 S. Gros, M. Diehl 9 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence w 2 w 1 S. Gros, M. Diehl 9 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence −∇ Φ is the gravity w 2 −∇ Φ ( w ) w 1 S. Gros, M. Diehl 9 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 − µ ∇ h ( w ) h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence −∇ Φ is the gravity w 2 − µ ∇ h is the force of the fence. Sign −∇ Φ ( w ) µ ≥ 0 means the fence can only ”push” the ball. µ = 0 . 77376 w 1 S. Gros, M. Diehl 9 / 12

Some intuitions on the KKT conditions min Φ( x ) w s.t. h ( w ) ≤ 0 − µ ∇ h ( w ) h ( w ) ≤ 0 Ball rolling down a valley blocked by a fence −∇ Φ is the gravity w 2 − µ ∇ h is the force of the fence. Sign −∇ Φ ( w ) µ ≥ 0 means the fence can only ”push” the ball. µ = 0 . 77376 w 1 Balance of the forces: ∇L = ∇ Φ ( w ) + µ ∇ h ( w ) = 0 S. Gros, M. Diehl 9 / 12