15. Duality Upper and lower bounds General duality Constraint - PowerPoint PPT Presentation

CS/ECE/ISyE 524 Introduction to Optimization Spring 2017–18 15. Duality ❼ Upper and lower bounds ❼ General duality ❼ Constraint qualifications ❼ Counterexample ❼ Complementary slackness ❼ Examples ❼ Sensitivity analysis Laurent Lessard (www.laurentlessard.com)

Upper bounds Optimization problem (not necessarily convex!): minimize f 0 ( x ) x ∈ D subject to: f i ( x ) ≤ 0 for i = 1 , . . . , m h j ( x ) = 0 for j = 1 , . . . , r ❼ D is the domain of all functions involved. ❼ Suppose the optimal value is p ⋆ . ❼ Upper bounds: if x ∈ D satisfies f i ( x ) ≤ 0 and h j ( x ) = 0 for all i and j , then: p ⋆ ≤ f 0 ( x ). ❼ Any feasible x yields an upper bound for p ⋆ . 15-2

Lower bounds Optimization problem (not necessarily convex!): minimize f 0 ( x ) x ∈ D subject to: f i ( x ) ≤ 0 for i = 1 , . . . , m h j ( x ) = 0 for j = 1 , . . . , r ❼ As with LPs, use the constraints to find lower bounds ❼ For any λ i ≥ 0 and ν j ∈ R , if x ∈ D is feasible, then m r � � f 0 ( x ) ≥ f 0 ( x ) + λ i f i ( x ) + ν j h j ( x ) i =1 j =1 15-3

Lower bounds m r � � f 0 ( x ) ≥ f 0 ( x ) + λ i f i ( x ) + ν j h j ( x ) i =1 j =1 � �� � Lagrangian L ( x , λ, ν ) This is a lower bound on f 0 , but we want a lower bound on p ⋆ . Minimize right side over x ∈ D and left side over feasible x . � � p ⋆ ≥ x ∈ D L ( x , λ, ν ) inf = g ( λ, ν ) This inequality holds whenever λ ≥ 0. 15-4

Lower bounds m r � � L ( x , λ, ν ) := f 0 ( x ) + λ i f i ( x ) + ν j h j ( x ) i =1 j =1 Whenever λ ≥ 0, we have: � � ≤ p ⋆ g ( λ, ν ) := x ∈ D L ( x , λ, ν ) inf Useful fact: g ( λ, ν ) is a concave function. This is true even if the original optimization problem is not convex! (because g is a pointwise minimum of affine functions) 15-5

General duality Primal problem (P) Dual problem (D) minimize f 0 ( x ) maximize g ( λ, ν ) x ∈ D λ,ν subject to: f i ( x ) ≤ 0 ∀ i subject to: λ ≥ 0 h j ( x ) = 0 ∀ j If x and λ are feasible points of (P) and (D) respectively: g ( λ, ν ) ≤ d ⋆ ≤ p ⋆ ≤ f 0 ( x ) This is called the Lagrange dual . Bad news: strong duality ( p ⋆ = d ⋆ ) does not always hold! 15-6

Example (Srikant) x 2 + 1 minimize x subject to: ( x − 2)( x − 4) ≤ 0 30 25 20 x 2 + 1 15 10 ( x - 2 ) ( x - 4 ) 5 5 x - 1 1 2 3 4 - 5 ❼ optimum occurs at x = 2, has value p ⋆ = 5 15-7

Example (Srikant) L ( x , λ ) = x 2 + 1 + λ ( x − 2)( x − 4) Lagrangian: 30 25 20 15 10 5 x - 1 1 2 3 4 5 - 5 ❼ Plot for different values of λ ≥ 0 ❼ g ( λ ) = inf x L ( x , λ ) should be a lower bound on p ⋆ = 5 for all λ ≥ 0. 15-8

Example (Srikant) L ( x , λ ) = x 2 + 1 + λ ( x − 2)( x − 4) Lagrangian: ❼ Minimize the Lagrangian: g ( λ ) = inf L ( x , λ ) x x ( λ + 1) x 2 − 6 λ x + (8 λ + 1) = inf If λ ≤ − 1, it is unbounded. If λ > − 1, the minimum 3 λ occurs when 2( λ + 1) x − 6 λ = 0, so ˆ x = λ +1 . � − 9 λ 2 / (1 + λ ) + 1 + 8 λ λ > − 1 g ( λ ) = −∞ λ ≤ − 1 15-9

Example (Srikant) − 9 λ 2 / (1 + λ ) + 1 + 8 λ maximize λ subject to: λ ≥ 0 g ( λ ) 5 λ - 1 1 2 3 4 5 - 5 - 10 ❼ optimum occurs at λ = 2, has value d ⋆ = 5 ❼ same optimal value as primal problem! (strong duality) 15-10

Constraint qualifications ❼ weak duality ( d ⋆ ≤ p ⋆ ) always holds. Even when the optimization problem is not convex. ❼ strong duality ( d ⋆ = p ⋆ ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality. An example we’ve already seen: ❼ If the optimization problem is an LP, strong duality holds 15-11

Slater’s constraint qualification minimize f 0 ( x ) x ∈ D subject to: f i ( x ) ≤ 0 for i = 1 , . . . , m h j ( x ) = 0 for j = 1 , . . . , r Slater’s constraint qualification: If the optimization problem is convex and strictly feasible, then strong duality holds. ❼ convexity requires: D and f i are convex and h j are affine. ❼ strict feasibility means there exists some ˜ x in the interior of D such that f i (˜ x ) < 0 for i = 1 , . . . , m . 15-12

Slater’s constraint qualification If the optimization problem is convex and strictly feasible, then strong duality holds. ❼ Good news: Slater’s constraint qualification is rather weak. i.e. it is usually satisfied by convex problems. ❼ Can be relaxed so that strict feasibility is not required for the linear constraints. 15-13

Counterexample (Boyd) e − x minimize x ∈ R , y > 0 x 2 / y ≤ 0 subject to: ❼ The function x 2 / y is convex for y > 0 (see plot) ❼ The objective e − x is convex ❼ Feasible set: { (0 , y ) | y > 0 } ❼ Solution is trivial ( p ⋆ = 1) 15-14

Counterexample (Boyd) e − x minimize x ∈ R , y > 0 x 2 / y ≤ 0 subject to: ❼ Lagrangian: L ( x , y , λ ) = e − x + λ x 2 / y ❼ Dual function: g ( λ ) = inf x , y > 0 ( e − x + λ x 2 / y ) = 0. ❼ The dual problem is: maximize 0 λ ≥ 0 So we have d ⋆ = 0 < 1 = p ⋆ . ❼ Slater’s constraint qualification is not satisfied! 15-15

About Slater’s constraint qualification Slater’s condition is only sufficient . (Slater) = ⇒ (strong duality) ❼ There exist problems where Slater’s condition fails, yet strong duality holds. ❼ There exist nonconvex problems with strong duality. 15-16

Complementary slackness Assume strong duality holds. If x ⋆ is primal optimal and ( λ ⋆ , ν ⋆ ) is dual optimal, then we have: g ( λ ⋆ , ν ⋆ ) = d ⋆ = p ⋆ = f 0 ( x ⋆ ) � � m r � � f 0 ( x ⋆ ) = g ( λ ⋆ , ν ⋆ ) = inf λ ⋆ ν ⋆ f 0 ( x ) + i f i ( x ) + j h j ( x ) x ∈ D i =1 j =1 m r � � ≤ f 0 ( x ⋆ ) + λ ⋆ i f i ( x ⋆ ) + ν ⋆ j h j ( x ⋆ ) i =1 j =1 ≤ f 0 ( x ⋆ ) The last inequality holds because x ⋆ is primal feasible. We conclude that the inequalities must all be equalities. 15-17

Complementary slackness ❼ We concluded that: m r � � f 0 ( x ⋆ ) = f 0 ( x ⋆ ) + λ ⋆ i f i ( x ⋆ ) + ν ⋆ j h j ( x ⋆ ) i =1 j =1 But f i ( x ⋆ ) ≤ 0 and h j ( x ⋆ ) = 0. Therefore: λ ⋆ i f i ( x ⋆ ) = 0 for i = 1 , . . . , m ❼ This property is called complementary slackness . We’ve seen it before for linear programs. λ ⋆ ⇒ f i ( x ⋆ ) = 0 f i ( x ⋆ ) < 0 = ⇒ λ ⋆ i > 0 = and i = 0 15-18

Dual of an LP c T x minimize x ≥ 0 subject to: Ax ≥ b ❼ Lagrangian: L ( x , λ ) = c T x + λ T ( b − Ax ) x ≥ 0 ( c − A T λ ) T x + λ T b ❼ Dual function: g ( λ ) = min � λ T b if A T λ ≤ c g ( λ ) = −∞ otherwise 15-19

Dual of an LP c T x minimize x ≥ 0 subject to: Ax ≥ b ❼ Dual is: λ T b maximize λ ≥ 0 A T λ ≤ c subject to: ❼ This is the same result that we found when we were studying duality for linear programs. 15-20

Dual of an LP What if we treat x ≥ 0 as a constraint instead? ( D = R n ). c T x minimize x subject to: Ax ≥ b x ≥ 0 ❼ Lagrangian: L ( x , λ, µ ) = c T x + λ T ( b − Ax ) − µ T x ❼ Dual function: g ( λ, µ ) = min x ( c − A T λ − µ ) T x + λ T b � λ T b if A T λ + µ = c g ( λ ) = −∞ otherwise 15-21

Dual of an LP What if we treat x ≥ 0 as a constraint instead? ( D = R n ). c T x minimize x subject to: Ax ≥ b x ≥ 0 ❼ Dual is: λ T b maximize λ ≥ 0 , µ ≥ 0 A T λ + µ = c subject to: ❼ Solution is the same, µ acts as the slack variable. 15-22

Dual of a convex QP Suppose Q ≻ 0. Let’s find the dual of the QP: 1 2 x T Qx minimize x subject to: Ax ≥ b ❼ Lagrangian: L ( x , λ ) = 1 2 x T Qx + λ T ( b − Ax ) � 1 � 2 x T Qx + λ T ( b − Ax ) ❼ Dual function: g ( λ ) = min x x = Q − 1 A T λ Minimum occurs at: ˆ g ( λ ) = − 1 2 λ T AQ − 1 A T λ + λ T b 15-23

Dual of a convex QP Suppose Q ≻ 0. Let’s find the dual of the QP: 1 2 x T Qx minimize x subject to: Ax ≥ b ❼ Dual is also a QP: − 1 2 λ T AQ − 1 A T λ + λ T b maximize λ subject to: λ ≥ 0 ❼ It’s still easy to solve (maximizing a concave function) 15-24

Sensitivity analysis min f 0 ( x ) g ( λ, ν ) − λ T u − ν T v max x ∈ D λ,ν s.t. f i ( x ) ≤ u i ∀ i s.t. λ ≥ 0 h j ( x ) = v j ∀ j ❼ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ❼ A change in u i causes a bigger change in p ⋆ if λ ⋆ i is larger. ❼ A change in v j causes a bigger change in p ⋆ if ν ⋆ j is larger. ❼ If p ⋆ ( u , v ) is differentiable, then: i = − ∂ p ⋆ (0 , 0) j = − ∂ p ⋆ (0 , 0) λ ⋆ ν ⋆ and ∂ u i ∂ v j 15-25