14. Convex programming Convex sets and functions Convex programs - PowerPoint PPT Presentation

CS/ECE/ISyE 524 Introduction to Optimization Spring 2017–18 14. Convex programming ❼ Convex sets and functions ❼ Convex programs ❼ Hierarchy of complexity ❼ Example: geometric programming Laurent Lessard (www.laurentlessard.com)

Convex sets A set of points C ⊆ R n is convex if for all points x , y ∈ C and any real number 0 ≤ α ≤ 1, we have α x + (1 − α ) y ∈ C . ❼ all points in C can see each other. ❼ can be closed or open (includes y boundary or not), or some combination where only some C x boundary points are included. ❼ can be bounded or unbounded. 14-2

Convex sets Intersections preserve convexity: If I is a collection of convex sets { C i } i ∈I , then the intersection S = � i ∈I C i is convex. proof: Suppose x , y ∈ S and 0 ≤ α ≤ 1. By definition, x , y ∈ C i for each i ∈ I . By the convexity of C i , we must have α x + (1 − α ) y ∈ C i as well. Therefore α x + (1 − α ) y ∈ S , and we are done. note: The union of convex sets C 1 ∪ C 2 is need not be convex! 14-3

Convex sets Constraints can be characterized by sets! ❼ If we define C 1 := { x ∈ R n | Ax ≤ b } then: Ax ≤ b ⇐ ⇒ x ∈ C 1 ❼ If we define C 2 := { x ∈ R n | Fx = g } then: Ax ≤ b and Fx = g ⇐ ⇒ x ∈ C 1 ∩ C 2 14-4

Convex sets Example: SOCP � � Ax + b � ≤ c T x + d x ∈ R n � � � Let C := . To prove C is convex, suppose x , y ∈ C and let z := α x + (1 − α ) y . Then: � Az + b � = � A ( α x + (1 − α ) y ) + b � = � α ( Ax + b ) + (1 − α )( Ay + b ) � ≤ α � Ax + b � + (1 − α ) � Ay + b � ≤ α ( c T x + d ) + (1 − α )( c T y + d ) = c T z + d Therefore, � Az + b � ≤ c T z + d , i.e. C is convex. 14-5

Convex sets Example: spectrahedron � 1 � � � � x 1 x 2 � x ∈ R 3 Let C := � 0 . To prove C is 1 � x 1 x 3 � x 2 x 3 1 � � X = X T � 0 � X ∈ R 3 × 3 � � convex, consider the set S := Note that S is the PSD cone. It is convex because if we define Z := α X + (1 − α ) Y where X , Y ∈ S and 0 ≤ α ≤ 1, then w T Zw = w T ( α X + (1 − α ) Y ) w = α w T Xw + (1 − α ) w T Yw So if X � 0 and Y � 0, then Z � 0. So S is convex. Now, C is convex because it’s the intersection of two convex sets: the PSD cone S and the affine space { X ∈ R 3 × 3 | X ii = 1 } . 14-6

Convex functions ❼ If C ⊆ R n , a function f : C → R is convex if C is a convex set and for all x , y ∈ C and 0 ≤ α ≤ 1, we have: f ( α x + (1 − α ) y ) ≤ α f ( x ) + (1 − α ) f ( y ) ❼ f is concave if − f is convex. f ( x ) 3 y 2 x 1 x - 1 1 2 3 4 14-7

Convex and concave functions Convex functions on R : ❼ Affine: ax + b . ❼ Absolute value: | x | . ❼ Quadratic: ax 2 for any a ≥ 0. ❼ Exponential: a x for any a > 0. ❼ Powers: x α for x > 0, α ≥ 1 or α ≤ 0. ❼ Negative entropy: x log x for x > 0. Concave functions on R : ❼ Affine: ax + b . ❼ Quadratic: ax 2 for any a ≤ 0. ❼ Powers: x α for x > 0, 0 ≤ α ≤ 1. ❼ Logarithm: log x for x > 0. 14-8

Convex and concave functions Convex functions on R n : ❼ Affine: a T x + b . ❼ Norms: � x � 2 , � x � 1 , � x � ∞ ❼ Quadratic form: x T Qx for any Q � 0 14-9

Building convex functions 1. Nonnegative weighted sum: If f ( x ) and g ( x ) are convex and α, β ≥ 0, then α f ( x ) + β g ( x ) is convex. 2. Composition with an affine function: If f ( x ) is convex, so is g ( x ) := f ( Ax + b ) 3. Pointwise maximum: If f 1 ( x ) , . . . , f k ( x ) are convex, then g ( x ) := max { f 1 ( x ) , . . . , f k ( x ) } is convex. proof: Let z := α x + (1 − α ) y as usual. g ( z ) = f ( Az + b ) = f ( α ( Ax + b ) + (1 − α )( Ay + b )) ≤ α f ( Ax + b ) + (1 − α ) f ( Ay + b ) = α g ( x ) + (1 − α ) g ( y ) 14-10

Convex functions vs sets Level set: If f is a convex function, then the set of points satisfying f ( x ) ≤ a is a convex set. ❼ Converse is false: if all level sets of f are convex, it does not necessarily imply that f is a convex function! 14-11

Convex functions vs sets Epigraph: f : R n → R is a convex function if and only if the set { ( x , t ) ∈ R n +1 | f ( x ) ≤ t } is convex. f ( x ) t 4 4 3 3 2 2 1 1 x x - 1 1 2 3 4 - 1 1 2 3 4 plot of f ( x ) { ( x , t ) | f ( x ) ≤ t } 14-12

Convex programs The standard form for a convex optimization problem: minimize f 0 ( x ) x subject to: f i ( x ) ≤ 0 for i = 1 , . . . , k Ax = b x ∈ C ❼ f 0 , f 1 , . . . , f k are convex functions ❼ C is a convex set 14-13

Convex programs ❼ Can turn f 0 ( x ) into a linear constraint (use epigraph) ❼ Can characterize constraints using sets. Minimalist form: c T x minimize x ∈ S ❼ S is a convex set 14-14

Key properties and advantages 1. The set of optimal points x ⋆ is itself a convex set. ◮ Proof : If x ⋆ and y ⋆ are optimal, then we must have f ⋆ = f 0 ( x ⋆ ) = f 0 ( y ⋆ ). Also, f ⋆ ≤ f 0 ( z ) for any z . Choose z := α x ⋆ + (1 − α ) y ⋆ with 0 ≤ α ≤ 1. By convexity of f 0 , f ⋆ ≤ f 0 ( α x ⋆ + (1 − α ) y ⋆ ) ≤ α f 0 ( x ⋆ ) + (1 − α ) f 0 ( y ⋆ ) = f ⋆ . Therefore, f 0 ( z ) = f ⋆ , i.e. z is also an optimal point. 2. If x is a locally optimal point, then it is globally optimal. ◮ Follows from the result above. A very powerful fact! 3. Upper an lower bounds available via duality (more later!) 4. Often numerically tractable (not always!) 14-15

Hierarchy of programs From least general to most general model: 1. LP: linear cost and linear constraints 2. QP: convex quadratic cost and linear constraints 3. QCQP: convex quadratic cost and constraints 4. SOCP: linear cost, second order cone constraints 5. SDP: linear cost, semidefinite constraints 6. CVX: convex cost and constraints Less general (simpler) models are typically preferable 14-16

Solving convex problems Simpler models are usually more efficient to solve Factors affecting solver speed: ❼ How difficult is it to verify that x ∈ C ? ❼ How difficult is it to project onto C ? ❼ How difficult is it to evaluate f ( x ) ? ❼ How difficult is it to compute ∇ f ( x ) ? ❼ Can the solver take advantage of sparsity? 14-17

Example: geometric programming The log-sum-exp function (shown left) is convex: � n � � f ( x ) := log exp x k k =1 It’s a smoothed version of max { x 1 , . . . , x k } (shown right) 14-18

Example: geometric programming Suppose we have positive decision variables x i > 0, and constraints of the form (with each c j > 0 and α jk ∈ R ): α j 1 α j 2 α jn � · · · x ≤ 1 c j x 1 x n 2 j =1 Then by using the substitution y i := log( x i ), we have: � � n � log exp ( a j 0 + a j 1 y 1 + · · · + a jn y n ) ≤ 0 j =1 (where a j 0 := log c j ). This is a log-sum-exp function composed with an affine function (convex!) 14-19

Example: geometric programming Example: We want to design a box of height h , width w , and depth d with maximum volume ( hwd ) subject to the limits: ❼ total wall area: 2( hw + hd ) ≤ A wall ❼ total floor area: wd ≤ A flr ❼ height-width aspect ratio: α ≤ h w ≤ β ❼ width-depth aspect ratio: γ ≤ d w ≤ δ We can make some of the constraints linear, but not all of them. This appears to be a nonconvex optimization problem... 14-20

Example: geometric programming Example: We want to design a box of height h , width w , and depth d with maximum volume ( hwd ) subject to the limits: ❼ total wall area: 2( hw + hd ) ≤ A wall ❼ total floor area: wd ≤ A flr ❼ height-width aspect ratio: α ≤ h w ≤ β ❼ width-depth aspect ratio: γ ≤ d w ≤ δ h − 1 w − 1 d − 1 minimize h , w , d > 0 2 2 1 subject to: A wall hw + A wall hd ≤ 1 , A flr wd ≤ 1 β hw − 1 ≤ 1 α h − 1 w ≤ 1 , 1 γ wd − 1 ≤ 1 , 1 δ w − 1 d ≤ 1 14-21

Example: geometric programming h − 1 w − 1 d − 1 minimize h , w , d > 0 2 2 1 subject to: A wall hw + A wall hd ≤ 1 , A flr wd ≤ 1 β hw − 1 ≤ 1 α h − 1 w ≤ 1 , 1 γ wd − 1 ≤ 1 , δ w − 1 d ≤ 1 1 ❼ Define: x := log h , y := log w , and z := log d . ❼ Express the problem in terms of the new variables x , y , z . Note: h , w , d are positive but x , y , z are unconstrained. 14-22

Example: geometric programming log( e − x − y − z ) minimize x , y , z log( e log(2 / A wall )+ x + y + e log(2 / A wall )+ x + z ) ≤ 0 subject to: log( e log(1 / A flr )+ y + z ) ≤ 0 log( e log α − x + y ) ≤ 0 , log( e − log β + x − y ) ≤ 0 log( e log γ + y − z ) ≤ 0 , log( e − log δ − y + z ) ≤ 0 ❼ this is a convex model, but it can be simplified! ❼ most of the constraints are actually linear. 14-23

Example: geometric programming minimize − x − y − z x , y , z log( e log(2 / A wall )+ x + y + e log(2 / A wall )+ x + z ) ≤ 0 subject to: y + z ≤ log A flr log α ≤ x − y ≤ log β log γ ≤ z − y ≤ log δ ❼ This is a convex optimization problem. 14-24