Primary objectives: Convex optimization Ellipsoid method A - PowerPoint PPT Presentation

Primary objectives: � Convex optimization � Ellipsoid method � A polynomial algorithm for linear programming 141

P ART 6 C ONVEX OPTIMIZATION 142

Reminder: Convex functions Convex function f : R n − → R is convex function, if domain of f is convex and for each x , y ∈ dom ( f ) and 0 � λ � 1 one has f ( λ x + (1 − λ ) y ) � λ f ( x ) + (1 − λ ) f ( y ) ( y, f ( y )) ( x, f ( x )) Example �·� (any norm) is a convex function, since � α · x � = | α |·� x � and � x + y � � � x �+� y � . Thus � λ x + (1 − λ ) y � � λ � x �+ (1 − λ ) � y � . 143

Sublevel sets Definition C α f : R n − → R convex and α ∈ R , C α = { x ∈ R n : f ( x ) � α } is α -sublevel set of f . Lemma 6.1 If f is convex, then C α is a convex set for each α ∈ R . Epigraph f : R n − → R convex, epi ( f ) = {( x , t ): x ∈ dom ( f ), f ( x ) � t } is epigraph of f . Lemma 6.2 f is convex if and only if epi ( f ) is convex set. 144

Convex optimization problem Convex optimization problem A convex optimization problem is of the form minimize f 0 ( x ) subject to f i ( x ) � b i for i = 1,..., m , where f i , i = 0,..., m are convex functions. Example: Quadratic programming Q ∈ R n × n positive semidefinite, c ∈ R n A ∈ R m × n and b ∈ R m . Convex quadratic program min x T Qx + c T x Ax = b x 0, � is convex optimization problem. 145

Binary search for minimum � Suppose we can efficiently test whether convex set is empty or not � Search smallest β ∈ R such that convex set C β = { x ∈ R n : f 0 ( x ) � β , f 1 ( x ) � b 1 ,..., f m ( x ) � b m } is non-empty. � Keep upper bound U and lower bound L � Test : Whether C ( L + U )/2 = � . If yes, then L : = ( L + U )/2. If no, then U : = ( L + U )/2. � After O (log(( U − L )/ ε ) tests, one obtains a value of distance � ε from the optimum value. 146

Separating hyperplane Theorem 6.3 If S ⊆ R n is closed and convex and x ∗ ∉ S, then there exists a hyperplane c T x = δ such that c T s < δ for each s ∈ S and c T x ∗ > δ . x ∗ 147

Balls and ellipsoids The unit ball is the set B = { x ∈ R n | � x � � 1}. An ellipsoid E ( A , b ) is the image of the unit ball under an affine map t : R n → R n with t ( x ) = Ax + b , where A ∈ R n × n is an invertible matrix and b ∈ R n is a vector. Clearly E ( A , b ) = { x ∈ R n | � A − 1 x − A − 1 b � � 1}. (13) Exercise �� x (1) � 1 3 � Consider the mapping t ( x ) = . Draw the ellipsoid which is 2 5 x (2) defined by t. What are the axes of the ellipsoid? Volume of unit ball � 2 e π � n /2 . 1 The volume of the unit ball is V n ∼ π n n Volume of ellipsoid E ( A , b ) is equal to | det( A ) |· V n . 148

Lemma 6.4 (Half-Ball Lemma) The half-ball H = { x ∈ R n | � x � � 1, x (1) � 0} is contained in the ellipsoid + n 2 − 1 � � 2 � � 2 � n � n + 1 1 x ∈ R n | x ( i ) 2 � 1 � E = x (1) − (14) n 2 n n + 1 i = 2 149 x (1) � 0

Proof Let x be contained in the unit ball, i.e., � x � � 1 and suppose further that 0 � x (1) holds. We need to show that + n 2 − 1 � 2 � � 2 n � n + 1 1 x ( i ) 2 � 1 � x (1) − (15) n 2 n n + 1 i = 2 i = 2 x ( i ) 2 � 1 − x (1) 2 holds we have holds. Since � n + n 2 − 1 � 2 � � 2 n � n + 1 1 x ( i ) 2 � x (1) − n 2 n n + 1 i = 2 + n 2 − 1 � 2 � � 2 � n + 1 1 (1 − x (1) 2 ) x (1) − � n 2 n n + 1 (16) This shows that (15) holds if x is contained in the half-ball and x (1) = 0 or x (1) = 1. 150

Proof cont. Now consider the right-hand-side of (16) as a function of x (1), i.e., consider + n 2 − 1 � 2 � � 2 � n + 1 1 (1 − x (1) 2 ). f ( x (1)) = x (1) − (17) n 2 n n + 1 The first derivative is − 2 · n 2 − 1 � 2 � � n + 1 1 � f ′ ( x (1)) = 2 · x (1) − x (1). (18) n 2 n n + 1 We have f ′ (0) < 0 and since both f (0) = 1 and f (1) = 1, we have f ( x (1)) � 1 for all 0 � x (1) � 1 and the assertion follows. 151

Corollary 6.5 The half-ball { x ∈ R n | x (1) � 0, � x � � 1} is contained in an ellipsoid E, 1 whose volume is bounded by e − 2( n + 1) · V n . Ellipsoids: Convenient notation An ellipsoid E ( A , a ) is the set E ( A , a ) = { x ∈ R n | ( x − a ) T A − 1 ( x − a ) � 1}, where A ∈ R n × n is a symmetric positive definite matrix and a ∈ R n is a vector. Half-ellipsoid: E ( A , a ) ∩ ( c T x � c T a ) where c ∈ R n 152

Half-ellipsoid theorem Proof of the correctness of next formula can be found in book of Grötschel, Lovász and Schrijver: Geometric algorithms and combinatorial optimization . Lemma 6.6 (Half-Ellipsoid-Theorem) The half-ellipsoid E ( A , b ) ∩ ( c T x � c T a ) is contained in the ellipsoid E ′ ( A ′ , a ′ ) and one has vol( E ′ )/vol( E ) � e − 1/(2 n ) . Here E ′ ( A ′ , a ′ ) is defined by 1 a ′ = a − n + 1 b (19) n 2 � 2 � A ′ n + 1 bb T = A − , (20) n 2 − 1 � c T Ac. where b is the vector b = Ac / 153

Ellipsoid method S ⊆ R n convex compact set. Suppose the following: I) We have an ellipsoid E init which contains S . II) We have separation oracle for S Ellipsoid method decides whether vol( S ) < L or computes a point x ∗ ∈ S Ellipsoid method a) (Initialize): Set E ( A , a ) : = E init b) If vol( E ( A , a )) < L , then stop. c) If a ∈ S , then assert S �= � and stop d) Otherwise, compute inequality c T x � β which is valid for S and satisfies c T a > β and replace E ( A , a ) by E ( A ′ , a ) computed with formula (19) and goto step c). 154

Theorem 6.7 The ellipsoid method computes a point in S or asserts that vol( S ) < L. The number of iterations is bounded by 2 · n ln(vol( E init )/ L ) . Further remarks � The ellipsoid method can be used to solve convex programming problems in polynomial time under certain conditions. The exact formulation of the result involves some rounding arguments and is beyond the scope of a lecture on Optimization Methods in Finance. Instead we refer to the book of Grötschel, Lovász and Schrijver: Geometric algorithms and combinatorial optimization for a thorough account. � The ellipsoid algorithm was in particular the first polynomial time method for linear programming. 155

Primary objectives: � Convex optimization ✔ � Ellipsoid method ✔ � A polynomial algorithm for linear programming 156