Computational Optimization Convexity and Unconstrained Optimization - PowerPoint PPT Presentation

Computational Optimization Convexity and Unconstrained Optimization 1/29/08 and 2/1(revised)

Convex Sets A set S is convex if the line segment joining any two points in the set is also in the set, i.e., for any x,y ∈ S, λ x+(1- λ )y ∈ S for all 0 ≤ λ ≤ 1 }. convex not convex convex not convex not convex

Proving Convexity Prove {x|Ax<=b} is convex. { } ≤ Let x and y be elements of C= x|Ax b . λ ∈ For any (0,1), λ + − λ = λ + − λ ( (1 ) ) (1 ) A x y Ax Ay ≤ λ + − λ = (1 ) b b b ⇒ λ + − λ ∈ (1 ) x y C

You Try Prove D= {x | ||x|| ≤ 1} is convex.

Convex Functions A function f is (strictly) convex on a convex set S, if and only if for any x,y ∈ S, f( λ x+(1- λ )y)(<) ≤ λ f(x)+ (1- λ )f(y) for all 0 ≤ λ ≤ 1. f( λ x+(1- λ )y) f(y) f(x) λ x+(1- λ )y x y

Proving Function Convex Linear functions n ∑ = = ∈ n ( ) ' f x w x w x where x R i i = i 1 ∈ λ ∈ n , (0,1) For any x y R λ + − λ = λ + − λ ( (1 ) ) '( (1 ) ) f x y w x y = λ + − λ ≤ λ + − λ ' (1 ) ' ( ) (1 ) ( ) w x w y f x f y �

You Try 2 2 x 2 + 2 1 x = ) 2 f x x , 1 (

Hint: x 2 is convex λ ∈ Consider any two points x,y and (0,1) λ + − λ = λ + − λ + − λ λ + λ − λ 2 2 2 2 2 2 2 2 (1 ) (1 ) (1 ) (1 ) x y x y x y ( ) 2 = λ + − λ − λ − λ + − λ λ + λ − λ 2 2 (1 ) 2 (1 ) (1 ) (1 ) x y xy x y ( ) 2 = λ + − λ + λ − λ − 2 (1 ) (1 )( ) x y x y ( ) 2 ≥ λ + − λ (1 ) x y λ = λ + − λ λ − λ 2 2 2 2 2 First line uses (1 ) an d similarly for (1 ) . x x x y λ + − λ 2 2 2 2 Second line completes the square of (1 ) . x y Third line observes the remaining terms are a square. λ − λ − ≥ 2 Fouth line follows since (1 )( ) 0. x y

Handy Facts Let be convex functions … ( ), , ( ) g x g x 1 m And a>0. m ∑ Then is convex. = ( ) ( ) f x g i x = 1 i And = is convex. ( ) ( ) h x ag x 1

Convexity and Curvature Convex functions have positive curvature everywhere. Curvature can be measured by the second derivative or Hessian. Properties of the Hessian indicate if a function is convex or not.

Convex Functions A function f is (strictly) convex on a convex set S, if and only if for any x,y ∈ S, f( λ x+(1- λ )y)(<) ≤ λ f(x)+ (1- λ )f(y) for all 0 ≤ λ ≤ 1. f(x+(1- λ )y) f(y) f(x) λ x+(1- λ )y x y

Theorem Let f be twice continuously differentiable. f(x) is convex on S if and only if for all x ∈ S, the Hessian at x 2 ( ) ∇ f x is positive semi-definite.

Definition The matrix H is positive semi-definite (p.s.d.) if and only if for any vector y ′ ≥ 0 y Hy The matrix H is positive definite (p.d.) if and only if for any nonzero vector y ′ > 0 y Hy Similarly for negative (semi-) definite.

Theorem Let f be twice continuously differentiable. f(x) is strictly convex on S if and only if for all x ∈ X, the Hessian at x 2 ( ) ∇ f x is positive definite.

Checking Matrix H is p.s.d/p.d. Manually − ⎡ ⎤ ⎡ ⎤ 4 1 x [ ] = − − + 1 2 2 ⎢ ⎥ ⎢ ⎥ 4 3 x x x x x x x x − 1 2 1 2 1 1 2 2 ⎣ ⎦ ⎣ ⎦ 1 3 x 2 = − + 2 2 4 2 3 x x x x 1 1 2 2 = − + + > ∀ ≠ 2 2 ( ) ^ 2 3 2 0 [ ] 0 x x x x x x 1 2 1 2 1, 2 so matrix is positive definite

Useful facts The sum of convex functions is convex The composition of convex functions is convex.

via eigenvalues The eigenvalues of − ⎡ ⎤ 4 1 are 4.618 and 2.382 ⎢ ⎥ − ⎣ ⎦ 1 3 so matrix is positive definite

Summary: using eigenvalues If all eigenvalues are positive, then matrix is positive definite, p.d. If all eigenvalues are nonnegative, then matrix is positive semi-definite, p.s.d If all eigenvalues are negative, then matrix is negative definite, n.d. If all eigenvalues are nonpositive, then matrix is negative semi-definite, n.s.d Otherwise the matrix is indefinite.

Try with Hessians = + 2 2 ( , ) 2 f x x x x 1 2 1 2 ⎡ ⎤ 2 x ∇ = ⎢ 1 ( ) ⎥ f x ⎣ 4 ⎦ x 2 ⎡ ⎤ 2 0 ∇ = ⎢ 2 ( ) ⎥ f x ⎣ ⎦ 0 4 ⎡ ⎤ ⎡ ⎤ = 2 0 a + > ≠ 2 2 [ ] 2 4 0 [ ] 0 ab ⎢ ⎥ ⎢ ⎥ a b for any ab ⎣ 0 4 ⎦ ⎣ ⎦ b StrictlyConvex

Check Hessian H = [ 2 0; 0 4] Eigs(H) are 2 and 4 So Hessian matrix is always p .d. So function is strictly convex

Differentiability and Convexity For convex function, linear approximation underestimates function f(x) (x*,f(x*)) ′ = + − ∇ ( ) ( *) ( *) ( *) g x f x x x f x

Theorem Assume f is continuously differentiable on a Set S. f is convex on S if and only if ≥ + − ∇ ∀ ∈ ( ) ( ) ( )' ( ) , f y f x y x f x x y S

Theorem Consider problem min f(x) unconstrained. ∇ = If and f is convex, then ( ) 0 f x is a global minimum. x Proof: ∀ y ≥ + − ∇ ( ) ( ) ( )' ( ) byconvexityof f y f x y x f x f = ∇ = ( ) since ( ) 0. f x f x

Unconstrained Optimality Conditions Basic Problem: min ( ) f x (1) ∈ x S Where S is an open set e.g. R n

First Order Necessary Conditions Theorem: Let f be continuously differentiable. If x* is a local minimizer of (1), then ∇ = ( * ) 0 f x

Stationary Points Note that this condition is not sufficient ∇ = ( * ) 0 f x Also true for local max and saddle points

Proof ∇ ≠ Assume false, e.g., ( *) 0 f x = −∇ Let ( *), then d f x ′ + λ = + λ ∇ + λ α λ ( * ) ( *) ( *) ( *, ) f x d f x d f x d x d ⇓ + λ − ( * ) ( *) f x d f x ′ = ∇ + α λ ( *) ( *, ) d f x d x d λ ⇓ + λ − < λ ( * ) ( *) 0 for sufficiently small f x d f x ′∇ < α λ → since ( *) 0and ( *, ) 0. d f x x d !! * is a local min. CONTRADICTION x

Second Order Sufficient Conditions Theorem: Let f be twice continuously differentiable. ∇ = If and ( * ) 0 f x ∇ 2 ( *) is positive definite f x then x* is a strict local minimizer of (1).

Proof Any point x in neighborhood of x* can be written as x*+ λ d for some vector d with norm 1 and λ <= λ *. Since f is twice continuously differentiable, we can choose λ * 2 ( )is p.d. for all ∇ ε ε ε − ≤ λ such that such that * * f x 1 ′ ∀ λ ≤ λ + λ = + λ ∇ + λ ∇ ε 2 2 , *, ( * ) ( *) ( *) ' ( ) d f x d f x d f x d f d 2 ⇓ ∇ = s i n c e ( * ) 0 f x 1 + λ − = λ ∇ > 2 2 ( * ) ( *) ' ( *) 0 f x d f x d f x d 2 ⇓ therefore * is a strict local min. x

Second Order Necessary Conditions Theorem: Let f be twice continuously differentiable. If x* is a local minimizer of (1) then ∇ = ( * ) 0 f x ∇ 2 ( *) is positive semi definite f x

Proof by contradiction Assume false, namely there exists some d such that ′∇ < 2 ( *) 0 then d f x d 1 ′ + λ = + λ ∇ + λ ∇ + λ 2 α λ 2 2 ( * ) ( *) ( *) ' ( *) ( *, ) f x d f x d f x d f x d d x d 2 ⇓ + λ − ( * ) ( *) 1 f x d f x = ∇ + 2 α λ 2 ' ( *) ( *, ) d f x d d x d λ 2 2 ⇓ + λ − < λ ( * ) ( *) 0 for sufficiently small f x d f x ′∇ < α λ → since ( *) 0for some and ( *, ) 0. d f x d d x d !!! x* is a local min. Contrad iction

Example Say we are minimizing 1 = − + − − 2 2 ( , ) 2 15 4 f x x x x x x x x 1 2 1 1 2 2 1 2 2 [8,2]???

Solve FONC Solve FONC to find stationary point * ⎡ ⎤⎡ ⎤ ⎡ ⎤ 1 2 − 15 x ∇ = − = ⎢ ⎥⎢ ⎥ ⎢ ⎥ 2 1 ( , ) 0 f x x 1 2 1 ⎢ ⎥⎣ ⎦ ⎣ ⎦ 4 4 x − ⎣ ⎦ 2 2 − 1 ⎡ ⎤ 1 2 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ − * 15 8 x ⇒ = = ⎢ ⎥ 1 2 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎢ 1 ⎥ ⎣ ⎦ ⎣ ⎦ * 4 4 2 x − ⎣ ⎦ 2 2

Check SOSC The Hessian at x* ⎡ ⎤ 1 2 − ∇ =⎢ ⎥ 2 2 ( *, *) f x x 1 2 1 ⎢ 4 ⎥ − ⎣ ⎦ 2 is p.d. since the eigenvalues, 4.118 and 1.882, are positive. Therefore SOSC are satisfied. x* is a strictly local min;

Alternative Argument The Hessian at every value x is ⎡ ⎤ 1 2 − ∇ =⎢ ⎥ 2 2 ( , ) f x x 1 2 ⎢ 1 ⎥ 4 − ⎣ ⎦ 2 which is p.d. since the eigenvalues, 4.118 and 1.882, are positive. Therefore the function is strictly convex. ∇ Since f(x*)=0 and f is a strictly convex, x* is the unique strict global minimum.