MATH 4211/6211 Optimization Convex Optimization Problems Xiaojing - PowerPoint PPT Presentation

MATH 4211/6211 – Optimization Convex Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0

A set Ω ⊂ R n is called convex if for any x , y ∈ Ω , there is Definition. α x + (1 − α ) y ∈ Ω for all α ∈ [0 , 1] Definition. A function f : Ω → R , where Ω is a convex set, is called convex if for any x , y ∈ Ω and α ∈ [0 , 1] , there is f ( α x + (1 − α ) y ) ≤ αf ( x ) + (1 − α ) f ( y ) . Moreover, f is called strictly convex if for any distinct x , y ∈ Ω and α ∈ (0 , 1) , there is f ( α x + (1 − α ) y ) < αf ( x ) + (1 − α ) f ( y ) A function f is called (strictly) concave if − f is (strictly) convex. Xiaojing Ye, Math & Stat, Georgia State University 1

There is an alternative definition based on the convexity of the epigraph of f . Definition. The graph of f : Ω → R is defined by { [ x ; f ( x )] ∈ R n +1 : x ∈ Ω } Definition. The epigraph of f : Ω → R is defined by epi( f ) := { [ x ; β ] ∈ R n +1 : x ∈ Ω , β ≥ f ( x ) } Definition. A function f : Ω → R , where Ω is a convex set, is called convex if epi( f ) is a convex set. Xiaojing Ye, Math & Stat, Georgia State University 2

Example. Let f ( x ) = x 1 x 2 be defined on Ω := { x : x ≥ 0 } . Is f convex? Solution. f is not convex. The set Ω ⊂ R 2 is convex. But if we choose x = [1; 2] and y = [2; 1] , then α x + (1 − α ) y = [2 − α ; 1 + α ] . On the one hand f ( α x + (1 − α ) y ) = 2 + α − α 2 . On the other hand, αf ( x ) + (1 − α ) f ( y ) = 2 . Choosing α = 1 / 2 yields f ( α x + (1 − α ) y ) > αf ( x ) + (1 − α ) f ( y ) which means that f is not convex. Xiaojing Ye, Math & Stat, Georgia State University 3

There are several sufficient and necessary conditions for the convexity of f . Theorem. If f : R n → R is C 1 and Ω is convex, then f is convex on Ω iff for all x , y ∈ Ω , f ( y ) ≥ f ( x ) + ∇ f ( x ) ⊤ ( y − x ) Proof. ( ⇒ ) Suppose f is convex, then for any x , y ∈ Ω and α ∈ (0 , 1] , f ((1 − α ) x + α y ) ≤ (1 − α ) f ( x ) + αf ( y ) Rearrange terms to obtain f ( x + α ( y − x )) − f ( x ) ≤ f ( y ) − f ( x ) α Taking the limit as α → 0 yields f ( y ) ≥ f ( x ) + ∇ f ( x ) ⊤ ( y − x ) Xiaojing Ye, Math & Stat, Georgia State University 4

Proof (cont.) ( ⇐ ) For any x , y ∈ Ω and α ∈ [0 , 1] , define x α = α x + (1 − α ) y . Then f ( x ) ≥ f ( x α ) + ∇ f ( x α ) ⊤ ( x α − x ) f ( y ) ≥ f ( x α ) + ∇ f ( x α ) ⊤ ( x α − y ) Multiplying the two inequalities by α and 1 − α respectively, and adding to- gether yields f ( α x + (1 − α ) y ) ≤ αf ( x ) + (1 − α ) f ( y ) Xiaojing Ye, Math & Stat, Georgia State University 5

Theorem. Let f : R n → R be C 2 and Ω be convex, then f is convex on Ω iff ∇ 2 f ( x ) � 0 for all x ∈ Ω . Proof. ( ⇒ ) If not, then exist x ∈ Ω and d ∈ R n , such that d ⊤ ∇ 2 f ( x ) d < 0 Since ∇ 2 f ( x ) is continuous, there exists s > 0 sufficiently small, such that for y = x + s d ∈ Ω , there is f ( y ) = f ( x ) + ∇ f ( x ) ⊤ ( y − x ) + 1 2( y − x ) ⊤ ∇ 2 f ( x + t d )( y − x ) < f ( x ) + ∇ f ( x ) ⊤ ( y − x ) for some t ∈ (0 , s ) since ( y − x ) ⊤ ∇ 2 f ( x + t d )( y − x ) = s 2 d ⊤ ∇ 2 f ( x + t d ) d < 0 . Hence f is not convex, a contradiction. Xiaojing Ye, Math & Stat, Georgia State University 6

Proof (cont.) ( ⇐ ) For any x , y ∈ Ω , there is f ( y ) = f ( x ) + ∇ f ( x ) ⊤ ( y − x ) + 1 2( y − x ) ⊤ ∇ 2 f ( x + t d )( y − x ) ≥ f ( x ) + ∇ f ( x ) ⊤ ( y − x ) where d := y − x and t ∈ (0 , 1) . Note that we used the fact that ∇ 2 f ( x + t d ) � 0 . Hence f is convex. Xiaojing Ye, Math & Stat, Georgia State University 7

Examples. Determine if any of the following functions is convex. f 1 ( x ) = − 8 x 2 f 2 ( x ) = 4 x 2 1 + 3 x 2 2 + 5 x 2 3 + 6 x 1 x 2 + x 1 x 3 − 3 x 1 − 2 x 2 + 15 f 3 ( x ) = 2 x 1 x 2 − x 2 1 − x 2 2 Solution. f ′′ 1 ( x ) = − 16 < 0 , so f 1 is concave. For f 2 , we have   8 6 1 ∇ 2 f 2 = 6 6 0     1 0 10 whose leading principal minors are 8 , 12 , 114 . Hence f 2 is convex. For f 3 , we have � � − 2 2 ∇ 2 f 3 = 2 − 2 whose eigenvalues are − 4 and 0 , hence f 3 is negative semidefinite. Xiaojing Ye, Math & Stat, Georgia State University 8

Theorem. Suppose f : Ω → R is convex. Then x is a global minimizer of f on Ω iff it is a local minimizer of f . Proof. The necessity is trivial. Suppose x is a local minimizer, then ∃ r > 0 such that f ( x ) ≤ f ( z ) for all z ∈ B ( x , r ) . If ∃ y , such that f ( x ) > f ( y ) , r then let α = � y − x � and r x α = (1 − α ) x + α y = x + � y − x � ( y − x ) . Then x α ∈ B ( x , r ) and f ( x α ) ≥ f ( x ) > (1 − α ) f ( x ) + αf ( y ) , which is a contradiction. Hence x must be a global minimizer. Xiaojing Ye, Math & Stat, Georgia State University 9

Lemma. Suppose f : Ω → R is convex. Then the sub-level set of f Γ c = { x ∈ Ω : f ( x ) ≤ c } is empty or convex for any c ∈ R . Proof. If x , y ∈ Γ c , then f ( x ) , f ( y ) ≤ c . Since f is convex, we have f ( α x + (1 − α ) y ) ≤ αf ( x ) + (1 − α ) f ( y ) ≤ c i.e., α x + (1 − α ) y ∈ Γ c for all α ∈ [0 , 1] . Hence Γ c is a convex set. Xiaojing Ye, Math & Stat, Georgia State University 10

Corollary. Suppose f : Ω → R is convex. Then the set of all global minimiz- ers of f over Ω is convex. Proof. Let f ∗ = min x ∈ Ω f ( x ) . Then Γ f ∗ is the set of all global minimizers. By the lemma above, we knwo Γ c is a convex set. Xiaojing Ye, Math & Stat, Georgia State University 11

Lemma. Suppose f : Ω → R is convex and C 1 . Then x ∗ is a global minimizer of f over Ω iff ∇ f ( x ∗ ) ⊤ ( x − x ∗ ) ≥ 0 , ∀ x ∈ Ω . Proof. ( ⇒ ) If not, then ∃ x ∈ Ω , s.t. ∇ f ( x ∗ ) ⊤ ( x − x ∗ ) < 0 Denote x α = (1 − α ) x ∗ + α x = x ∗ + α ( x − x ∗ ) for α ∈ (0 , 1) . Since f ∈ C 1 , we know there exists α small enough, s.t. ∀ α ′ ∈ (0 , α ) ∇ f ( x α ′ ) ⊤ ( x − x ∗ ) < 0 , Xiaojing Ye, Math & Stat, Georgia State University 12

Proof (cont.) Moreover, there exists α ′ ∈ (0 , α ) s.t. f ( x α ) = f ( x ∗ ) + ∇ f ( x α ′ ) ⊤ ( x α − x ∗ ) = f ( x ∗ ) + α ∇ f ( x α ′ ) ⊤ ( x − x ∗ ) < f ( x ∗ ) which contradicts to x ∗ being a global minimizer. ( ⇐ ) For all x ∈ Ω , there is f ( x ) ≥ f ( x ∗ ) + ∇ f ( x ∗ ) ⊤ ( x − x ∗ ) ≥ f ( x ∗ ) Hence x ∗ is a global minimizer. Xiaojing Ye, Math & Stat, Georgia State University 13

Then x ∗ is a global Suppose f : Ω → R is convex and C 1 . Theorem. minimizer of f over Ω iff for any feasible direction d at x ∗ there is d ⊤ ∇ f ( x ∗ ) ≥ 0 . Proof. ( ⇒ ) Let d be feasible, then ∃ x ∈ Ω s.t. x − x ∗ = α d for some α > 0 . Hence by the Lemma above, we have ∇ f ( x ∗ ) ⊤ ( x − x ∗ ) = α ∇ f ( x ∗ ) ⊤ d ≥ 0 . So ∇ f ( x ∗ ) ⊤ d ≥ 0 . ( ⇐ ) For any x ∈ Ω , we know x α = (1 − α ) x ∗ + α x ∈ Ω for all α ∈ (0 , 1) . Hence d = x − x ∗ = ( x α − x ∗ ) /α is a feasible direction. Therefore ∇ f ( x ∗ ) ⊤ ( x − x ∗ ) = ∇ f ( x ∗ ) ⊤ d ≥ 0 . As x ∈ Ω is arbitrary, we know x ∗ is a global minimizer. Xiaojing Ye, Math & Stat, Georgia State University 14

Corollary. Suppose f : Ω → R is convex and C 1 . If x ∗ ∈ Ω is such that ∇ f ( x ∗ ) = 0 , then x ∗ is a global minimizer of f . Proof. For any feasible d there is ∇ f ( x ∗ ) ⊤ d = 0 . Hence x ∗ is a global minimizer. Xiaojing Ye, Math & Stat, Georgia State University 15

Theorem. Let f : R n → R and f ∈ C 1 be convex, and Ω = { x ∈ R n : h ( x ) = 0 } where h : R n → R m such that Ω is convex. Then x ∗ ∈ Ω is a global minimizer of f over Ω iff there exists λ ∗ ∈ R m such that ∇ f ( x ∗ ) + D h ( x ∗ ) ⊤ λ ∗ = 0 . Proof. ( ⇒ ) By the KKT condition. ( ⇐ ) Note that f being convex implies f ( x ) ≥ f ( x ∗ ) + ∇ f ( x ∗ ) ⊤ ( x − x ∗ ) , ∀ x ∈ Ω Also note that ∇ f ( x ∗ ) = − D h ( x ∗ ) ⊤ λ ∗ , we know f ( x ) ≥ f ( x ∗ ) − λ ∗⊤ D h ( x ∗ )( x − x ∗ ) Xiaojing Ye, Math & Stat, Georgia State University 16

Proof (cont.) For any x ∈ Ω , we know x ∗ + α ( x − x ∗ ) ∈ Ω for all α ∈ (0 , 1) . Hence h ( x ∗ + α ( x − x ∗ )) = 0 and h ( x ∗ + α ( x − x ∗ )) − h ( x ∗ ) D h ( x ∗ )( x − x ∗ ) = lim = 0 α α → 0 Hence f ( x ) ≥ f ( x ∗ ) for all x ∈ Ω . Therefore x ∗ is a global minimizer. Xiaojing Ye, Math & Stat, Georgia State University 17

Theorem. Let f : R n → R and f ∈ C 1 be convex, and Ω = { x ∈ R n : h ( x ) = 0 , g ( x ) ≤ 0 } where h : R n → R m and g : R n → R p are C 1 and such that Ω is convex. Then x ∗ ∈ Ω is a global minimizer of f over Ω iff there exist λ ∗ ∈ R m , µ ∗ ∈ R p + such that ∇ f ( x ∗ ) ⊤ + λ ∗⊤ D h ( x ∗ ) + µ ∗⊤ D g ( x ∗ ) = 0 ⊤ , g ( x ∗ ) ⊤ µ ∗ = 0 . Proof. ( ⇒ ) By the KKT condition. Xiaojing Ye, Math & Stat, Georgia State University 18