CS675: Convex and Combinatorial Optimization Fall 2019 Convex Functions Instructor: Shaddin Dughmi
Outline Convex Functions 1 Examples of Convex and Concave Functions 2 Convexity-Preserving Operations 3
Convex Functions A function f : R n → R is convex if the line segment between any points on the graph of f lies above f . i.e. if x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≤ θf ( x ) + (1 − θ ) f ( y ) Convex Functions 1/24
Convex Functions A function f : R n → R is convex if the line segment between any points on the graph of f lies above f . i.e. if x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≤ θf ( x ) + (1 − θ ) f ( y ) Inequality called Jensen’s inequality (basic form) Convex Functions 1/24
Convex Functions A function f : R n → R is convex if the line segment between any points on the graph of f lies above f . i.e. if x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≤ θf ( x ) + (1 − θ ) f ( y ) Inequality called Jensen’s inequality (basic form) f is convex iff its restriction to any line { x + tv : t ∈ R } is convex Convex Functions 1/24
Convex Functions A function f : R n → R is convex if the line segment between any points on the graph of f lies above f . i.e. if x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≤ θf ( x ) + (1 − θ ) f ( y ) Inequality called Jensen’s inequality (basic form) f is convex iff its restriction to any line { x + tv : t ∈ R } is convex f is strictly convex if inequality strict when x � = y . Convex Functions 1/24
Convex Functions A function f : R n → R is convex if the line segment between any points on the graph of f lies above f . i.e. if x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≤ θf ( x ) + (1 − θ ) f ( y ) Inequality called Jensen’s inequality (basic form) f is convex iff its restriction to any line { x + tv : t ∈ R } is convex f is strictly convex if inequality strict when x � = y . Analogous definition when the domain of f is a convex subset D of R n Convex Functions 1/24
Concave and Affine Functions A function is f : R n → R is concave if − f is convex. Equivalently: Line segment between any points on the graph of f lies below f . If x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≥ θf ( x ) + (1 − θ ) f ( y ) Convex Functions 2/24
Concave and Affine Functions A function is f : R n → R is concave if − f is convex. Equivalently: Line segment between any points on the graph of f lies below f . If x, y ∈ R n and θ ∈ [0 , 1] , then f ( θx + (1 − θ ) y ) ≥ θf ( x ) + (1 − θ ) f ( y ) f : R n → R is affine if it is both concave and convex. Equivalently: Line segment between any points on the graph of f lies on the graph of f . f ( x ) = a ⊺ x + b for some a ∈ R n and b ∈ R . Convex Functions 2/24
We will now look at some equivalent definitions of convex functions First Order Definition A differentiable f : R n → R is convex if and only if the first-order approximation centered at any point x underestimates f everywhere. f ( y ) ≥ f ( x ) + ( ▽ f ( x )) ⊺ ( y − x ) Convex Functions 3/24
We will now look at some equivalent definitions of convex functions First Order Definition A differentiable f : R n → R is convex if and only if the first-order approximation centered at any point x underestimates f everywhere. f ( y ) ≥ f ( x ) + ( ▽ f ( x )) ⊺ ( y − x ) Local information → global information If ▽ f ( x ) = 0 then x is a global minimizer of f Convex Functions 3/24
Second Order Definition A twice differentiable f : R n → R is convex if and only if its Hessian matrix ▽ 2 f ( x ) is positive semi-definite for all x . (We write ▽ 2 f ( x ) � 0 ) Convex Functions 4/24
Second Order Definition A twice differentiable f : R n → R is convex if and only if its Hessian matrix ▽ 2 f ( x ) is positive semi-definite for all x . (We write ▽ 2 f ( x ) � 0 ) Intepretation Recall definition of PSD: z ⊺ ▽ 2 f ( x ) z ≥ 0 for all z ∈ R n When n = 1 , this is f ′′ ( x ) ≥ 0 . More generally, z ⊺ ▽ 2 f ( x ) z is the second derivative of f along the || z || 2 line { x + tz : t ∈ R } . So if ▽ 2 f ( x ) � 0 then f curves upwards along any line. Moving from x to x + δ� z , infitisimal change in gradient is δ ▽ 2 f ( x ) z . When ▽ 2 f ( x ) � 0 , this is in roughly the same direction as � z . Convex Functions 4/24
Epigraph The epigraph of f is the set of points above the graph of f . Formally, epi ( f ) = { ( x, t ) : t ≥ f ( x ) } Convex Functions 5/24
Epigraph The epigraph of f is the set of points above the graph of f . Formally, epi ( f ) = { ( x, t ) : t ≥ f ( x ) } Epigraph Definition f is a convex function if and only if its epigraph is a convex set. Convex Functions 5/24
Jensen’s Inequality (General Form) f : R n → R is convex if and only if For every x 1 , . . . , x k in the domain of f , and θ 1 , . . . , θ k ≥ 0 such that � i θ i = 1 , we have � � f ( θ i x i ) ≤ θ i f ( x i ) i i Given a probability measure D on the domain of f , and x ∼ D , f ( E [ x ]) ≤ E [ f ( x )] Convex Functions 6/24
Jensen’s Inequality (General Form) f : R n → R is convex if and only if For every x 1 , . . . , x k in the domain of f , and θ 1 , . . . , θ k ≥ 0 such that � i θ i = 1 , we have � � f ( θ i x i ) ≤ θ i f ( x i ) i i Given a probability measure D on the domain of f , and x ∼ D , f ( E [ x ]) ≤ E [ f ( x )] Adding noise to x can only increase f ( x ) in expectation. Convex Functions 6/24
Local and Global Optimality Local minimum x is a local minimum of f if there is a an open ball B containing x where f ( y ) ≥ f ( x ) for all y ∈ B . Local and Global Optimality When f is convex, x is a local minimum of f if and only if it is a global minimum. Convex Functions 7/24
Local and Global Optimality Local minimum x is a local minimum of f if there is a an open ball B containing x where f ( y ) ≥ f ( x ) for all y ∈ B . Local and Global Optimality When f is convex, x is a local minimum of f if and only if it is a global minimum. This fact underlies much of the tractability of convex optimization. Convex Functions 7/24
Sub-level sets � x 2 + y 2 Level sets of f ( x, y ) = Sublevel set The α -sublevel set of f is { x ∈ domain ( f ) : f ( x ) ≤ α } . Convex Functions 8/24
Sub-level sets � x 2 + y 2 Level sets of f ( x, y ) = Sublevel set The α -sublevel set of f is { x ∈ domain ( f ) : f ( x ) ≤ α } . Fact Every sub-level set of a convex function is a convex set. This fact also underlies tractability of convex optimization Convex Functions 8/24
Sub-level sets � x 2 + y 2 Level sets of f ( x, y ) = Sublevel set The α -sublevel set of f is { x ∈ domain ( f ) : f ( x ) ≤ α } . Fact Every sub-level set of a convex function is a convex set. This fact also underlies tractability of convex optimization Note: converse false, but nevertheless useful check. Convex Functions 8/24
Other Basic Properties Continuity Real-valued convex functions are continuous on the interior of their domain. Convex Functions 9/24
Other Basic Properties Continuity Real-valued convex functions are continuous on the interior of their domain. Extended-value extension If a function f : D → R is convex on its domain, and D is convex, then it can be extended to a convex function on R n by setting f ( x ) = ∞ whenever x / ∈ D . f : D → R � ∞ is “convex” This simplifies notation. Resulting function � with respect to the ordering on R � ∞ Convex Functions 9/24
Outline Convex Functions 1 Examples of Convex and Concave Functions 2 Convexity-Preserving Operations 3
Functions on the reals Affine: ax + b Exponential: e ax convex for any a ∈ R Powers: x a convex on R ++ when a ≥ 1 or a ≤ 0 , and concave for 0 ≤ a ≤ 1 Logarithm: log x concave on R ++ . Examples of Convex and Concave Functions 10/24
Norms Norms are convex. || θx + (1 − θ ) y || ≤ || θx || + || (1 − θ ) y || = θ || x || + (1 − θ ) || y || Uses both norm axioms: triangle inequality, and homogeneity. Applies to matrix norms, such as the spectral norm (radius of induced ellipsoid) Examples of Convex and Concave Functions 11/24
Norms Norms are convex. || θx + (1 − θ ) y || ≤ || θx || + || (1 − θ ) y || = θ || x || + (1 − θ ) || y || Uses both norm axioms: triangle inequality, and homogeneity. Applies to matrix norms, such as the spectral norm (radius of induced ellipsoid) Max max i x i is convex max i ( θx + (1 − θ ) y ) i = max i ( θx i + (1 − θ ) y i ) ≤ max θx i + max i (1 − θ ) y i i = θ max x i + (1 − θ ) max y i i i If i’m allowed to pick the maximum entry of θx and θy independently, I can do only better. Examples of Convex and Concave Functions 11/24
Log-sum-exp: log( e x 1 + e x 2 + . . . + e x n ) is convex Geometric mean: ( � n 1 n is concave i =1 x i ) Log-determinant: log det X is concave Quadratic form: x ⊺ Ax is convex iff A � 0 Other examples in book f ( x, y ) = log( e x + e y ) Examples of Convex and Concave Functions 12/24
Recommend
More recommend
