Convex Optimization 4. Convex Optimization Problems Prof. Ying Cui - PowerPoint PPT Presentation

Convex Optimization 4. Convex Optimization Problems Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2018 SJTU Ying Cui 1 / 64

Outline Optimization problems Convex optimization Linear optimization problems Quadratic optimization problems Geometric programming Semidefinite programing Vector optimization SJTU Ying Cui 2 / 64

Optimization problem in standard form min f 0 ( x ) x s.t. f i ( x ) ≤ 0 , i = 1 , · · · , m i = 1 , · · · , p h i ( x ) = 0 , ◮ optimization variable: x ∈ R n ◮ objective function: f 0 : R n → R ◮ inequality constraint functions: f i : R n → R , i = 1 , · · · , m ◮ equality constraint functions: h i : R n → R , i = 1 , · · · , p ◮ domain: D = � m i =0 dom f i ∩ � p i =1 dom h i ◮ feasible point x : x ∈ D and x satisfies all constraints ◮ feasible set or constraint set X : set of all feasible points ◮ feasible problem: problem with nonempty feasible set SJTU Ying Cui 3 / 64

Optimal and locally optimal points ◮ optimal value: p ∗ = inf { f 0 ( x ) | f i ( x ) ≤ 0 , i = 1 , · · · , m , h i ( x ) = 0 , i = 1 , · · · , p } ◮ p ∗ = ∞ if problem is infeasible (standard convention: the infimum of the empty set is ∞ ) ◮ p ∗ = −∞ if problem is unbounded below ◮ (globally) optimal point x ∗ : x ∗ is feasible and f 0 ( x ∗ ) = p ∗ ◮ optimal set X opt : set of optimal points ◮ if X opt is nonempty, optimal value is achieved and problem is solvable ◮ otherwise, optimal value is not achieved (always occurs when problem unbounded below) ◮ locally optimal point x : ∃ R > 0 such that x is optimal for min f 0 ( z ) z s . t . f i ( z ) ≤ 0 , i = 1 , · · · , m i = 1 , · · · , p h i ( z ) = 0 , � z − x � 2 ≤ R SJTU Ying Cui 4 / 64

Optimal and locally optimal points ◮ if f i ( x ) = 0 for feasible point x , i -th inequality constraint f i ( x ) ≤ 0 is active at x ◮ if f i ( x ) < 0 for feasible point x , i -th inequality constraint f i ( x ) ≤ 0 is inactive at x ◮ equality constraints are active at all feasible points ◮ a constraint is redundant if deleting it does not change the feasible set examples (simple unconstrained problems, n = 1 , m = p = 0) ◮ f 0 ( x ) = 1 / x , dom f 0 = R ++ : p ∗ = 0, not achieved ◮ f 0 ( x ) = − log x , dom f 0 = R ++ : p ∗ = −∞ , problem unbounded below ◮ f 0 ( x ) = x log x , dom f 0 = R ++ : p ∗ = − 1 / e , achieved at the unique optimal point x ∗ = 1 / e ◮ f 0 ( x ) = x 3 − 3 x , p ∗ = −∞ , problem unbounded below, locally optimal point x = 1 SJTU Ying Cui 5 / 64

Explicit and implicit constraints ◮ explicit constraints: f i ( x ) ≤ 0 , i = 1 , · · · , m , h i ( x ) = 0 , i = 1 , · · · , p ◮ a problem is unconstrained if it has no explicit constraints ( m = p = 0) ◮ implicit constraint m p � � x ∈ D = dom f i ∩ dom h i i =0 i =1 example: k � log( b i − a T min f 0 ( x ) = − i x ) x i =1 is an unconstrained problem with implicit constraints a T i x < b i , i = 1 , · · · , k SJTU Ying Cui 6 / 64

Feasibility problems The feasibility problem is to determine whether the constraints are consistent, and if so, find a point that satisfies them, i.e., find x s . t . f i ( x ) ≤ 0 , i = 1 , ..., m h i ( x ) = 0 , i = 1 , ..., p It can be considered a special case of the general problem with f 0 ( x ) = 0, i.e., min 0 x f i ( x ) ≤ 0 , s . t . i = 1 , ..., m h i ( x ) = 0 , i = 1 , ..., p ◮ p ∗ = 0 if the feasible set X is nonempty ◮ any feasible point x ∈ X is optimal ◮ p ∗ = ∞ if the feasible set X is empty SJTU Ying Cui 7 / 64

Convex optimization problems in standard form min f 0 ( x ) x s . t . f i ( x ) ≤ 0 , i = 1 , ..., m a T i x = b i , i = 1 , ..., p (or Ax = b ) ◮ problem is convex, if objective function f 0 and inequality constraint functions f 1 , ..., f m are convex, and equality constraints are affine ◮ problem is quasiconvex, if f 0 is quasiconvex and f 1 , ..., f m are convex ◮ feasible set of a convex optimization problem is convex ◮ intersection of domain D = � m i =0 dom f i with m sublevel sets { x | f i ( x ) ≤ 0 } and p hyperplanes { x | a T i x = b i } (all convex) SJTU Ying Cui 8 / 64

Abstract form convex optimization problem example f 0 ( x ) = x 2 1 + x 2 min 2 x f 1 ( x ) = x 1 / (1 + x 2 s . t . 2 ) ≤ 0 h 1 ( x ) = ( x 1 + x 2 ) 2 = 0 ◮ not a convex optimization problem in standard form (according to our definition) ◮ f 1 is not convex, h 1 is not affine ◮ minimize a convex function over a convex set ◮ f 0 is convex, feasible set { ( x 1 , x 2 ) | x 1 = − x 2 ≤ 0 } is convex ◮ equivalent (but not identical) to the convex problem x 2 1 + x 2 min 2 x s . t . x 1 ≤ 0 x 1 + x 2 = 0 SJTU Ying Cui 9 / 64

Local and global optima fundamental property : any locally optimal point of a convex problem is (globally) optimal proof: Suppose x is locally optimal, i.e., x is feasible and ( a ) = inf { f 0 ( z ) | z feasible , || z − x || 2 ≤ R } f 0 ( x ) for some R > 0. Suppose x is not globally optimal, i.e., there exists a feasible y such that f 0 ( y ) < f 0 ( x ). Evidently || y − x || 2 > R . R Consider z = θ y + (1 − θ ) x with θ = 2 � y − x � 2 . Then, we have || z − x || 2 = θ || y − x || 2 = R / 2 < R , and by convexity of the feasible set, z is feasible. By convexity of f 0 , we have f 0 ( z ) ≤ θ f 0 ( y ) + (1 − θ ) f 0 ( x ) < f 0 ( x ) which contradicts (a). SJTU Ying Cui 10 / 64

Optimality criterion for differentiable f 0 x is optimal iff x ∈ X and ∇ f 0 ( x ) T ( y − x ) ≥ 0 for all y ∈ X where X = { x | f i ( x ) ≤ 0 , i = 1 , · · · , m , h i ( x ) = 0 , i = 1 , · · · , p } denotes the feasible set −∇ f 0 ( x ) x X Figure 4.2 Geometric interpretation of the optimality condition (4.21). The feasible set X is shown shaded. Some level curves of f 0 are shown as dashed lines. The point x is optimal: −∇ f 0 ( x ) defines a supporting hyperplane (shown as a solid line) to X at x . ◮ geometric interpretation: if ∇ f 0 ( x ) � = 0, −∇ f 0 ( x ) defines a supporting hyperplane to feasible set X at x SJTU Ying Cui 11 / 64

Optimality criterion for differentiable f 0 ◮ unconstrained problem: min x f 0 ( x ) x is optimal iff x ∈ dom f 0 , ∇ f 0 ( x ) = 0 ◮ equality constrained problem: min x f 0 ( x ) s.t. Ax = b x is optimal iff there exists a v ∈ R p such that x ∈ dom f 0 , Ax = b , ∇ f 0 ( x ) + Av = 0 ◮ minimization over nonnegative orthant: min x f 0 ( x ) s.t. x � 0 x is optimal iff x ∈ dom f 0 , x � 0 , ∇ f 0 ( x ) � 0 , ( ∇ f 0 ( x )) i x i = 0 , i = 1 , · · · , n ◮ condition ( ∇ f 0 ( x )) i x i = 0 is called complementarity: the sparsity patterns (i.e., the set of indices corresponding to nonzero components) of the vectors x and ∇ f 0 ( x ) are complementary (i.e., have empty intersection) SJTU Ying Cui 12 / 64

Equivalent convex problems ◮ two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa ◮ some common equivalent transformations preserve convexity SJTU Ying Cui 13 / 64

Equivalent convex problems eliminating equality constraints min f 0 ( x ) x s . t . f i ( x ) ≤ 0 , i = 1 , ..., m Ax = b is equivalent to min f 0 ( Fz + x 0 ) z s . t . f i ( Fz + x 0 ) ≤ 0 , i = 1 , ..., m where F and x 0 are such that Ax = b ⇔ x = Fz + x 0 for some z ◮ in principle, we can restrict our attention to convex optimization problems without equality constraints ◮ in many cases, however, it is better to retain equality constraints, for ease of analysis, or not to ruin efficiency of an algorithm that solves it SJTU Ying Cui 14 / 64

Equivalent convex problems introducing equality constraints min f 0 ( A 0 x + b 0 ) x s . t . f i ( A i x + b i ) ≤ 0 , i = 1 , ..., m where A i ∈ R k i × n , is equivalent to min f 0 ( y 0 ) x , y s . t . f i ( y i ) ≤ 0 , i = 1 , ..., m y i = A i x + b i , i = 0 , ..., m ◮ introduce a new variable y i ∈ R k i , replace f i ( A i x + b i ) with f i ( y i ), and add the linear equality constraint y i = A i x + b i SJTU Ying Cui 15 / 64

Equivalent convex problems introducing slack variables for linear inequalities min f 0 ( x ) x a T s . t . i x ≤ b i , i = 1 , ..., m is equivalent to min f 0 ( x ) x , s a T s . t . i x + s i = b i , i = 1 , ..., m s i ≥ 0 , i = 1 , ..., m ◮ s i is called the slack variable associated with a T i x ≤ b i ◮ replace each linear inequality constraint with a linear equality constraint and a nonnegativity constraint SJTU Ying Cui 16 / 64

Equivalent convex problems epigraph problem form min f 0 ( x ) x s . t . f i ( x ) ≤ 0 , i = 1 , ..., m Ax = b is equivalent to min t x , t s . t . f 0 ( x ) − t ≤ 0 f i ( x ) ≤ 0 , i = 1 , ..., m Ax = b ◮ an optimization problem in the ‘graph space’ ( x , t ): minimize t over the epigraph of f 0 , subject to the constraints on x ◮ linear objective is universal for convex optimization, as convex optimization is readily transformed to one with linear objective ◮ can simplify theoretical analysis and algorithm development SJTU Ying Cui 17 / 64