Chapter 4: Problem Statement CK Cheng Dept. of Computer Science and - PowerPoint PPT Presentation

CSE203B Convex Optimization: Chapter 4: Problem Statement CK Cheng Dept. of Computer Science and Engineering University of California, San Diego 1

Convex Optimization Formulation 1. Introduction I. Eliminating equality constants II. Slack variables III. Absolute values, softmax 2. Optimality Conditions I. Local vs. global optimum II. Optimality criterion for differentiable 𝑔 0 i. Optimization without constraints ii. Opt. with inequality constraints iii. Opt. with equality constraints III. Quasi-convex optimization 3. Linear Optimization 4. Quadratic Optimization 5. Geometric Programming 6. Generalized Inequality Constraints 2

1. Introduction Formulation: One of the most critical processes to conduct a project. min 𝑔 0 (𝑦) 𝑡. 𝑢. 𝑔 𝑗 𝑦 ≤ 0 𝑗 = 1, … , 𝑛 ℎ 𝑗 𝑦 = 0 𝑗 = 1, … , 𝑞 (𝐵𝑦 = 𝑐 Affine set) 𝑦 ∈ 𝑆 𝑜 0 : 𝑆 𝑜 → 𝑆 𝐸 0 𝑔 𝑔 𝑗 : 𝑆 𝑜 → 𝑆 𝐸 𝑔 𝑗 𝑔 𝐸 ℎ 𝑗 ℎ 𝑗 : 𝑆 𝑜 → 𝑆 𝑔 0 , 𝑔 𝑗 , … , 𝑔 𝑛 𝑏𝑠𝑓 𝑑𝑝𝑜𝑤𝑓𝑦 𝐸 =∩ 𝑗=0,𝑛 𝐸 𝑔 ∩ 𝑗=0,𝑞 𝐸 ℎ 𝑗 Domain of functions, but not the feasible set. Feasible Set: The set which satisfies the constraints (is convex for convex problems). 3

1.1 Introduction: Eliminating Equality Constraints min 𝑔 0 (𝑦) 𝑡. 𝑢. 𝑔 𝑗 𝑦 ≤ 0 𝑗 = 1, … , 𝑛 𝐵𝑦 = 𝑐 Convert 𝑦 𝐵𝑦 = 𝑐 𝑢𝑝 𝐺𝑨 + 𝑦 0 𝑨 ∈ 𝑆 𝑙 a. b. We have a equivalent problem min 𝑔 0 (𝐺𝑨 + 𝑦 0 ) 𝑡. 𝑢. 𝑔 𝑗 𝐺𝑨 + 𝑦 0 ≤ 0 Remark: Matrix 𝐺 contains columns of null space basis 4

1.2 Introduction: Slack Variables min 𝑔 0 (𝑦) 𝑡. 𝑢. 𝑔 𝑗 𝑦 ≤ 0, 𝑗 = 1, … , 𝑛 𝐵𝑦 = 𝑐 Add slack variables to convert to an equivalent problem a. Convert the objective function with variable t min 𝑢 𝑡. 𝑢. 𝑔 0 𝑦 − 𝑢 ≤ 0 𝑔 𝑗 𝑦 ≤ 0 , 𝑗 = 1, … , 𝑛 𝐵 𝑈 𝑦 = 𝑐 b. Convert the inequality with variables 𝑡 𝑗 min 𝑔 0 (𝑦) 𝑡. 𝑢. 𝑔 𝑗 𝑦 + 𝑡 𝑗 = 0 𝐵 𝑈 𝑦 = 𝑐 𝑡 𝑗 ∈ 𝑆 + , 𝑗 = 1, … , 𝑛 5

1.3 Introduction: Absolute values and Softmax a. Absolute values 𝑔 𝑗 (𝑦) ≤ 𝑐 ⇒ 𝑔 𝑗 𝑦 ≤ 𝑐 𝑏𝑜𝑒 −𝑔 𝑗 𝑦 ≤ 𝑐 b. Maximum values max{𝑔 1 , 𝑔 2 , … , 𝑔 𝑛 } 1 1 + 𝑓 𝛽𝑔 1 + ⋯ + 𝑓 𝛽𝑔 𝛽 log ( 𝑓 𝛽𝑔 Soft𝑛𝑏𝑦: 𝑛 ) 𝐹𝑦𝑏𝑛𝑞𝑚𝑓: max{1, 5, 10, 2, 3} ⇒ Softmax 1 𝛽 log(𝑓 𝛽 + 𝑓 5𝛽 + 𝑓 10𝛽 + 𝑓 2𝛽 + 𝑓 3𝛽 ) ≈ 10 6

ҧ 2.1 Optimality Conditions: Local vs. Global Optima Definition: Local Optima 𝑦 ∈ 𝑆 𝑜 Given a convex optimization problem and a point ҧ If there exists a 𝑠 > 0 𝑡. 𝑢. 𝑔 0 𝑨 ≥ 𝑔 𝑦 for all 𝑨 ∈ Feasible Set, and 𝑨 − ҧ 𝑦 2 ≤ 𝑠 0 Then ҧ 𝑦 is a local optimum . 7

ҧ ҧ ҧ ҧ ҧ 2.2 Optimality Conditions Theorem: Given a convex opt. problem If ҧ 𝑦 is a local optimum, then ҧ 𝑦 is a global optimum Proof: By contradiction Suppose that ∃𝑧 ∈ 𝐺𝑓𝑏𝑡𝑗𝑐𝑚𝑓 𝑇𝑓𝑢 𝑡. 𝑢. 𝑔 𝑦 > 𝑔 0 𝑧 0 We have 𝑔 𝑦 > 1 − 𝜄 𝑔 𝑦 + 𝜄𝑔 0 ത 𝑧 𝑐𝑧 𝑏𝑡𝑡𝑣𝑛𝑞𝑢𝑗𝑝𝑜 0 0 > 𝑔 0 ( 1 − 𝜄 𝑦 + 𝜄ത 𝑧) 𝑔 0 𝑗𝑡 𝑑𝑝𝑜𝑤𝑓𝑦 And 1 − 𝜄 𝑦 + 𝜄ത 𝑧 is feasible (Feasible set is convex) The inequality contradicts to the assumption of local optima. 8

2.2 Optimality Criterion for Differentiable 𝑔 0 𝑦 0 𝑦 𝑈 𝑧 − 𝑦 ≥ 0 , for a given 𝑦 ∈ Feasible Set Theorem: If 𝛼𝑔 and for all 𝑧 ∈ Feasible Set, then 𝑦 is optimal . 0 𝑦 ∈ 𝐿 ∗ ) ( i. e. 𝐿 = 𝑧 − 𝑦 𝑧 ∈ 𝑔𝑓𝑏𝑡𝑗𝑐𝑚𝑓 𝑡𝑓𝑢 , ∇𝑔 Proof: From the first order condition of convex function, we 0 𝑦 𝑈 (𝑧 − 𝑦) . have 𝑔 0 𝑧 ≥ 𝑔 0 𝑦 + 𝛼𝑔 𝑈 𝑦 Given the condition that 𝛼𝑔 𝑧 − 𝑦 ≥ 0, ∀𝑧 in feasible set . 0 We have 𝑔 0 𝑧 ≥ 𝑔 0 𝑦 , ∀𝑧 in feasible set, which implies that 𝑦 is optimal. 𝑈 𝑦 Remark: 𝛼𝑔 𝑧 − 𝑦 = 0 is a supporting hyperplane to 0 feasible set at 𝑦 . 9

2.2.1 Optimality Criterion without Constraints 0 𝑦 , 𝑦 ∈ 𝑆 𝑜 , where 𝑔 Theorem: For problem min 𝑔 0 is convex, the optimal condition is ∇𝑔 0 𝑦 = 0. Proof: ( ∇𝑔 0 𝑦 = 0 ⇒ Optimality) 0 𝑦 𝑈 𝑧 − 𝑦 , ∀𝑦, 𝑧 ∈ 𝑆 𝑜 (first order Since 𝑔 0 𝑧 ≥ 𝑔 0 𝑦 + 𝛼𝑔 condition of convex function) We have 𝑔 0 𝑧 ≥ 𝑔 0 𝑦 . Therefore, x is an optimal solution. ( ∇𝑔 0 𝑦 = 0 ⇐ Optimality) By contradiction 10

ҧ ҧ ҧ 2.2.2 Opt. with Inequality Constraints Problem: Min 𝑔 0 𝑦 s.t. 𝐵𝑦 ≤ 𝑐 , 𝐵 ∈ 𝑆 𝑛×𝑜 Suppose that 𝐵 ҧ 𝑦 = 𝑐 (one particular case). Let 𝑦 = ҧ 𝑦 + 𝑣 . We can write ቊ min 𝑔 𝑦 + 𝑣 0 𝐵𝑣 ≤ 0 0 𝑦 𝑈 𝑣 ≥ 0, ∀{𝑣|𝐵𝑣 ≤ 0} ≡ 𝐿 Opt. condition : 𝛼𝑔 In other words , 𝑦 ∈ 𝐿 ∗ 𝑝𝑔 𝐿 = 𝑣 𝐵𝑣 ≤ 0 𝑏𝑜𝑒 𝐿 ∗ = {−𝐵 𝑈 𝑤|𝑤 ≥ 0} 𝛼𝑔 0 𝑦 = −𝐵 𝑈 𝑤 , ∃𝑤 ∈ 𝑆 + 𝑛 i.e. 𝛼𝑔 0 𝑦) + 𝐵 𝑈 𝑤 = 0, 𝑤 ≥ 0. 𝛼𝑔 0 ( ҧ 11

ҧ ҧ ҧ 2.2.3 Opt. with Equality Constraints ቊ min 𝑔 0 𝑦 𝑡. 𝑢. 𝐵𝑦 = 𝑐 Let 𝑦 = ҧ 𝑦 + 𝑣 and 𝐵 ҧ 𝑦 = 𝑐, we have ቊ min 𝑔 𝑦 + 𝑣 0 , 𝐿 = {𝑣|𝐵𝑣 = 0} 𝐵𝑣 = 0 𝑦 ∈ 𝐿 ∗ , 𝐿 ∗ = {𝐵 𝑈 𝑤|𝑤 ∈ 𝑆 𝑞 } 𝛼𝑔 0 𝑦 + 𝐵 𝑈 𝑤 = 0 𝛼𝑔 0 Let 𝐿 1 = 𝑣 𝐵𝑣 ≥ 0 𝐿 2 = 𝑣 −𝐵𝑣 ≥ 0 W e have ∗ = 𝐵 𝑈 𝑤 𝑤 ≥ 0 𝐿 1 ∗ = −𝐵 𝑈 𝑤 𝑤 ≥ 0 = 𝐵 𝑈 𝑤 𝑤 ≤ 0 𝐿 2 ∗ ∪ 𝐿 2 ∗ = {𝐵 𝑈 𝑤|𝑤 ∈ 𝑆 𝑞 } (𝐿 1 ∩ 𝐿 2 ) ∗ = 𝐿 1 12

2.2.3 Opt. with Equality Constraints: Example 2 + 𝑦 2 2 min 𝑦 𝑔 𝑦 = 𝑦 1 𝑦 1 𝑡. 𝑢. 2 1 𝑦 2 = 3 ∗ = ( We can derive 𝑦 ∗ = 𝑦 1 6 3 ∗ , 𝑦 2 5 , 5 ) 12 12 ∗ 𝛼𝑔 𝑦 ∗ = 2𝑦 1 + 2 , 𝛼𝑔 𝑦 ∗ + 𝐵 𝑈 𝑤 = 6 5 5 ∗ = 1 × − 5 = 0 6 6 2𝑦 2 5 5 New Problem: 2𝑦 1 2𝑦 2 + 2 1 𝑤 = 0 𝛼𝑔 𝑦 + 𝐵 𝑈 𝑤 = 0 ⇒ 𝑦 1 𝐵𝑦 = 𝑐 2 1 𝑦 2 = 3 13

2.3 Quasiconvex Functions 𝑔: 𝑆 𝑜 → 𝑆 is called quasiconvex (unimodal) sublevel set 𝑇 𝑢 = 𝑦 𝑦 ∈ 𝑒𝑝𝑛 𝑔, 𝑔 𝑦 ≤ 𝑢} if its domain and all sublevel sets 𝑇 𝑢 , ∀𝑢 ∈ 𝑆 are convex, 𝑔: 𝑆 𝑜 → 𝑆 is called quasiconcave if −𝑔 is quasiconvex. quasiconvex and quasiconcave → quasilinear 𝑔(𝑦) Ex: log 𝑦, 𝑦 ∈ 𝑆 ++ 14

2.3 Quasiconvex Functions Ex: Ceiling function 𝐷𝑓𝑗𝑚 𝑦 = inf 𝑨 ∈ 𝑎 𝑨 > 𝑦 : quasilinear 𝑦 1 0 1 1 2 𝑦 1 𝑦 2 Ex: 𝑔 𝑦 1 , 𝑦 2 = 𝑦 1 𝑦 2 = 𝑦 2 1 0 2 𝑦 1 𝑦 2 ≥ 𝑢} 2 , 𝑇 𝑢 = 𝑦 ∈ 𝑆 + is quasiconcave in 𝑆 + 𝑏 𝑈 𝑦+𝑐 𝑑 𝑈 𝑦+𝑒 for 𝑑 𝑈 𝑦 + 𝑒 > 0 Ex: 𝑔 𝑦 = 𝑇 𝑢 = 𝑦 𝑑 𝑈 𝑦 + 𝑒 > 0, 𝑏 𝑈 + 𝑐 ≤ 𝑢(𝑑 𝑈 𝑦 + 𝑒)} open halfspace closed halfspace → 𝑇 𝑢 is convex ( 𝑢 is given here) quasiconvex → 𝑔(𝑦) is quasiconcave → quasilinear ൠ 15

2.3 Quasiconvex Optimization min 𝑔 𝑝 (𝑦) (𝑔 𝑝 (𝑦) is quasiconvex, 𝑔 𝑗 ′𝑡 are convex. ) 𝑡. 𝑢. 𝑔 𝑗 𝑦 ≤ 0, 𝑗 = 1, … , 𝑛 𝐵𝑦 = 𝑐 Remark: A locally opt. solution (𝑦, 𝑔 0 𝑦 ) may not be globally opt. Algorithm: Bisection method for quasiconvex optimization. Given 𝑚 ≤ 𝑞 ∗ ≤ 𝑣, 𝜗 > 0 Find a Repeat 1. 𝑢 = (𝑚 + 𝑣)/2 convex function 2. F ind a feasible solution 𝑦 : 𝑡. 𝑢. Φ 𝑢 𝑦 ≤ 0 𝑔 0 𝑦 ≤ 𝑢 ⇔ Φ t 𝑦 ≤ 0 𝑔 𝑗 𝑦 ≤ 0 𝐵𝑦 = 𝑐 3. If solution is feasible, 𝑣 = 𝑢, 𝑓𝑚𝑡𝑓 𝑚 = 𝑢 Until 𝑣 − 𝑚 ≤ 𝜗 𝑞 𝑦 Ex: 𝑔 𝑦 = 𝑟 𝑦 ≤ 𝑢 → 𝑞 𝑦 − 𝑢𝑟 𝑦 ≤ 0 ( p is convex & q is concave) 16

3. Linear Programming: Format General Form : min 𝑑 𝑈 𝑦 𝑡. 𝑢. 𝐻𝑦 ≤ ℎ, 𝐻 ∈ 𝑆 𝑛∗𝑜 , 𝐵 ∈ 𝑆 𝑞∗𝑜 𝐵𝑦 = 𝑐 Standard Form : min 𝑑 𝑈 𝑦 𝑡. 𝑢. 𝐵𝑦 = 𝑐 𝑦 ≥ 0 Remark: Figure out three possible situations 1. No feasible solutions 2. Unbounded solutions 3. Bounded solutions 17

3. Linear Programming: Cases min 𝑑 𝑈 𝑦 𝑡. 𝑢. 𝐵𝑦 = 𝑐 (1) No feasible solutions: 𝑐 ∉ 𝑆(𝐵) ( b is not in the range of A ) 1 1 2 𝑦 1 𝑦 2 = 1 2 2 e.g. 2 3 3 (2) Unbounded solutions: 𝑐 ∈ 𝑆(𝐵) but 𝑑 ∉ 𝑆(𝐵 𝑈 ) 𝑦 1 e.g. min 1 1 𝑦 2 𝑦 1 (The solution → −∞) 𝑦 2 = 2 1 2 (3) Bounded solutions: b ∈ 𝑆 𝐵 , 𝑑 ∈ 𝑆 𝐵 𝑈 𝑦 1 e.g. min 1 1 𝑦 2 𝑦 1 1 1 𝑦 2 = 2 1 2 2 Thus 𝑦 ∗ = 2 2 0 , 𝑔 𝑦 ∗ = 1 0 = 2 1 18