7. Separating Hyperplane Theorems II Daisuke Oyama Mathematics II - PowerPoint PPT Presentation

7. Separating Hyperplane Theorems II Daisuke Oyama Mathematics II May 7, 2020

Farkas’ Lemma Proposition 7.16 (Farkas’ Lemma) Let A ∈ R M × N and b ∈ R N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x = b and x ≥ 0 . 2. For any y ∈ R N , if Ay ≥ 0 , then b T y ≥ 0 . For proof, we will use the following: Lemma 7.17 { A T x ∈ R N | x ∈ R M + } is a closed set. 1 / 32

Proof of Farkas’ Lemma ▶ (1) ⇒ (2): Immediate. ▶ (2) ⇒ (1): Suppose that (1) does not hold. Let K = { A T x ∈ R N | x ∈ R M + } . Then b / ∈ K . ▶ K is convex, and by Lemma 7.17 is closed. ▶ Then by the Separating Hyperplane Theorem, there exist y ∈ R N with y ̸ = 0 and c ∈ R such that y T b < c ≤ y T z for all z ∈ K, and therefore, y T b < inf z ∈ K y T z . ▶ Since K is a cone, it follows that inf z ∈ K y T z = 0 . ( → Homework) ▶ Thus we have y T b < 0 , and y T A T x ≥ 0 for all x ≥ 0 , which implies that y T A T ≥ 0 T . 2 / 32

Proof of Lemma 7.17 Show that K = { A T x ∈ R N | x ∈ R M + } is closed. ▶ Denote the column vectors in A T by a 1 , . . . , a M , so that K = Cone { a 1 , . . . , a M } . ▶ Let { z m } be a sequence in K , and suppose that z m → ¯ z . We want to show that ¯ z ∈ K . eodory’s Theorem, for each m , z m is written as ▶ By Carath´ a conic combination of a linearly independent subset of { a 1 , . . . , a M } . ▶ Since there are finitely many such subsets, there is a linearly independent subset { a i 1 , . . . , a i L } such that infinitely many elements of { z m } are written as its conic combinations. ▶ Denote B = ( a i 1 a i L ) ∈ R N × L , and denote the · · · corresponding subsequence again by { z m } . 3 / 32

▶ Denote z m = Bλ m , where λ m ∈ R L + . ▶ We have B T z m = B T Bλ m , where B T B ∈ R L × L is non-singular: ▶ Let B T Bx = 0 . ▶ Then x T B T Bx = 0 , where x T B T Bx = ∥ Bx ∥ 2 . ▶ Therefore, x T B T Bx = 0 if and only if Bx = 0 . ▶ Since the columns of B are linearly independent, this holds if and only if x = 0 . ▶ Therefore, we have λ m = ( B T B ) − 1 B T z m . ▶ By the continuity of ( B T B ) − 1 B T z in z , λ m converges to ¯ z , where ¯ λ = ( B T B ) − 1 B T ¯ λ ∈ R L + . ▶ Thus, by the continuity of Bλ in λ , we have z = lim m →∞ Bλ m = B ¯ ¯ λ , so that ¯ z ∈ K . 4 / 32

Variants of Farkas’ Lemma Proposition 7.18 (Farkas’ Lemma: Inequality version) Let A ∈ R M × N and b ∈ R N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x ≤ b and x ≥ 0 . 2. For any y ∈ R N , if y ≥ 0 and Ay ≥ 0 , then b T y ≥ 0 . 5 / 32

Proof ▶ (1) ⇒ (2): Immediate. ▶ (2) ⇒ (1): Assume (2). ( A ) It implies that for all y ∈ R N , if y ≥ 0 , then b T y ≥ 0 . I ▶ By Farkas’ Lemma, there exist x ∈ R M and z ∈ R N such that x ≥ 0 , z ≥ 0 , and ) ( x ) ( A T I = b, z or A T x + z = b , and therefore, A T x ≤ b . 6 / 32

Linear Programming Let A ∈ R K × N , f ∈ R N , c ∈ R K . Primal problem: x ∈ R N f T x (P) max s. t. Ax ≤ c x ≥ 0 . Dual problem: λ ∈ R K c T λ (D) min A T λ ≥ f s. t. λ ≥ 0 . The Lagrangians for the two problems coincide (the nonnegativity constraints aside): L ( x, λ ) = f T x − λ T ( Ax − c ) = c T λ − x T ( A T λ − f ) . 7 / 32

Weak Duality Proposition 7.19 If x ∈ R N and λ ∈ R K are feasible for (P) and (D) , respectively, then f T x ≤ c T λ . Proof ▶ If x ∈ R N and λ ∈ R K are feasible for (P) and (D), then f T x ≤ ( A T λ ) T x = λ T ( Ax ) ≤ λ T c. x ∈ R N and ¯ λ ∈ R K are feasible and if f T ¯ x = c T ¯ Therefore, if ¯ λ , x and ¯ then ¯ λ are solutions to (P) and (D), respectively. 8 / 32

Strong Duality Proposition 7.20 Suppose that both (P) and (D) are feasible. Then both (P) and (D) have solutions, and max { f T x | Ax ≤ c, x ≥ 0 } = min { c T λ | A T λ ≥ f, λ ≥ 0 } . 9 / 32

Proof ▶ Suppose that (P) and (D) are feasible. We want to show that there exist x ∈ R N and λ ∈ R K such that Ax ≤ c , A T λ ≥ f , f T x ≥ c T λ , x ≥ 0 , and λ ≥ 0 , or  A O   c  ( x )  , x ≥ 0 , λ ≥ 0 . − A T O ≤ − f    λ − f T c T 0 ▶ By Proposition 7.18, this is equivalent to the condition that for all p ∈ R K , q ∈ R N , and r ∈ R ,   A O  ≥ 0 , p ≥ 0 , q ≥ 0 , r ≥ 0 ( p T q T ) − A T r O  − f T c T  c   ≥ 0 . ( p T q T ) ⇒ r − f  0 10 / 32

▶ That is, (1) A T p ≥ rf, Aq ≤ rc, p ≥ 0 , q ≥ 0 , r ≥ 0 implies (2) c T p − f T q ≥ 0 . We want to show that this holds whenever (P) and (D) are feasible. ▶ For r > 0 , (1) implies that q/r and p/r are feasible solutions to (P) and (D), so that we have c T p − f T q = r [ c T ( p/r ) − f T ( q/r )] ≥ 0 by Weak Duality. ▶ For r = 0 , let x and λ be feasible solutions to (P) and (D). From (1), we have c T p − f T q ≥ x T A T p − λ T Aq ≥ 0 . 11 / 32

Strong Duality Proposition 7.21 1. Suppose that (D) has a solution. Then (P) has a solution, and max { f T x | Ax ≤ c, x ≥ 0 } = min { c T λ | A T λ ≥ f, λ ≥ 0 } . 2. Suppose that (P) has a solution. Then (D) has a solution, and max { f T x | Ax ≤ c, x ≥ 0 } = min { c T λ | A T λ ≥ f, λ ≥ 0 } . 12 / 32

Proof ▶ Suppose that (D) has a solution. In light of Proposition 7.20, it suffices to show that (P) has a feasible solution. ▶ Let λ ∗ ∈ R K be a solution to (D). To apply Proposition 7.18, let z ∈ R K be such that A T z ≥ 0 and z ≥ 0 . ▶ Then λ ∗ + z ≥ 0 , and A T ( λ ∗ + z ) = A T λ ∗ + A T z ≥ f , which means that λ ∗ + z is feasible in (D). ▶ Therefore, by the optimality of λ ∗ , we have 0 ≤ c T ( λ ∗ + z ) − c T λ ∗ = c T z . ▶ By Proposition 7.18, there exists x ∈ R N such that Ax ≤ c and x ≥ 0 . 13 / 32

Variants of Farkas’ Lemma Proposition 7.22 (Gale’s Theorem/Fan’s Theorem) Let A ∈ R M × N and b ∈ R N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x ≤ b . 2. For any y ∈ R N , if y ≥ 0 and Ay = 0 , then b T y ≥ 0 . 14 / 32

Proof ▶ (1) ⇒ (2): Immediate. ▶ (2) ⇒ (1): Assume (2). ( A ) It implies that for all y ∈ R N , if y ≥ 0 and y ≥ 0 , then − A b T y ≥ 0 . ▶ By Proposition 7.18, there exist x ∈ R M and z ∈ R M such that x ≥ 0 , z ≥ 0 , and − A T ) ( x ) ( A T ≤ b, z or A T ( x − z ) ≤ b . 15 / 32

Variants of Farkas’ Lemma Proposition 7.23 (Gordan’s Theorem) Let A ∈ R M × N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x ≫ 0 . 2. For any y ∈ R N , if y ≥ 0 and Ay = 0 , then y = 0 . 16 / 32

Proof ▶ (1) ⇒ (2): Immediate. ▶ (2) ⇒ (1): Assume (2). It implies that for all y ∈ R N , if y ≥ 0 and ( − A ) y = 0 , then ( − 1 T ) y ≥ 0 . ▶ By Gale’s Theorem (Proposition 7.22), there exists x ∈ R M such that ( − A T ) x ≤ − 1 , or A T x ≥ 1 ≫ 0 . 17 / 32

Variants of Farkas’ Lemma Proposition 7.24 (Ville/von Neumann-Morgenstern I) Let A ∈ R M × N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x ≫ 0 and x ≫ 0 . 2. For any y ∈ R N , if y ≥ 0 and Ay ≤ 0 , then y = 0 . 18 / 32

Variants of Farkas’ Lemma ▶ In fact, “there exists x ∈ R M such that A T x ≫ 0 and x ≫ 0 ” is equivalent to “there exists x ∈ R M such that A T x ≫ 0 and x ≥ 0 ”. ▶ Given an x ≥ 0 in the latter, consider x + ε 1 for sufficiently small ε > 0 . Proposition 7.25 (Ville/von Neumann-Morgenstern II) Let A ∈ R M × N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x ≫ 0 and x ≥ 0 . 2. For any y ∈ R N , if y ≥ 0 and Ay ≤ 0 , then y = 0 . 19 / 32

Proof of Proposition 7.24 ▶ (1) ⇒ (2): Immediate. ▶ (2) ⇒ (1): Assume (2). ▶ For y ∈ R N and z ∈ R M , assume that ) ( y ) ( A I = 0 , y ≥ 0 , z ≥ 0 . z ▶ Then we have Ay + z = 0 , and hence Ay = − z ≤ 0 by z ≥ 0 . ▶ Then by (2), we have y = 0 , and hence z = 0 . ▶ By Gordan’s Theorem (Proposition 7.23), there exists x ∈ R M such that ( A T ) x ≫ 0 , I or A T x ≫ 0 and x ≫ 0 . 20 / 32

Variants of Farkas’ Lemma Proposition 7.26 Let A ∈ R M × N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x ≤ 0 and x ≫ 0 . 2. For any y ∈ R N , if y ≥ 0 and Ay ≥ 0 , then Ay = 0 . 21 / 32

Proof ▶ (1) ⇒ (2): Immediate. ▶ (2) ⇒ (1): Assume (2). Then there do not exist y ∈ R N and z ∈ R M such that Ay = z, 1 T z = 1 , y ≥ 0 , z ≥ 0 , or ( − A ) ( y ) ( 0 M ) I = , y ≥ 0 , z ≥ 0 . 0 T 1 T z 1 N ▶ Then by “not (1) ⇒ not (2)” in Farkas’ Lemma, there exist x ∈ R M and α ∈ R such that ( − A T ) ( x ) ) ( x ) 0 N ( 0 T ≥ 0 , 1 < 0 , M I α α 1 or A T x ≤ 0 , x ≥ − α 1 , α < 0 , as desired. 22 / 32

Variants of Farkas’ Lemma Proposition 7.27 (Stiemke’s Lemma) Let A ∈ R M × N . The following conditions are equivalent: 1. There exists x ∈ R M such that A T x = 0 and x ≫ 0 . 2. For any y ∈ R N , if Ay ≥ 0 , then Ay = 0 . 23 / 32