Introduction to LP and SDP Hierarchies Madhur Tulsiani Princeton - PowerPoint PPT Presentation

The Lasserre Hierarchy Start with a 0/1 integer quadratic program. Think “big" variables Z S = � i ∈ S z i . Associated psd matrix Y (moment matrix)   � � � Y S 1 , S 2 = E Z S 1 · Z S 2 = E z i   i ∈ S 1 ∪ S 2

The Lasserre Hierarchy Start with a 0/1 integer quadratic program. Think “big" variables Z S = � i ∈ S z i . Associated psd matrix Y (moment matrix)   �  = P [ All vars in S 1 ∪ S 2 are 1 ] � � Y S 1 , S 2 = E Z S 1 · Z S 2 = E z i  i ∈ S 1 ∪ S 2

The Lasserre Hierarchy Start with a 0/1 integer quadratic program. Think “big" variables Z S = � i ∈ S z i . Associated psd matrix Y (moment matrix)   �  = P [ All vars in S 1 ∪ S 2 are 1 ] � � Y S 1 , S 2 = E Z S 1 · Z S 2 = E z i  i ∈ S 1 ∪ S 2 ( Y � 0 ) + original constraints + consistency constraints.

The Lasserre hierarchy (constraints) Y is psd. (i.e. find vectors U S satisfying Y S 1 , S 2 = � U S 1 , U S 2 � )

The Lasserre hierarchy (constraints) Y is psd. (i.e. find vectors U S satisfying Y S 1 , S 2 = � U S 1 , U S 2 � ) Y S 1 , S 2 only depends on S 1 ∪ S 2 . ( Y S 1 , S 2 = P [ All vars in S 1 ∪ S 2 are 1 ])

The Lasserre hierarchy (constraints) Y is psd. (i.e. find vectors U S satisfying Y S 1 , S 2 = � U S 1 , U S 2 � ) Y S 1 , S 2 only depends on S 1 ∪ S 2 . ( Y S 1 , S 2 = P [ All vars in S 1 ∪ S 2 are 1 ]) Original quadratic constraints as inner products. SDP for Independent Set � 2 � � � maximize � U { i } i ∈ V � � subject to U { i } , U { j } = 0 ∀ ( i , j ) ∈ E � � � � U S 1 , U S 2 = U S 3 , U S 4 ∀ S 1 ∪ S 2 = S 3 ∪ S 4 � � U S 1 , U S 2 ∈ [ 0 , 1 ] ∀ S 1 , S 2

The “Mixed” hierarchy Motivated by [Raghavendra 08]. Used by [CS 08] for Hypergraph Independent Set. Captures what we actually know how to use about Lasserre solutions.

The “Mixed” hierarchy Motivated by [Raghavendra 08]. Used by [CS 08] for Hypergraph Independent Set. Captures what we actually know how to use about Lasserre solutions. Level r has Variables X S for | S | ≤ r and all Sherali-Adams constraints. Vectors U 0 , U 1 , . . . , U n satisfying � U i , U j � = X { i , j } , � U 0 , U i � = X { i } and | U 0 | = 1 .

Hands-on: Deriving some constraints

The triangle inequality | U i − U j | 2 + | U j − U k | 2 ≥ | U i − U k | 2 is equivalent to � U i − U j , U k − U j � ≥ 0

The triangle inequality | U i − U j | 2 + | U j − U k | 2 ≥ | U i − U k | 2 is equivalent to � U i − U j , U k − U j � ≥ 0 Mix ( 3 ) = ⇒ ∃ distribution on z i , z j , z k such that E [ z i · z j ] = � U i , U j � (and so on).

The triangle inequality | U i − U j | 2 + | U j − U k | 2 ≥ | U i − U k | 2 is equivalent to � U i − U j , U k − U j � ≥ 0 Mix ( 3 ) = ⇒ ∃ distribution on z i , z j , z k such that E [ z i · z j ] = � U i , U j � (and so on). For all integer solutions ( z i − z j ) · ( z k − z j ) ≥ 0.

The triangle inequality | U i − U j | 2 + | U j − U k | 2 ≥ | U i − U k | 2 is equivalent to � U i − U j , U k − U j � ≥ 0 Mix ( 3 ) = ⇒ ∃ distribution on z i , z j , z k such that E [ z i · z j ] = � U i , U j � (and so on). For all integer solutions ( z i − z j ) · ( z k − z j ) ≥ 0. � U i − U j , U k − U j � = E [( z i − z j ) · ( z k − z j )] ≥ 0 ∴

“Clique constraints” for Independent Set For every clique K in a graph, adding the constraint � x i ≤ 1 i ∈ K makes the independent set LP tight for perfect graphs.

“Clique constraints” for Independent Set For every clique K in a graph, adding the constraint � x i ≤ 1 i ∈ K makes the independent set LP tight for perfect graphs. Too many constraints, but all implied by one level of the mixed hierarchy.

“Clique constraints” for Independent Set For every clique K in a graph, adding the constraint � x i ≤ 1 i ∈ K makes the independent set LP tight for perfect graphs. Too many constraints, but all implied by one level of the mixed hierarchy. For i , j ∈ K , � U i , U j � = 0. Also, ∀ i � U 0 , U i � = | U i | 2 = x i . By Pythagoras, � 2 x 2 � U 0 , U i ≤ | U 0 | 2 = 1 � � i = ⇒ ≤ 1 . | U i | x i i ∈ K i ∈ B Derived by Lovász using the ϑ -function.

The Lovász-Schrijver Hierarchy

The Lovász-Schrijver Hierarchy Start with a 0/1 integer program and a relaxation P . Define tigher relaxation LS ( P ) .

The Lovász-Schrijver Hierarchy Start with a 0/1 integer program and a relaxation P . Define tigher relaxation LS ( P ) . Hope: Fractional ( x 1 , . . . , x n ) = E [( z 1 , . . . , z n )] for integral ( z 1 , . . . , z n )

The Lovász-Schrijver Hierarchy Start with a 0/1 integer program and a relaxation P . Define tigher relaxation LS ( P ) . Hope: Fractional ( x 1 , . . . , x n ) = E [( z 1 , . . . , z n )] for integral ( z 1 , . . . , z n ) Restriction: x = ( x 1 , . . . , x n ) ∈ LS ( P ) if ∃ Y satisfying (think Y ij = E [ z i z j ] = P [ z i ∧ z j ] ) Y = Y T Y ii = x i ∀ i Y i ∈ P , x − Y i ∈ P ∀ i 1 − x i x i Y � 0

The Lovász-Schrijver Hierarchy Start with a 0/1 integer program and a relaxation P . Define tigher relaxation LS ( P ) . Hope: Fractional ( x 1 , . . . , x n ) = E [( z 1 , . . . , z n )] for integral ( z 1 , . . . , z n ) Restriction: x = ( x 1 , . . . , x n ) ∈ LS ( P ) if ∃ Y satisfying (think Y ij = E [ z i z j ] = P [ z i ∧ z j ] ) Y = Y T Y ii = x i ∀ i Y i ∈ P , x − Y i ∈ P ∀ i 1 − x i x i Y � 0 Above is an LP (SDP) in n 2 + n variables.

Lovász-Schrijver in action r th level optimizes over distributions conditioned on r variables. 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3

Lovász-Schrijver in action r th level optimizes over distributions conditioned on r variables. 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3

Lovász-Schrijver in action r th level optimizes over distributions conditioned on r variables. 1 / 3 1 / 3 1 / 3 2 / 3 × 1 / 3 × 1 / 2 1 / 3 1 / 3 0 1 / 3 1 / 2 1 / 2 0 0 1 / 2 1 / 2 0 0 0 1

Lovász-Schrijver in action r th level optimizes over distributions conditioned on r variables. 1 / 3 1 / 3 1 / 3 2 / 3 × 1 / 3 × 1 / 2 1 / 3 1 / 3 0 1 / 3 1 / 2 1 / 2 0 0 1 / 2 1 / 2 0 0 0 1

Lovász-Schrijver in action r th level optimizes over distributions conditioned on r variables. 1 / 3 1 / 3 1 / 3 2 / 3 × 1 / 3 × 1 / 2 1 / 3 1 / 3 0 1 / 3 1 / 2 1 / 2 0 0 1 / 2 1 / 2 0 0 0 1 1 / 2 × 0 0 0 0 1 0

Lovász-Schrijver in action r th level optimizes over distributions conditioned on r variables. 1 / 3 1 / 3 1 / 3 2 / 3 × 1 / 3 × 1 / 2 1 / 3 1 / 3 0 1 / 3 1 / 2 1 / 2 0 0 1 / 2 1 / 2 0 0 0 1 1 / 2 × 1 / 2 × ? 0 ? ? 0 0 ? ? 0 1 0 0

And if you just woke up . . .

And if you just woke up . . . . . . . . . Las ( 2 ) Mix ( 2 ) Las ( 1 ) Mix ( 1 ) . . . . . . LS ( 2 ) SA ( 2 ) + . LS ( 1 ) . SA ( 1 ) . + LS ( 2 ) LS ( 1 )

And if you just woke up . . . U S . SA + U i . . . . . Las ( 2 ) Mix ( 2 ) Las ( 1 ) Mix ( 1 ) . . . . . . LS ( 2 ) X S Y � 0 SA ( 2 ) + . LS ( 1 ) . SA ( 1 ) . + LS ( 2 ) LS ( 1 ) Y i , x − Y i x i 1 − x i

Algorithmic Applications Many known LP/SDP relaxations captured by 2-3 levels. [Chlamtac 07]: Explicitly used level-3 Lasserre SDP for graph coloring. [CS 08]: Algorithms using Mixed and Lasserre hierarchies for hypergraph independent set (guarantee improves with more levels). [KKMN 10]: Hierarchies yield a PTAS for Knapsack. [BRS 11, GS 11]: Algorithms for Unique Games using n ǫ levels of Lassere.

Lower bound techniques Expansion in CSP instances (Proof Complexity)

Lower bound techniques Expansion in CSP instances (Proof Complexity) Reductions

Lower bound techniques Expansion in CSP instances (Proof Complexity) Reductions [ABLT 06, STT 07, dlVKM 07, CMM 09]: Distributions from local probabilistic processes. [Charikar 02, GMPT 07, BCGM 10]: Polynomial tensoring. [RS 09, KS 09]: Higher level distributions from level-1 vectors.

Integrality Gaps for Expanding CSPs

CSP Expansion MAX k-CSP: m constraints on k -tuples of ( n ) boolean variables. Satisfy maximum. e.g. MAX 3-XOR (linear equations mod 2) z 1 + z 2 + z 3 = 0 z 3 + z 4 + z 5 = 1 · · ·

CSP Expansion MAX k-CSP: m constraints on k -tuples of ( n ) boolean variables. Satisfy maximum. e.g. MAX 3-XOR (linear equations mod 2) z 1 + z 2 + z 3 = 0 z 3 + z 4 + z 5 = 1 · · · Expansion: Every set S of constraints involves at least β | S | variables (for | S | < α m ).

CSP Expansion MAX k-CSP: m constraints on k -tuples of ( n ) boolean variables. Satisfy maximum. e.g. MAX 3-XOR (linear equations mod 2) z 1 + z 2 + z 3 = 0 z 3 + z 4 + z 5 = 1 · · · Expansion: Every set S of constraints involves at least β | S | variables (for | S | < α m ). z 1 C 1 . . . . . . z n C m

CSP Expansion MAX k-CSP: m constraints on k -tuples of ( n ) boolean variables. Satisfy maximum. e.g. MAX 3-XOR (linear equations mod 2) z 1 + z 2 + z 3 = 0 z 3 + z 4 + z 5 = 1 · · · Expansion: Every set S of constraints involves at least β | S | variables (for | S | < α m ). z 1 C 1 . . . . . . z n C m

CSP Expansion MAX k-CSP: m constraints on k -tuples of ( n ) boolean variables. Satisfy maximum. e.g. MAX 3-XOR (linear equations mod 2) z 1 + z 2 + z 3 = 0 z 3 + z 4 + z 5 = 1 · · · Expansion: Every set S of constraints involves at least β | S | variables (for | S | < α m ). z 1 C 1 . . . . . . z n C m In fact, γ | S | variables appearing in only one constraint in S .

CSP Expansion MAX k-CSP: m constraints on k -tuples of ( n ) boolean variables. Satisfy maximum. e.g. MAX 3-XOR (linear equations mod 2) z 1 + z 2 + z 3 = 0 z 3 + z 4 + z 5 = 1 · · · Expansion: Every set S of constraints involves at least β | S | variables (for | S | < α m ). z 1 C 1 . . . . . . z n C m In fact, γ | S | variables appearing in only one constraint in S . Used extensively in proof complexity e.g. [BW01], [BGHMP03]. For LS + by [AAT04].

Sherali-Adams LP for CSPs X ( S ,α ) for | S | ≤ t , partial assignments α ∈ { 0 , 1 } S Variables: m � � C i ( α ) · X ( T i ,α ) maximize i = 1 α ∈{ 0 , 1 } Ti subject to X ( S ∪{ i } ,α ◦ 0 ) + X ( S ∪{ i } ,α ◦ 1 ) = X ( S ,α ) ∀ i / ∈ S X ( S ,α ) ≥ 0 X ( ∅ , ∅ ) = 1

Sherali-Adams LP for CSPs X ( S ,α ) for | S | ≤ t , partial assignments α ∈ { 0 , 1 } S Variables: m � � C i ( α ) · X ( T i ,α ) maximize i = 1 α ∈{ 0 , 1 } Ti subject to X ( S ∪{ i } ,α ◦ 0 ) + X ( S ∪{ i } ,α ◦ 1 ) = X ( S ,α ) ∀ i / ∈ S X ( S ,α ) ≥ 0 X ( ∅ , ∅ ) = 1 X ( S ,α ) ∼ P [ Vars in S assigned according to α ]

Sherali-Adams LP for CSPs X ( S ,α ) for | S | ≤ t , partial assignments α ∈ { 0 , 1 } S Variables: m � � C i ( α ) · X ( T i ,α ) maximize i = 1 α ∈{ 0 , 1 } Ti subject to X ( S ∪{ i } ,α ◦ 0 ) + X ( S ∪{ i } ,α ◦ 1 ) = X ( S ,α ) ∀ i / ∈ S X ( S ,α ) ≥ 0 X ( ∅ , ∅ ) = 1 X ( S ,α ) ∼ P [ Vars in S assigned according to α ] Need distributions D ( S ) such that D ( S 1 ) , D ( S 2 ) agree on S 1 ∩ S 2 .

Sherali-Adams LP for CSPs X ( S ,α ) for | S | ≤ t , partial assignments α ∈ { 0 , 1 } S Variables: m � � C i ( α ) · X ( T i ,α ) maximize i = 1 α ∈{ 0 , 1 } Ti subject to X ( S ∪{ i } ,α ◦ 0 ) + X ( S ∪{ i } ,α ◦ 1 ) = X ( S ,α ) ∀ i / ∈ S X ( S ,α ) ≥ 0 X ( ∅ , ∅ ) = 1 X ( S ,α ) ∼ P [ Vars in S assigned according to α ] Need distributions D ( S ) such that D ( S 1 ) , D ( S 2 ) agree on S 1 ∩ S 2 . Distributions should “locally look like" supported on satisfying assignments.

Local Satisfiability z 1 • Take γ = 0 . 9 z 2 C 1 • Can show any three 3-XOR constraints are z 3 simultaneously satisfiable. C 2 z 4 z 5 z 6 C 3

Local Satisfiability z 1 • Take γ = 0 . 9 z 2 C 1 • Can show any three 3-XOR constraints are z 3 simultaneously satisfiable. C 2 z 4 z 5 z 6 C 3 E z 1 ... z 6 [ C 1 ( z 1 , z 2 , z 3 ) · C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 )]

Local Satisfiability z 1 • Take γ = 0 . 9 z 2 C 1 • Can show any three 3-XOR constraints are z 3 simultaneously satisfiable. z 4 C 2 z 5 z 6 C 3 E z 1 ... z 6 [ C 1 ( z 1 , z 2 , z 3 ) · C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 )] = E z 2 ... z 6 [ C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 ) · E z 1 [ C 1 ( z 1 , z 2 , z 3 )]]

Local Satisfiability z 1 • Take γ = 0 . 9 z 2 C 1 • Can show any three 3-XOR constraints are z 3 simultaneously satisfiable. z 4 C 2 z 5 z 6 C 3 E z 1 ... z 6 [ C 1 ( z 1 , z 2 , z 3 ) · C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 )] = E z 2 ... z 6 [ C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 ) · E z 1 [ C 1 ( z 1 , z 2 , z 3 )]] = E z 4 , z 5 , z 6 [ C 3 ( z 4 , z 5 , z 6 ) · E z 3 [ C 2 ( z 3 , z 4 , z 5 )] · ( 1 / 2 )]

Local Satisfiability z 1 • Take γ = 0 . 9 z 2 C 1 • Can show any three 3-XOR constraints are z 3 simultaneously satisfiable. z 4 C 2 z 5 z 6 C 3 E z 1 ... z 6 [ C 1 ( z 1 , z 2 , z 3 ) · C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 )] = E z 2 ... z 6 [ C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 ) · E z 1 [ C 1 ( z 1 , z 2 , z 3 )]] = E z 4 , z 5 , z 6 [ C 3 ( z 4 , z 5 , z 6 ) · E z 3 [ C 2 ( z 3 , z 4 , z 5 )] · ( 1 / 2 )] = 1 / 8

Local Satisfiability z 1 • Take γ = 0 . 9 z 2 C 1 • Can show any three 3-XOR constraints are z 3 simultaneously satisfiable. z 4 C 2 • Can take γ ≈ ( k − 2 ) and any α n constraints. z 5 • Just require E [ C ( z 1 , . . . , z k )] over any k − 2 z 6 C 3 vars to be constant. E z 1 ... z 6 [ C 1 ( z 1 , z 2 , z 3 ) · C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 )] = E z 2 ... z 6 [ C 2 ( z 3 , z 4 , z 5 ) · C 3 ( z 4 , z 5 , z 6 ) · E z 1 [ C 1 ( z 1 , z 2 , z 3 )]] = E z 4 , z 5 , z 6 [ C 3 ( z 4 , z 5 , z 6 ) · E z 3 [ C 2 ( z 3 , z 4 , z 5 )] · ( 1 / 2 )] = 1 / 8

Obtaining integrality gaps for CSPs [BGMT 09] z 1 C 1 . . . . . . z n C m Want to define distribution D ( S ) for set S of variables.

Obtaining integrality gaps for CSPs [BGMT 09] z 1 C 1 . . . . . . z n C m Want to define distribution D ( S ) for set S of variables.

Obtaining integrality gaps for CSPs [BGMT 09] z 1 C 1 . . . . . . z n C m Want to define distribution D ( S ) for set S of variables.

Obtaining integrality gaps for CSPs [BGMT 09] z 1 C 1 . . . . . . z n C m Want to define distribution D ( S ) for set S of variables. Find set of constraints C such that G − C − S remains expanding. D ( S ) = uniform over assignments satisfying C

Obtaining integrality gaps for CSPs [BGMT 09] z 1 C 1 . . . . . . z n C m Want to define distribution D ( S ) for set S of variables. Find set of constraints C such that G − C − S remains expanding. D ( S ) = uniform over assignments satisfying C Remaining constraints “independent" of this assignment. Gives optimal integrality gaps for Ω( n ) levels in the mixed hierarchy.

Vectors for Linear CSPs

A “new look” Lasserre Start with a {− 1 , 1 } quadratic integer program. ( z 1 , . . . , z n ) → (( − 1 ) z 1 , . . . , ( − 1 ) z n )

A “new look” Lasserre Start with a {− 1 , 1 } quadratic integer program. ( z 1 , . . . , z n ) → (( − 1 ) z 1 , . . . , ( − 1 ) z n ) Define big variables ˜ i ∈ S ( − 1 ) z i . Z S = �

A “new look” Lasserre Start with a {− 1 , 1 } quadratic integer program. ( z 1 , . . . , z n ) → (( − 1 ) z 1 , . . . , ( − 1 ) z n ) Define big variables ˜ i ∈ S ( − 1 ) z i . Z S = � Consider the psd matrix ˜ Y   � � ˜ Z S 1 · ˜ ˜ � ( − 1 ) z i Y S 1 , S 2 = E Z S 2 = E   i ∈ S 1 ∆ S 2

A “new look” Lasserre Start with a {− 1 , 1 } quadratic integer program. ( z 1 , . . . , z n ) → (( − 1 ) z 1 , . . . , ( − 1 ) z n ) Define big variables ˜ i ∈ S ( − 1 ) z i . Z S = � Consider the psd matrix ˜ Y   � � ˜ Z S 1 · ˜ ˜ � ( − 1 ) z i Y S 1 , S 2 = E Z S 2 = E   i ∈ S 1 ∆ S 2 Write program for inner products of vectors W S s.t. ˜ Y S 1 , S 2 = � W S 1 , W S 2 �

Gaps for 3-XOR SDP for MAX 3-XOR 1 + ( − 1 ) b i � � W { i 1 , i 2 , i 3 } , W ∅ � maximize 2 C i ≡ ( z i 1 + z i 2 + z i 3 = b i ) � W S 1 , W S 2 � = � W S 3 , W S 4 � ∀ S 1 ∆ S 2 = S 3 ∆ S 4 subject to | W S | = 1 ∀ S , | S | ≤ r