introduction to the low degree polynomial method alex wein
play

Introduction to the Low-Degree Polynomial Method Alex Wein Courant - PowerPoint PPT Presentation

Nov 28, 2023 •456 likes •2.11k views

Introduction to the Low-Degree Polynomial Method Alex Wein Courant Institute, New York University 1 / 31 Part I: Why Low-Degree Polynomials? 2 / 31 Problems in High-Dimensional Statistics Example: finding a large clique in a random graph 3

  1. The Low-Degree Polynomial Method Claim: low-degree polynomials provide a unified explanation of information-computation gaps in detection/recovery/optimization For all of these problems... planted clique, sparse PCA, community detection, tensor PCA, planted CSPs, spiked Wigner/Wishart, planted submatrix, planted dense subgraph, p-spin optimization, max independent set ...it is the case that ◮ the best known poly-time algorithms are low-degree (spectral/AMP/local) ◮ low-degree polynomials fail in the “hard” regime “Low-degree conjecture” (informal): low-degree polynomials are as powerful as all poly-time algorithms for “natural” high-dimensional problems [Hopkins ’18] 6 / 31

  2. Overview This talk: techniques to prove that all low-degree polynomials fail 7 / 31

  3. Overview This talk: techniques to prove that all low-degree polynomials fail ◮ Gives evidence for computational hardness 7 / 31

  4. Overview This talk: techniques to prove that all low-degree polynomials fail ◮ Gives evidence for computational hardness Settings: ◮ Detection [Hopkins, Steurer ’17] [Hopkins, Kothari, Potechin, Raghavendra, Schramm, Steurer ’17] [Hopkins ’18] (PhD thesis) [Kunisky, W., Bandeira ’19] (survey) ◮ Recovery [Schramm, W. ’20] ◮ Optimization [Gamarnik, Jagannath, W. ’20] 7 / 31

  5. Part II: Detection 8 / 31

  6. Detection (e.g. [Hopkins, Steurer ’17] ) Goal: hypothesis test with error probability o (1) between: ◮ Null model Y ∼ Q n e.g. G ( n , 1 / 2) ◮ Planted model Y ∼ P n e.g. G ( n , 1 / 2) ∪ { random k -clique } 9 / 31

  7. Detection (e.g. [Hopkins, Steurer ’17] ) Goal: hypothesis test with error probability o (1) between: ◮ Null model Y ∼ Q n e.g. G ( n , 1 / 2) ◮ Planted model Y ∼ P n e.g. G ( n , 1 / 2) ∪ { random k -clique } Look for a degree- D polynomial f : R n × n → R that distinguishes P from Q 9 / 31

  8. Detection (e.g. [Hopkins, Steurer ’17] ) Goal: hypothesis test with error probability o (1) between: ◮ Null model Y ∼ Q n e.g. G ( n , 1 / 2) ◮ Planted model Y ∼ P n e.g. G ( n , 1 / 2) ∪ { random k -clique } Look for a degree- D polynomial f : R n × n → R that distinguishes P from Q ◮ f ( Y ) is “big” when Y ∼ P and “small” when Y ∼ Q 9 / 31

  9. Detection (e.g. [Hopkins, Steurer ’17] ) Goal: hypothesis test with error probability o (1) between: ◮ Null model Y ∼ Q n e.g. G ( n , 1 / 2) ◮ Planted model Y ∼ P n e.g. G ( n , 1 / 2) ∪ { random k -clique } Look for a degree- D polynomial f : R n × n → R that distinguishes P from Q ◮ f ( Y ) is “big” when Y ∼ P and “small” when Y ∼ Q Compute “advantage”: E Y ∼ P [ f ( Y )] mean in P Adv ≤ D := max fluctuations in Q � E Y ∼ Q [ f ( Y ) 2 ] f deg D 9 / 31

  10. Detection (e.g. [Hopkins, Steurer ’17] ) Goal: hypothesis test with error probability o (1) between: ◮ Null model Y ∼ Q n e.g. G ( n , 1 / 2) ◮ Planted model Y ∼ P n e.g. G ( n , 1 / 2) ∪ { random k -clique } Look for a degree- D polynomial f : R n × n → R that distinguishes P from Q ◮ f ( Y ) is “big” when Y ∼ P and “small” when Y ∼ Q Compute “advantage”: E Y ∼ P [ f ( Y )] mean in P Adv ≤ D := max fluctuations in Q � E Y ∼ Q [ f ( Y ) 2 ] f deg D � ω (1) “degree- D polynomial succeed” = O (1) “degree- D polynomials fail” 9 / 31

  11. Detection (e.g. [Hopkins, Steurer ’17] ) Prototypical result (planted clique): 10 / 31

  12. Detection (e.g. [Hopkins, Steurer ’17] ) Prototypical result (planted clique): Theorem [BHKKMP16,Hop18] : For a planted k -clique in G ( n , 1 / 2), 10 / 31

  13. Detection (e.g. [Hopkins, Steurer ’17] ) Prototypical result (planted clique): Theorem [BHKKMP16,Hop18] : For a planted k -clique in G ( n , 1 / 2), ◮ if k = Ω( √ n ) then Adv ≤ D = ω (1) for some D = O (log n ) low-degree polynomials succeed when k � √ n 10 / 31

  14. Detection (e.g. [Hopkins, Steurer ’17] ) Prototypical result (planted clique): Theorem [BHKKMP16,Hop18] : For a planted k -clique in G ( n , 1 / 2), ◮ if k = Ω( √ n ) then Adv ≤ D = ω (1) for some D = O (log n ) low-degree polynomials succeed when k � √ n ◮ if k = O ( n 1 / 2 − ǫ ) then Adv ≤ D = O (1) for any D = O (log n ) low-degree polynomials fail when k ≪ √ n 10 / 31

  15. Detection (e.g. [Hopkins, Steurer ’17] ) Prototypical result (planted clique): Theorem [BHKKMP16,Hop18] : For a planted k -clique in G ( n , 1 / 2), ◮ if k = Ω( √ n ) then Adv ≤ D = ω (1) for some D = O (log n ) low-degree polynomials succeed when k � √ n ◮ if k = O ( n 1 / 2 − ǫ ) then Adv ≤ D = O (1) for any D = O (log n ) low-degree polynomials fail when k ≪ √ n Sometimes can rule out polynomials of degree D = n δ 10 / 31

  16. Detection (e.g. [Hopkins, Steurer ’17] ) Prototypical result (planted clique): Theorem [BHKKMP16,Hop18] : For a planted k -clique in G ( n , 1 / 2), ◮ if k = Ω( √ n ) then Adv ≤ D = ω (1) for some D = O (log n ) low-degree polynomials succeed when k � √ n ◮ if k = O ( n 1 / 2 − ǫ ) then Adv ≤ D = O (1) for any D = O (log n ) low-degree polynomials fail when k ≪ √ n Sometimes can rule out polynomials of degree D = n δ Extended low-degree conjecture [Hopkins ’18] : degree- D polynomials ⇔ n ˜ Θ( D ) -time algorithms D = n δ exp( n δ ± o (1) ) time ⇔ 10 / 31

  17. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D 11 / 31

  18. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) 11 / 31

  19. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i 11 / 31

  20. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T 11 / 31

  21. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T Numerator: Y ∼ P [ f ( Y )] E 11 / 31

  22. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] Numerator: Y ∼ P [ f ( Y )] = E f S E | S |≤ D 11 / 31

  23. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D 11 / 31

  24. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] E Denominator: 11 / 31

  25. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ E Denominator: (orthonormality) S | S |≤ D 11 / 31

  26. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ S = � ˆ f � 2 E Denominator: (orthonormality) | S |≤ D 11 / 31

  27. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ S = � ˆ f � 2 E Denominator: (orthonormality) | S |≤ D � ˆ f , c � Adv ≤ D = max � ˆ ˆ f � f 11 / 31

  28. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ S = � ˆ f � 2 E Denominator: (orthonormality) | S |≤ D � ˆ f , c � Adv ≤ D = max � ˆ ˆ f � f f ∗ = c Optimizer: ˆ 11 / 31

  29. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ S = � ˆ f � 2 E Denominator: (orthonormality) | S |≤ D � ˆ f , c � = � c , c � Adv ≤ D = max � ˆ � c � ˆ f � f f ∗ = c Optimizer: ˆ 11 / 31

  30. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ S = � ˆ f � 2 E Denominator: (orthonormality) | S |≤ D � ˆ f , c � = � c , c � � c � = � c � Adv ≤ D = max � ˆ ˆ f � f f ∗ = c Optimizer: ˆ 11 / 31

  31. Detection (e.g. [Hopkins, Steurer ’17] ) E Y ∼ P [ f ( Y )] Goal: compute Adv ≤ D := max � E Y ∼ Q [ f ( Y ) 2 ] f deg D Suppose Q is i.i.d. Unif ( ± 1) Y S := � | S |≤ D ˆ f S Y S Write f ( Y ) = � S ⊆ [ m ] i ∈ S Y i { Y S } S ⊆ [ m ] are orthonormal: E Y ∼ Q [ Y S Y T ] = ✶ S = T � ˆ Y ∼ P [ Y S ] =: � ˆ Numerator: Y ∼ P [ f ( Y )] = E f S E f , c � | S |≤ D Y ∼ Q [ f ( Y ) 2 ] = � f 2 ˆ S = � ˆ f � 2 E Denominator: (orthonormality) | S |≤ D � � 2 � ˆ f , c � = � c , c � � � � � c � = � c � = � Y ∼ P [ Y S ] Adv ≤ D = max E � ˆ � ˆ f � f | S |≤ D f ∗ = c Optimizer: ˆ 11 / 31

  32. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: 12 / 31

  33. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) 12 / 31

  34. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) ◮ Best degree- D test (maximizer of Adv ≤ D ) is f ∗ = L ≤ D := projection of L onto deg- D subspace 12 / 31

  35. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) ◮ Best degree- D test (maximizer of Adv ≤ D ) is f ∗ = L ≤ D := projection of L onto deg- D subspace orthogonal projection w.r.t. � f , g � := E Y ∼ Q [ f ( Y ) g ( Y )] 12 / 31

  36. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) ◮ Best degree- D test (maximizer of Adv ≤ D ) is f ∗ = L ≤ D := projection of L onto deg- D subspace orthogonal projection w.r.t. � f , g � := E Y ∼ Q [ f ( Y ) g ( Y )] “low-degree likelihood ratio” 12 / 31

  37. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) ◮ Best degree- D test (maximizer of Adv ≤ D ) is f ∗ = L ≤ D := projection of L onto deg- D subspace orthogonal projection w.r.t. � f , g � := E Y ∼ Q [ f ( Y ) g ( Y )] “low-degree likelihood ratio” ◮ Adv ≤ D = � L ≤ D � � Y ∼ Q [ f ( Y ) 2 ] � f � := � f , f � = E 12 / 31

  38. ✶ Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) ◮ Best degree- D test (maximizer of Adv ≤ D ) is f ∗ = L ≤ D := projection of L onto deg- D subspace orthogonal projection w.r.t. � f , g � := E Y ∼ Q [ f ( Y ) g ( Y )] “low-degree likelihood ratio” ◮ Adv ≤ D = � L ≤ D � � Y ∼ Q [ f ( Y ) 2 ] � f � := � f , f � = E “norm of low-degree likelihood ratio” 12 / 31

  39. Detection (e.g. [Hopkins, Steurer ’17] ) Remarks: ◮ Best test is likelihood ratio (Neyman-Pearson lemma) L ( Y ) = d P d Q ( Y ) ◮ Best degree- D test (maximizer of Adv ≤ D ) is f ∗ = L ≤ D := projection of L onto deg- D subspace orthogonal projection w.r.t. � f , g � := E Y ∼ Q [ f ( Y ) g ( Y )] “low-degree likelihood ratio” ◮ Adv ≤ D = � L ≤ D � � Y ∼ Q [ f ( Y ) 2 ] � f � := � f , f � = E “norm of low-degree likelihood ratio” Proof: ˆ Y ∼ Q [ L ( Y ) Y S ] = E Y ∼ P [ Y S ] f ∗ ˆ Y ∼ P [ Y S ] ✶ | S |≤ D S = E L S = E 12 / 31

  40. Detection (e.g. [Hopkins, Steurer ’17] ) User-friendly results: 13 / 31

  41. Detection (e.g. [Hopkins, Steurer ’17] ) User-friendly results: ◮ Additive Gaussian model: P : Y = X + Z Q : Y = Z vs 13 / 31

  42. Detection (e.g. [Hopkins, Steurer ’17] ) User-friendly results: ◮ Additive Gaussian model: P : Y = X + Z Q : Y = Z vs D 1 � Adv 2 X , X ′ � X , X ′ � d ≤ D = d ! E d =0 13 / 31

  43. Detection (e.g. [Hopkins, Steurer ’17] ) User-friendly results: ◮ Additive Gaussian model: P : Y = X + Z Q : Y = Z vs D 1 � Adv 2 X , X ′ � X , X ′ � d ≤ D = d ! E d =0 ◮ Rademacher model Y ∈ {± 1 } m : P : E [ Y | X ] = X Q : E [ Y ] = 0 vs 13 / 31

  44. Detection (e.g. [Hopkins, Steurer ’17] ) User-friendly results: ◮ Additive Gaussian model: P : Y = X + Z Q : Y = Z vs D 1 � Adv 2 X , X ′ � X , X ′ � d ≤ D = d ! E d =0 ◮ Rademacher model Y ∈ {± 1 } m : P : E [ Y | X ] = X Q : E [ Y ] = 0 vs D 1 � Adv 2 X , X ′ � X , X ′ � d ≤ D ≤ d ! E d =0 13 / 31

  45. Detection (e.g. [Hopkins, Steurer ’17] ) Recap (detection): 14 / 31

  46. Detection (e.g. [Hopkins, Steurer ’17] ) Recap (detection): ◮ Given P , Q , can compute (via linear algebra) E Y ∼ P [ f ( Y )] Adv ≤ D = � L ≤ D � = max � E Y ∼ Q [ f ( Y ) 2 ] f deg D 14 / 31

  47. Detection (e.g. [Hopkins, Steurer ’17] ) Recap (detection): ◮ Given P , Q , can compute (via linear algebra) E Y ∼ P [ f ( Y )] Adv ≤ D = � L ≤ D � = max � E Y ∼ Q [ f ( Y ) 2 ] f deg D ◮ Need to know orthogonal polynomials w.r.t. Q ◮ Possible when Q has independent coordinates 14 / 31

  48. Detection (e.g. [Hopkins, Steurer ’17] ) Recap (detection): ◮ Given P , Q , can compute (via linear algebra) E Y ∼ P [ f ( Y )] Adv ≤ D = � L ≤ D � = max � E Y ∼ Q [ f ( Y ) 2 ] f deg D ◮ Need to know orthogonal polynomials w.r.t. Q ◮ Possible when Q has independent coordinates ◮ To predict computational complexity: for D ≈ log n , � ω (1) ⇒ “easy” Adv ≤ D = ⇒ “hard” O (1) 14 / 31

  49. Detection (e.g. [Hopkins, Steurer ’17] ) Recap (detection): ◮ Given P , Q , can compute (via linear algebra) E Y ∼ P [ f ( Y )] Adv ≤ D = � L ≤ D � = max � E Y ∼ Q [ f ( Y ) 2 ] f deg D ◮ Need to know orthogonal polynomials w.r.t. Q ◮ Possible when Q has independent coordinates ◮ To predict computational complexity: for D ≈ log n , � ω (1) ⇒ “easy” Adv ≤ D = ⇒ “hard” O (1) ◮ These predictions are “correct” for: planted clique, sparse PCA, community detection, tensor PCA, spiked Wigner/Wishart, ... [BHKKMP16,HS17,HKPRSS17,Hop18,BK W 19,K W B19,DK W B19] 14 / 31

  50. Part III: Recovery 15 / 31

  51. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) 16 / 31

  52. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Detection : distinguish P : Y = X + Z vs Q : Y = Z w.h.p. Recovery : given Y ∼ P , recover v 16 / 31

  53. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Detection : distinguish P : Y = X + Z vs Q : Y = Z w.h.p. Recovery : given Y ∼ P , recover v If you can recover then you can detect (poly-time reduction) 16 / 31

  54. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Detection : distinguish P : Y = X + Z vs Q : Y = Z w.h.p. Recovery : given Y ∼ P , recover v If you can recover then you can detect (poly-time reduction) ◮ How: run recovery algorithm to get ˆ v ∈ { 0 , 1 } n ; check ˆ v ⊤ Y ˆ v 16 / 31

  55. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Detection : distinguish P : Y = X + Z vs Q : Y = Z w.h.p. Recovery : given Y ∼ P , recover v If you can recover then you can detect (poly-time reduction) ◮ How: run recovery algorithm to get ˆ v ∈ { 0 , 1 } n ; check ˆ v ⊤ Y ˆ v So if Adv ≤ D = O (1), this suggests recovery is hard 16 / 31

  56. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Detection : distinguish P : Y = X + Z vs Q : Y = Z w.h.p. Recovery : given Y ∼ P , recover v If you can recover then you can detect (poly-time reduction) ◮ How: run recovery algorithm to get ˆ v ∈ { 0 , 1 } n ; check ˆ v ⊤ Y ˆ v So if Adv ≤ D = O (1), this suggests recovery is hard But planted submatrix has a detection-recovery gap 16 / 31

  57. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Detection : distinguish P : Y = X + Z vs Q : Y = Z w.h.p. Recovery : given Y ∼ P , recover v If you can recover then you can detect (poly-time reduction) ◮ How: run recovery algorithm to get ˆ v ∈ { 0 , 1 } n ; check ˆ v ⊤ Y ˆ v So if Adv ≤ D = O (1), this suggests recovery is hard But planted submatrix has a detection-recovery gap How to show hardness of recovery when detection is easy? 16 / 31

  58. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) 17 / 31

  59. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Goal: given Y , estimate v 1 via polynomial f : R n × n → R 17 / 31

  60. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Goal: given Y , estimate v 1 via polynomial f : R n × n → R Low-degree minimum mean squared error: f deg D E ( f ( Y ) − v 1 ) 2 MMSE ≤ D = min 17 / 31

  61. Recovery [Schramm, W. ’20] Example (planted submatrix): observe n × n matrix Y = X + Z ◮ Signal: X = λ vv ⊤ v i ∼ Bernoulli ( ρ ) λ > 0 ◮ Noise: Z i.i.d. N (0 , 1) Goal: given Y , estimate v 1 via polynomial f : R n × n → R Low-degree minimum mean squared error: f deg D E ( f ( Y ) − v 1 ) 2 MMSE ≤ D = min Equivalent to low-degree maximum correlation: E [ f ( Y ) · v 1 ] Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D 1 ] − Corr 2 Fact: MMSE ≤ D = E [ v 2 ≤ D 17 / 31

  62. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D 18 / 31

  63. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? 18 / 31

  64. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D 18 / 31

  65. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D Numerator: E [ f ( Y ) · v 1 ] 18 / 31

  66. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D f S E [ Y S · v 1 ] ˆ � Numerator: E [ f ( Y ) · v 1 ] = | S |≤ D 18 / 31

  67. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D f S E [ Y S · v 1 ] =: � ˆ ˆ � Numerator: E [ f ( Y ) · v 1 ] = f , c � | S |≤ D 18 / 31

  68. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D f S E [ Y S · v 1 ] =: � ˆ ˆ � Numerator: E [ f ( Y ) · v 1 ] = f , c � | S |≤ D Denominator: E [ f ( Y ) 2 ] 18 / 31

  69. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D f S E [ Y S · v 1 ] =: � ˆ ˆ � Numerator: E [ f ( Y ) · v 1 ] = f , c � | S |≤ D f T E [ Y S · Y T ] � f S ˆ ˆ Denominator: E [ f ( Y ) 2 ] = S , T 18 / 31

  70. Recovery [Schramm, W. ’20] E [ f ( Y ) · v 1 ] For hardness, want upper bound on Corr ≤ D = max � E [ f ( Y ) 2 ] f deg D Same proof as detection? � f S Y S ˆ f = | S |≤ D f S E [ Y S · v 1 ] =: � ˆ ˆ � Numerator: E [ f ( Y ) · v 1 ] = f , c � | S |≤ D f T E [ Y S · Y T ] = ˆ � f S ˆ ˆ f ⊤ M ˆ Denominator: E [ f ( Y ) 2 ] = f S , T 18 / 31

Recommend

Computational Barriers to Estimation from Low-Degree Polynomials Alex Wein Courant Institute,

Computational Barriers to Estimation from Low-Degree Polynomials Alex Wein Courant Institute,

Computational Barriers to Estimation from Low-Degree Polynomials Alex Wein Courant Institute, New York University Joint work with: Tselil Schramm Stanford 1 / 23 Part I: Why Low-Degree Polynomials? 2 / 23 Problems in High-Dimensional

1.5k views • 139 slides
Hardness of Certification for Constrained PCA Alex Wein Courant Institute, NYU Joint work with:

Hardness of Certification for Constrained PCA Alex Wein Courant Institute, NYU Joint work with:

Hardness of Certification for Constrained PCA Alex Wein Courant Institute, NYU Joint work with: Afonso Bandeira (NYU) Tim Kunisky (NYU) 1 / 19 Part I: Statistical-to-Computational Gaps and the Low-Degree Method 2 / 19

1.25k views • 123 slides
The Power of Low-Degree Polynomials for Solving Statistical Problems Alex Wein Courant

The Power of Low-Degree Polynomials for Solving Statistical Problems Alex Wein Courant

The Power of Low-Degree Polynomials for Solving Statistical Problems Alex Wein Courant Institute, New York University Based on joint work with: David Gamarnik (MIT) Aukosh Jagannath (Waterloo) Tselil Schramm (Stanford) 1 / 10 Problems in

835 views • 51 slides
Low-Degree Hardness of Random Optimization Problems Alex Wein Courant Institute, New York

Low-Degree Hardness of Random Optimization Problems Alex Wein Courant Institute, New York

Low-Degree Hardness of Random Optimization Problems Alex Wein Courant Institute, New York University Joint work with: David Gamarnik Aukosh Jagannath MIT Waterloo 1 / 18 Random Optimization Problems Examples: 2 / 18 Random Optimization

1.22k views • 98 slides
Understanding Statistical-vs-Computational Tradeoffs via the Low-Degree Likelihood Ratio Alex

Understanding Statistical-vs-Computational Tradeoffs via the Low-Degree Likelihood Ratio Alex

Understanding Statistical-vs-Computational Tradeoffs via the Low-Degree Likelihood Ratio Alex Wein Courant Institute, NYU Joint work with: Afonso Bandeira Yunzi Ding Tim Kunisky (ETH Zurich) (NYU) (NYU) 1 / 27 Motivation 2 / 27

1.64k views • 162 slides
Statistical Estimation in the Presence of Group Actions Alex Wein MIT Mathematics 1 / 39 In

Statistical Estimation in the Presence of Group Actions Alex Wein MIT Mathematics 1 / 39 In

Statistical Estimation in the Presence of Group Actions Alex Wein MIT Mathematics 1 / 39 In memoriam Amelia Perry 1991 2018 2 / 39 My research interests Statistical and computational limits of average-case inference problems

2.12k views • 193 slides
Steins method and Malliavin calculus Ciprian A. Tudor Universit e de Lille 1 International

Steins method and Malliavin calculus Ciprian A. Tudor Universit e de Lille 1 International

Plan Steins method for normal approximation Applications Other target distributions : invariant measures of diffusions Examples Fouth Moment Theorem The case when the diffusion coefficient is a polynomial of second degree Steins method

1.04k views • 74 slides
Estimation in the Presence of Group Actions Alex Wein MIT Mathematics 1 / 28 Joint work with:

Estimation in the Presence of Group Actions Alex Wein MIT Mathematics 1 / 28 Joint work with:

Estimation in the Presence of Group Actions Alex Wein MIT Mathematics 1 / 28 Joint work with: Amelia Perry 1991 2018 2 / 28 Joint work with: Afonso Bandeira Ben Blum-Smith Jonathan Weed Ankur Moitra 3 / 28 Group actions G

1.15k views • 28 slides
The Kikuchi Hierarchy and Tensor PCA Alex Wein Courant Institute, NYU Joint work with: Ahmed El

The Kikuchi Hierarchy and Tensor PCA Alex Wein Courant Institute, NYU Joint work with: Ahmed El

The Kikuchi Hierarchy and Tensor PCA Alex Wein Courant Institute, NYU Joint work with: Ahmed El Alaoui Cris Moore Stanford Santa Fe Institute 1 / 19 Statistical Physics of Inference High-dimensional inference problems: compressed

1.43k views • 122 slides
Polynomial-Time Problems Lecture 4: The polynomial method Part II: All-pairs shortest paths

Polynomial-Time Problems Lecture 4: The polynomial method Part II: All-pairs shortest paths

Complexity Theory of Polynomial-Time Problems Lecture 4: The polynomial method Part II: All-pairs shortest paths Sebastian Krinninger Overview on APSP Algorithms Floyd-Warshall algorithm: ( 3 ) Inserts one node at a time

687 views • 42 slides
Polynomial-Time Problems Lecture 3: The polynomial method Part I: Orthogonal Vectors Sebastian

Polynomial-Time Problems Lecture 3: The polynomial method Part I: Orthogonal Vectors Sebastian

Complexity Theory of Polynomial-Time Problems Lecture 3: The polynomial method Part I: Orthogonal Vectors Sebastian Krinninger Organization of lecture No lecture on 26.05. (State holiday) 2 nd exercise sheet: Next week Tutorials:

380 views • 34 slides
Spectral Methods from Tensor Networks Alex Wein Courant Institute, NYU Joint work with Ankur

Spectral Methods from Tensor Networks Alex Wein Courant Institute, NYU Joint work with Ankur

Spectral Methods from Tensor Networks Alex Wein Courant Institute, NYU Joint work with Ankur Moitra (MIT) 1 / 19 Outline Tensors 2 / 19 Outline Tensors Statistical problems involving tensors 2 / 19 Outline Tensors

820 views • 49 slides
Polynomial Equations and Inequal- ities We will consider polynomial equations first and assume

Polynomial Equations and Inequal- ities We will consider polynomial equations first and assume

Polynomial Equations and Inequal- ities We will consider polynomial equations first and assume each equation has been put in the form p ( x ) = 0, where p ( x ) is a polynomial of degree n . Most polynomials we consider will have in- teger

176 views • 6 slides
Interpolation polynomial * Means finding an approximate value to a function as a polynomial of

Interpolation polynomial * Means finding an approximate value to a function as a polynomial of

Interpolation polynomial * Means finding an approximate value to a function as a polynomial of degree n using n+1 points. Methods of solution 1- Quadratic Lagrange x x x x 1 2 0 f x y y y 0 1 2 P

154 views • 13 slides
Cubic spline interpolation High-degree polynomial fitting has strong oscillations. Can we get a

Cubic spline interpolation High-degree polynomial fitting has strong oscillations. Can we get a

Cubic spline interpolation High-degree polynomial fitting has strong oscillations. Can we get a piecewise low degree polynomial interpolation instead? S ( x ) S n 2 S n 1 S j S 1 S j 1 S 0 S j ( x j 1 ) f ( x j 1 )

449 views • 7 slides
Power law method for finding soliton solutions of the 2+1 Ricci flow model Radu Constantinescu,

Power law method for finding soliton solutions of the 2+1 Ricci flow model Radu Constantinescu,

Introduction Integrability of the nonlinear differential equations Symmetries and their applications in nonlinear dynamics The auxiliary equation method The polynomial expansion method The example of the KdV The example of the Ricci 2 D equation

406 views • 28 slides
Polynomials and Fast Fourier Transform (FFT) Polynomials n-1 a i x i a polynomial of degree n-1

Polynomials and Fast Fourier Transform (FFT) Polynomials n-1 a i x i a polynomial of degree n-1

Polynomials and Fast Fourier Transform (FFT) Polynomials n-1 a i x i a polynomial of degree n-1 A(x) = i=0 Evaluate at a point x = b in time ? Polynomials n-1 a i x i a polynomial of degree n-1 A(x) = i=0 Evaluate at a point x = b in

440 views • 16 slides
Advanced Algorithms Polynomial Identity Testing (PIT) two polynomials f,

Advanced Algorithms Polynomial Identity Testing (PIT) two polynomials f,

Advanced Algorithms Polynomial Identity Testing (PIT) two polynomials f, g F [ x ] of degree d Input : Output : f g ? d X a i x i for of degree d : f ( x ) = a i F f F [ x ] i =0 a polynomial Input : of

718 views • 35 slides
Solving a Nonlinear Equation Problem: Given f ( x ), find a root (solution) of f ( x ) = 0. If

Solving a Nonlinear Equation Problem: Given f ( x ), find a root (solution) of f ( x ) = 0. If

Solving a Nonlinear Equation Problem: Given f ( x ), find a root (solution) of f ( x ) = 0. If f is a polynomial with degree 4 or less, formulas for roots exist. If f is a polynomial with degree larger than 4, no finite formula exists

482 views • 12 slides
Introduction Warping polynomial Span of warping polynomial Span and dealternating number Ayaka

Introduction Warping polynomial Span of warping polynomial Span and dealternating number Ayaka

Introduction Warping polynomial Span of warping polynomial Span and dealternating number Ayaka Shimizu (Osaka City University) The span of the warping polynomial of a knot diagram Introduction Warping polynomial Span of warping polynomial

569 views • 24 slides
From Polynomial to Rational Maps: Newtons Method as a Dynamical System Dierk Schleicher

From Polynomial to Rational Maps: Newtons Method as a Dynamical System Dierk Schleicher

From Polynomial to Rational Maps: Newtons Method as a Dynamical System Dierk Schleicher Barcelona, 26 March 2019 Dierk Schleicher From Polynomial to Rational Maps / Newton 2 Overview Guiding principle of talk : Holomorphic dynamics has

509 views • 25 slides
Finding one root of a polynomial system A brief review of Smales 17th problem Pierre Lairez

Finding one root of a polynomial system A brief review of Smales 17th problem Pierre Lairez

Finding one root of a polynomial system A brief review of Smales 17th problem Pierre Lairez Inria Saclay SIAM AG 17 SIAM conference on applied algebraic geometry 2 august 2017, Atlanta Solving polynomial systems in polynomial time? degree

395 views • 26 slides
Polynomial Interpolation (PI) Problem: Given n + 1 points ( x 0 , y 0 ), ( x 1 , y 1 ),. . . ,( x

Polynomial Interpolation (PI) Problem: Given n + 1 points ( x 0 , y 0 ), ( x 1 , y 1 ),. . . ,( x

Polynomial Interpolation (PI) Problem: Given n + 1 points ( x 0 , y 0 ), ( x 1 , y 1 ),. . . ,( x n , y n ), where x i are distinct, seek a polynomial p ( x ) with least degree such that p ( x i ) = y i for i = 0 , 1 , . . . , n , i.e., the

317 views • 5 slides
approximation algorithms I David Steurer Cornell Cargese Workshop, 2014 encoded as low-degree

approximation algorithms I David Steurer Cornell Cargese Workshop, 2014 encoded as low-degree

SUM - OF - SQUARES method and approximation algorithms I David Steurer Cornell Cargese Workshop, 2014 encoded as low-degree polynomial in meta-task 2 example: () = ,

453 views • 34 slides

More recommend