Approximating the covariance matrix with heavy tailed columns and - PowerPoint PPT Presentation

Approximating the covariance matrix with heavy tailed columns and RIP. Alexander Litvak University of Alberta based on a joint work with O. Guédon, A. Pajor and N. Tomczak-Jaegermann (the paper “On the interval ...” available at: http://www.math.ualberta.ca/˜alexandr/) Aleksander Pełczy´ nski Memorial Conference Bedlewo, 2014 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 1 / 21

Notations � · , · � denotes the canonical inner product on R n . | · | denotes the canonical Euclidean norm on R n . Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 2 / 21

Notations � · , · � denotes the canonical inner product on R n . | · | denotes the canonical Euclidean norm on R n . A random vector X ∈ R n is called isotropic if for all y ∈ R n . E |� X , y �| 2 = | y | 2 . E � X , y � = 0 and In other words, if X is centered and its covariance matrix is the identity: E X ⊗ X = Id (recall ( X ⊗ Y )( z ) = � X , z � Y or X ⊗ Y = { Y i X j } ij ) . Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 2 / 21

Notations � · , · � denotes the canonical inner product on R n . | · | denotes the canonical Euclidean norm on R n . A random vector X ∈ R n is called isotropic if for all y ∈ R n . E |� X , y �| 2 = | y | 2 . E � X , y � = 0 and In other words, if X is centered and its covariance matrix is the identity: E X ⊗ X = Id (recall ( X ⊗ Y )( z ) = � X , z � Y or X ⊗ Y = { Y i X j } ij ) . For an n × N matrix T its operator norm from ℓ N 2 to ℓ n 2 is denoted by � T � = sup | Tx | . | x | = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 2 / 21

KLS problem X 1 , . . . , X N are independent random vectors in R n . We consider the following model: For simplicity we assume that they are identically distributed and isotropic. Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 3 / 21

KLS problem X 1 , . . . , X N are independent random vectors in R n . We consider the following model: For simplicity we assume that they are identically distributed and isotropic. Approximation of covariance matrix (Kannan-Lovász-Simonovits (KLS) question): How many random vectors X i are needed for the empirical covariance matrix N 1 � X i ⊗ X i N i = 1 to approximate the identity with overwhelming probability? (In Asymptotic Geometric Analysis this question was first asked about vectors uniformly distributed in an isotropic convex body. The approximation was needed in order to estimate the complexity of an algorithm computing the volume of the body). Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 3 / 21

KLS problem Given ε > 0, how large N must be in order to have N � 1 � � � X i ⊗ X i − Id � � N � i = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � � � � N � N � y ∈ S n − 1 � � i = 1 i = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � ≤ ε � � � � N � N � y ∈ S n − 1 � i = 1 i = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � ≤ ε � � � � N � N � y ∈ S n − 1 � i = 1 i = 1 or, equivalently, N 1 − ε ≤ 1 � X i , y � 2 ≤ 1 + ε. � ∀ y ∈ S n − 1 N i = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � ≤ ε � � � � N � N � y ∈ S n − 1 � i = 1 i = 1 or, equivalently, N 1 − ε ≤ 1 � X i , y � 2 ≤ 1 + ε. � ∀ y ∈ S n − 1 N i = 1 KLS ( 95/97 ): N ∼ C ( ε, δ ) n 2 with Prob ≥ 1 − δ. Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � ≤ ε � � � � N � N � y ∈ S n − 1 � i = 1 i = 1 or, equivalently, N 1 − ε ≤ 1 � X i , y � 2 ≤ 1 + ε. � ∀ y ∈ S n − 1 N i = 1 KLS ( 95/97 ): N ∼ C ( ε, δ ) n 2 with Prob ≥ 1 − δ. Bourgain ( 96/99 ): N ∼ C ( ε, δ ) n ln 3 n with Prob ≥ 1 − δ . Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � ≤ ε � � � � N � N � y ∈ S n − 1 � i = 1 i = 1 or, equivalently, N 1 − ε ≤ 1 � X i , y � 2 ≤ 1 + ε. � ∀ y ∈ S n − 1 N i = 1 KLS ( 95/97 ): N ∼ C ( ε, δ ) n 2 with Prob ≥ 1 − δ. Bourgain ( 96/99 ): N ∼ C ( ε, δ ) n ln 3 n with Prob ≥ 1 − δ . Improved to N ∼ C ( ε, δ ) n ln 2 n by Rudelson and to N ∼ C ( ε, δ ) n ln n by Giannopoulos, Hartzoulaki, Tsolomitis and by Paouris. Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

KLS problem Given ε > 0, how large N must be in order to have � � N N � 1 1 � � � � � X i , y � 2 − 1 � � � � X i ⊗ X i − Id � = sup � ≤ ε � � � � N � N � y ∈ S n − 1 � i = 1 i = 1 or, equivalently, N 1 − ε ≤ 1 � X i , y � 2 ≤ 1 + ε. � ∀ y ∈ S n − 1 N i = 1 KLS ( 95/97 ): N ∼ C ( ε, δ ) n 2 with Prob ≥ 1 − δ. Bourgain ( 96/99 ): N ∼ C ( ε, δ ) n ln 3 n with Prob ≥ 1 − δ . Improved to N ∼ C ( ε, δ ) n ln 2 n by Rudelson and to N ∼ C ( ε, δ ) n ln n by Giannopoulos, Hartzoulaki, Tsolomitis and by Paouris. Aubrun ( 07 ): N ∼ n /ε 2 if X 1 is unconditional Prob ≥ 1 − exp ( − cn 1 / 5 ) . with Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 4 / 21

Solution of KLS Problem in log-concave setting. Theorem (Adamczak-LPT, 2010) Let X 1 , . . . , X N be independent isotropic log-concave random vectors. Let ε ∈ ( 0 , 1 ) . N ≥ Cn /ε 2 Then for one has � N � ≥ 1 − exp ( − c √ n ) . � 1 � � ( |� X i , y �| 2 − E |� X i , y �| 2 ) � P sup � ≤ ε � � N y ∈ S n − 1 i = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 5 / 21

Solution of KLS Problem in log-concave setting. Theorem (Adamczak-LPT, 2010) Let X 1 , . . . , X N be independent isotropic log-concave random vectors. Let ε ∈ ( 0 , 1 ) . N ≥ Cn /ε 2 Then for one has � N � ≥ 1 − exp ( − c √ n ) . � 1 � � ( |� X i , y �| 2 − E |� X i , y �| 2 ) � P sup � ≤ ε � � N y ∈ S n − 1 i = 1 In other words, for N ≥ Cn, with high probability we have � n N � 1 � � � X i ⊗ X i − Id � ≤ C N . � � N i = 1 Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 5 / 21

Solution of KLS Problem in log-concave setting. Theorem (Adamczak-LPT, 2010) Let X 1 , . . . , X N be independent isotropic log-concave random vectors. Let ε ∈ ( 0 , 1 ) . N ≥ Cn /ε 2 Then for one has � N � ≥ 1 − exp ( − c √ n ) . � 1 � � ( |� X i , y �| 2 − E |� X i , y �| 2 ) � P sup � ≤ ε � � N y ∈ S n − 1 i = 1 In other words, for N ≥ Cn, with high probability we have � n N � 1 � � � X i ⊗ X i − Id � ≤ C N . � � N i = 1 Remark. A measure µ on R n is log-concave if for every measurable A , B ⊂ R n and every θ ∈ [ 0 , 1 ] , µ ( θ A + ( 1 − θ ) B ) ≥ µ ( A ) θ µ ( B ) ( 1 − θ ) Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 5 / 21

Relations to standard Random Matrix Theory (RMT) RMT studies in particular limit behavior of singular numbers of random matrices. Recall for n × N matrix A , the largest and the smallest singular values are defined as | x | = 1 � Ax � = 1 / � A − 1 � . s 1 ( A ) = sup � Ax � = � A � and s n ( A ) = inf | x | = 1 Classical result is the Bai-Yin Theorem. Theorem (Bai-Yin) Let A be an n × N random matrix with i.i.d. entries whose 4-th moments are bounded. Let n β = lim N ∈ ( 0 , 1 ) . n →∞ Then √ √ � � 1 − β = lim n →∞ s n ( A / N ) ≤ lim n →∞ s 1 ( A / N ) = 1 + β. Alexander Litvak (Univ. of Alberta) Approximating the covariance matrix and RIP Bedlewo, 2014 6 / 21