Replica method: a statistical mechanics approach to - PowerPoint PPT Presentation

bg=black!2 Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Replica method: a statistical mechanics approach to probability-based information processing Toshiyuki Tanaka tt@i.kyoto-u.ac.jp Graduate School of Informatics, Kyoto University, Kyoto, Japan 17th International Symposium on Mathematical Theory of Networks and Systems, Kyoto, Japan, July 25, 2006 Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Introduction Replica method Developed in studies of spin glasses (=magnetic materials with random spin-spin interactions) Recently applied to problems in information sciences: Neural networks Statistical learning theory Combinatorial optimization problems Error-correcting codes CDMA (digital wireless communication) Eigenvalue distribution of random matrices Still lacks rigorous mathematical justification Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Introduction Objectives To give a review of the replica method, as well as its mathematically questionable point. To demonstrate its applications. Eigenvalue distribution of random matrices. Analysis of digital communication systems. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Problem Basic defs. A : N × N real symmetric random matrix λ 1 , . . . , λ N : Eigenvalues of A Empirical eigenvalue distribution N ρ A ( x ) = 1 � δ ( x − λ i ) N i =1 Problem To evaluate � � ρ ( x ) = lim ρ A ( x ) N →∞ E A Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Basic results Wigner’s semicircle law (Wigner, 1951) A = ( a ij ): N × N matrix, a ij ( i ≤ j ): i.i.d., mean 0, variance 1 / N . Mar˘ cenko-Pastur law (Mar˘ cenko & Pastur, 1967) A = Ξ T Ξ, Ξ = ( ξ µ i ): p × N matrix; ξ µ i i.i.d., mean 0, variance 1 / N . (Girko’s) full-circle law (Girko, 1985) A = ( a ij ), a ij : i.i.d., mean 0, variance 1 / N . ( A not symmetric) Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Wigner’s semicircle law Wigner’s semicircle law A = ( a ij ), a ij ( i ≤ j ): i.i.d, mean 0, variance 1 / N . 1 ⎧ � 4 − x 2 ( | x | < 2) ⎨ 2 π ⇒ ρ ( x ) = 0 ( | x | > 2) ⎩ Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Wigner’s semicircle law -2 -1 0 1 2 Histogram of eigenvalues of a 6000 × 6000 random symmetric matrix with entries following Gaussian distribution. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Mar˘ cenko-Pastur law Mar˘ cenko-Pastur law A = Ξ T Ξ, Ξ = ( ξ µ i ): p × N matrix; ξ µ i : i.i.d., mean 0, variance 1 / N . � ⎧ 4 α − ( x − 1 − α ) 2 χ α ( x ) ( α ≥ 1) ⎪ ⎪ 2 π x ⎨ ρ ( x ) = � 4 α − ( x − 1 − α ) 2 ⎪ ⎪ (1 − α ) δ ( x ) + χ α ( x ) (0 < α < 1) ⎩ 2 π x α ≡ p / N χ α ( x ): Characteristic function of interval [(1 − √ α ) 2 , (1 + √ α ) 2 ]. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Mar˘ cenko-Pastur law 0.7 0.6 0.5 0.4 ρ (x) 0.3 0.2 0.1 0 0 1 2 3 4 5 6 x α = 0 . 3 , 0 . 6 , 2; Terms proportional to δ ( x ) not shown. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Full-circle 1 0.5 Im( λ ) 0 -0.5 -1 -1 -0.5 0 0.5 1 Re( λ ) Eigenvalue distribution of a real 6000 × 6000 random matrix. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Eigenvalue distribution of random matrices Useful for what? Wide applications in mathematical physics Applications in Information Processing Statistical learning theory Digital communication (kernel) PCA (bioinformatics, mathematical finance, etc.) Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Eigenvalue distribution of random matrix Approaches Marginalization of joint eigenvalue distribution (ex. Mehta, 1967) Evaluation of moments (ex. Brody et al., 1981) “Locator” expansion (ex. Bray & Moore, 1979; Hertz et al., 1989) Cavity method Free probability theory (ex. Voiculescu, 1985; Hiai & Petz, 2000) Replica method (ex. Edwards & Jones, 1976) Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Reformulation N ρ A ( x ) = 1 � δ ( x − λ i ) N i =1 1 � � ρ A ( x ) = lim π ℑ m A ( x + i ǫ ) � ρ A ( x ) ǫ → +0 Stieltjes trans. m A ( z ) = x − z dx R m A ( z ) = 1 N tr( A − zI ) − 1 tr ∗ − 1 = (log det ∗ ) ′ = 2 d dz log Z A ( z ) N Z A ( z ) = ( − 2 π i ) N / 2 | A − zI | − 1 / 2 � − i Gaussian integ. � � 2 u T ( A − zI ) u = R N exp d u rep. of det. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Averaging over A � 1 N � � ρ ( x ) = E A δ ( x − λ i ) N i =1 1 � � ρ A ( x ) = lim π ℑ m A ( x + i ǫ ) � ρ A ( x ) ǫ → +0 Stieltjes trans. m A ( z ) = x − z dx � 1 R � N tr( A − zI ) − 1 m ( z ) = E A tr ∗ − 1 = (log det ∗ ) ′ � 1 � = 2 d N log Z A ( z ) dz E A � − i � � 2 u T ( A − zI ) u Z A ( z ) = R N exp d u Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Outline of the approach 1 Evaluate � 1 � f ( z ) = lim N log Z A ( z ) N →∞ E A where � � − i � � � 2 u T ( A − zI ) u Z A ( z ) = R N exp d u 2 Calculate Stieltjes transform m ( z ) of ρ ( x ) with m ( z ) = 2 d dz f ( z ) . 3 Evaluate the inverse Stieltjes transform to obtain ρ ( x ): 1 � � ρ ( x ) = lim π ℑ m ( x + i ǫ ) ǫ → +0 Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Replica method Rewriting of formulas � 1 � f ( z ) = lim N log Z A ( z ) N →∞ E A � ∂ = E A ( Z n log Z ) � Z n � � ∂ n log E A E A ( Z n ) 1 ∂ � n � �� = lim N lim ∂ n log E A Z A ( z ) n → 0 N →∞ � � Exchange order of lim n → 0 ∂/∂ n and lim N →∞ . ∂ 1 � n � �� = lim ∂ n lim N log E A Z A ( z ) n → 0 N →∞ Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Replica method Goal 1 � n � �� To evaluate lim N log E A Z A ( z ) . N →∞ The replica “trick” Evaluate it by assuming n to be a positive integer. Believe the result to be valid for real n . No mathematically rigorous justification. Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2 Introduction Basic results Limiting eigenvalue distribution of random matrices Approaches Digital communication Analysis via replica method Bibliography Random matrix ensemble Covariance matrix of random samples A = Ξ T Ξ, Ξ = ( ξ µ i ) { ξ µ i ; µ = 1 , . . . , p ; i = 1 , . . . , N } i.i.d., E ( ξ ) = 0, E ( ξ 2 ) = O (1 / N ), E ( ξ m ) = o (1 / N ) ( m ≥ 3) ⇒ Mar˘ cenko-Pastur Toshiyuki Tanaka MTNS2006: Replica method