Rotation invariant spin glass models and a matrix integral Yoshiyuki Kabashima Institute for Physics of Intelligence, The University of Tokyo, Japan 1/34
Outline • Background, motivation, and purpose • Replica analysis in rotationally invariant (RI) models • Expectation propagation (EP) in RI models • Summary 2/34
Background • Sherrington-Kirkpatrick (SK) model (1975) – Originally: ``Solvable’’ model of spin glass – Later: Also handled as “prototype” model of inference problem { } N S i ∈ + 1, − 1 ∑ ∑ ( ) = − − h H S J ij S i S j S i ( ) , ⎧ J ij ~ i.i.d. N 0, N − 1 J 2 ⎨ i < j i = 1 ⎩ h : external field Replica symmetric (RS) solution ⎛ ⎞ N q = 1 ∑ ⎡ ⎤ ( ) 2 ( ) Dz tanh 2 β h + S i , ⎣ ⎦ J ⎧ ⎜ ⎟ ∫ q = β ˆ N qz i = 1 ⎜ ⎟ ⎨ β = T − 1 : inverse temp. ⎝ ⎠ q = J 2 q ˆ ⎩ ( ) dz exp − z 2 2 ⎛ ⎞ Dz ! ⎜ ⎟ 2 π de Almeida-Thouless (AT) condition ⎝ ⎠ Replica symmetry breaking (RSB) occurs and inference becomes difficult. ( ) ( ) ( ) Dz 1 − tanh 2 β h + 2 ∫ β 2 J 2 > 1 ˆ qz Consequences of “static” analysis by the replica method 3/34
Background • YK (2003), Bolthausen (2014) – Employment of belief propagation (BP)(=AMP) for SK model ( ) t = tanh β h + γ i ( ) − J 2 β 2 1 − q t − 1 ( ) m i ⎧ t − 2 ⎪ t { } m i β J ij S i S j ⎨ t + 1 = Jm t e γ i ⎡ ⎤ ⎪ ⎣ ⎦ i ⎩ Macro. dynamics (state evolution: SE) ( ) ( ) { } q t = Dz tanh 2 β h + ⎧ S i ∫ q t z ⎪ ˆ ⎨ q t + 1 = J 2 q t { } e β hS i ⎪ ˆ ⎩ Micro. instability condition of the fixed point of AMP BP’s fixed point is unstable ⇒ Inference by BP fails. ( ) ( ) ( ) Dz 1 − tanh 2 β h + 2 ∫ β 2 J 2 > 1 ˆ qz Consequences of “dynamical” analysis by AMP 4/34
RS SP eq. vs. AMP in SK model RS phase RSB phase ◯： trajectory of AMP ＋： trajectory of iterative substitution of TAP equation Curves: trajectory of iterative substitution of RS saddle point equation Insets: difference between m t+1 and m t From YK, JPSJ 72, pp. 1645–1649 (2003) 5/34
Background • Replica-BP correspondence in SK model Replica method Belief propagation (AMP) RS saddle point equation Macro. dynamics (state evolution: SE) ( ) ( ) ( ) ( ) Dz tanh 2 β h + q t = Dz tanh 2 β h + ⎧ ⎧ ∫ ∫ q t z q = ⎪ ˆ qz ˆ ⎨ ⎨ q t + 1 = J 2 q t q = J 2 q ˆ ⎪ ⎩ ˆ ⎩ AT instability of RS solution Instability of AMP’s fixed point ( ) ( ) ( ) ( ) ( ) ( ) Dz 1 − tanh 2 β h + Dz 1 − tanh 2 β h + 2 2 ∫ ∫ β 2 J 2 > 1 β 2 J 2 > 1 ˆ ˆ qz qz ü Similar correspondence also holds for CDMA/Hopfield/CS models. 6/34
Motivation • Rotationally invariant (RI) models ( ) × O ⊤ ⎧ J = O × diag λ i N ⎪ ∑ ∑ ( ) = − − h H S J ij S i S j S i ⎨ O ~ uniform dist. on O ( N ) ( ) λ i ~ ρ λ ⎪ i < j i = 1 ⎩ – Parisi-Potters (1994), Opper-Winther (2001), Takeda-Uda-YK (2006), … – Components of connection matrices are (weakly) correlated. – Exact analysis is still possible by the replica method using a characteristic function for matrix ensemble, which we here refer to as “matrix integral” ) + Λ x ⎧ ⎫ − 1 − 1 2 ln x − 1 ( ) ! extr ( ) ln Λ − λ ( ∫ d λρ λ ⎨ ⎬ G x ⎩ ⎭ Λ 2 2 2 – BP-based analysis is also possible by the technique of “expectation- propagation” (EP), which was recently re-discovered as “vector approximate message passing (VAMP)” • Minka (2001), Opper-Winther (2005), Rangan et al (2017), … How is the replica-BP correspondence generalized? 7/34
Purpose • We here examine how the correspondence is generalized for the RI SG models using the matrix integral G(x) . 8/34
Short course of replica method { } • Random Hamiltonian → Necessity of config. avg. w.r.t. J ij – Edwards and Anderson (1975) Thermal average ( ) ( ) e − β H S J O S ( ) = Tr ( ) P O = Tr S O S β S J Random variable ( ) { } Z β J S depending on J ij Configurational (quenched) average All moments → Distribution of <O> → Full information about the system 9/34
Short course of replica method • Unfortunately, assessment of the config. avg. is difficult ( ) k ( ) e ⎛ ⎞ − β H S J ( ) ( ) = Tr Tr S O S − β H S J ( ) Z β J S e ∏ ⎡ ⎤ ∫ k ⎦ = ⎜ ⎟ O dJ ij P J ij ⎣ ( ) − β H S J ⎜ ⎟ Tr S e ( ) ⎝ ⎠ ij Main source of difficulty n ≥ k • This difficulty is resolved for ``extended’’ avgs. for ( ) ( ) ( ) n − k k ( ) ( ) ∏ ( ) e ∫ − β H S J − β H S J dJ ij P J ij Tr S e Tr S O S ( ) O n J ⎡ ⎤ k Z β ⎣ ⎦ ( ) ⎡ ⎤ k = ij ⎦ n ! O ( ) ⎣ ( ) n J ( ) ⎡ ⎤ n ( ) ∏ ∫ Z β − β H S J ⎣ ⎦ dJ ij P J ij Tr S e ( ) ij No negative power of partition functions • Can be assessed separately • 10/34
Short course of replica method • Key formula – For 𝑜 = 1,2, … ∈ Ν ( ) ∑ n ⎛ ⎞ H S a J − β ∑ ( ) ∑ − β H S J = e e ⎜ ⎟ a ⎝ ⎠ S 1 , S 2 , … , S n S – Note that this does not generally holds for real numbers 𝑜 ∈ ℝ 2 , … , S 1 , S n • Spins are called “replicas”. S 11/34
Short course of replica method • For 𝑜 = 1,2, … ∈ ℕ , extended avg. = Avg. w.r.t. joint dist. of “replicas” defined as ( ) ⎛ ⎞ ( ) n a J ∏ ∑ β H S ∫ exp − dJ ij P J ij ⎜ ⎟ ⎝ ⎠ ( ) ! ( ) a = 1 β S 1 , S 2 , … , S n ij P ( ) ∏ ( ) ∫ n J dJ ij P J ij Z β ( ) ij ( ) O S 1 ( ) O S 2 ( ) ! O S k ( ) ⎡ ⎤ ⎦ n = β S 1 , S 2 , … , S n k O S 1 , S 2 , … , S n P Tr ⎣ • The joint dist. = Canonical dist. of “non-random” Hamiltonian ( ) ⎛ ⎞ ⎛ ⎞ ( ) ( ) ! − 1 n a J ∏ ∑ β H S ∫ H S 1 , S 2 , … , S n exp − β ln dJ ij P J ij ⎜ ⎜ ⎟ ⎟ ⎝ ⎠ ⎝ ⎠ ( ) a = 1 ij Randomness is averaged out Standard stat. mech. techniques applicable 12/34
Short course of replica method • Replica method – Evaluate the config. avgs. by the following procedures 1. For 𝑜 = 1,2, … ∈ ℕ , analytically evaluate the extended avgs. as a function of 𝑜 . ( ) O S 1 ( ) O S 2 ( ) ! O S k ( ) ⎡ ⎤ k ⎦ n = β S 1 , S 2 , … , S n O S 1 , S 2 , … , S n P Tr ⎣ 2. Under appropriate assumptions, the obtained expression is likely to hold for real numbers 𝑜 ∈ ℝ . So, we exploit the expression to assess the config. avgs. as ( ) ( ) ( ) n − k k ( ) ( ) ∏ ( ) e ∫ − β H S J − β H S J dJ ij P J ij Tr S e Tr S O S ( ) O Z n J ⎡ ⎤ k ⎣ ⎦ ( ) ⎡ ⎤ k = ij ⎦ n ! O ( ) ⎣ ( ) Z n J ( ) ⎡ ⎤ n ∏ ( ) ∫ ⎣ ⎦ − β H S J dJ ij P J ij Tr S e ( ) ij ( ) k ( ) e ⎛ ⎞ − β H S J Tr S O S ( ) ∏ ∫ ⎡ ⎤ n → 0 ⎯ ⎯⎯ → ⎜ ⎟ = k dJ ij P J ij O ⎣ ⎦ ( ) − β H S J ⎜ ⎟ Tr S e ( ) ⎝ ⎠ ij 13/34
Short course of replica method • In practice, the computation is reduced to the following procedures $ 𝐾 1. For 𝑜 = 1,2, … ∈ ℕ , analytically evaluate 𝑂 !" ln 𝑎 # as a function of 𝑜 (using the saddle point method in most cases). ( ) ! − 1 ( ) n J ⎡ ⎤ φ β n β N ln Z β ⎣ ⎦ 2. Under appropriate assumptions, the obtained expression is likely to hold for real 𝑜 ∈ ℝ . So, we exploit the expression to assess the config. avg. of “free energy” as ∂ ∂ ⎦ ! − 1 1 ( ) ( ) ( ) ( ) n J ⎡ ⎤ ⎡ ⎤ f β ⎦ = − lim ⎦ = lim ∂ n φ β n ⎡ ⎤ ⎣ β N ln Z β J β N ln Z β ⎣ ⎣ ∂ n n → 0 n → 0 14/34
Replica analysis in RI models • Partition function ⎛ ⎞ ⎛ ⎞ ( ) + β h ⋅ S exp 1 ∑ ∑ ∑ ∑ ( ) = Z β exp β + β h = 2 Tr β JSS ⊤ J ij S i S j S i ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ i < j S i S • Rotationally invariant matrix ensemble ( ) × O ⊤ ⎧ J = O × diag λ i ⎪ ⎨ O ~ uniform dist. on O ( N ) ( ) λ i ~ ρ λ ⎪ ⎩ { } n ∈ 1,2, … • Moments of partition function for ⎡ ⎤ ⎛ ⎞ ⎛ ⎞ n n Z n β ( ) exp 1 ∑ ∑ ∏ ( ) S a S a ⊤ ⎡ ⎦ J = ⎤ 2 Tr β J × e β h ⋅ S a ⎢ ⎥ ⎣ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ a = 1 a = 1 S 1 , ! , S n J 15/34
Replica analysis in RI models Rotational invariance assumption for the coupling matrix yields ⎡ ⎤ ⎛ ⎞ ⎛ ⎞ n ( ) N ln exp 1 1 ( ) + n − 1 ( ) ∑ ⊤ ( ) + β nq ( ) G β 1 − q ( ) S a S a 2 Tr β J = G β 1 − q ⎢ ⎥ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ a = 1 𝑇 % 𝑇 % ' J $ Eigenvalues of matrix 𝛾 ∑ %&" for replica spins of the replica symmetric (RS) configuration ⎧ ( ) a = b ⎪ 1 N S a ⋅ S b = 1 ⎨ u = . xO ⊤ 1 ( ) a ≠ b q ⎪ ⎩ Here, the characteristic function is defined as ⎡ ⎤ ⎛ ⎞ ⎡ ⎤ ⎛ ⎞ N N ( ) ( ) ⊤ diag λ i ( ) ! 1 N ln exp x = 1 N ln exp x ( ) O ⊤ 1 ∑ ∑ 2 O ⊤ 1 ⎢ ⎥ ⎜ ⎟ Nx G x J ij ⎜ ⎟ ⎢ ⎥ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ ⎢ 2 ⎥ J ⎣ ⎦ i = 1 j = 1 O ( ) d u ⎡ ⎤ ⎛ ⎞ 2 − Nx N exp 1 ∑ ∫ λ i u i ⎟ δ u 2 ⎢ ⎥ ⎜ ⎝ ⎠ ) d λ + Λ x ⎧ ⎫ 2 = 1 − 1 − 1 2 ln x − 1 ( ) ln Λ − λ ( ⎢ ⎥ ∫ i = 1 ρ λ ! extr ⎨ ⎬ ( ) d u N ln 2 − Nx ∫ ⎢ ⎥ δ u ⎩ ⎭ Λ 2 2 2 ⎢ ⎥ ⎣ ⎦ O 16/34
Recommend
More recommend
