Applications of large deviation theory Hugo Touchette National - PDF document

Applications of large deviation theory Hugo Touchette National Institute for Theoretical Physics (NITheP) Stellenbosch, South Africa Maties Machine Learning 20 April 2018 Hugo Touchette (NITheP) Large deviations April 2018 1 / 10 Overview Goal Statistics Explain how deterministic behavior arises from randomness Probability • Stochastic processes , Large Simulations • Many interacting components deviations • Rare events Statistical • Typical or generic events Physics • Emergent determinism Stochastic processes Hugo Touchette (NITheP) Large deviations April 2018 2 / 10

Large deviation theory in one slide S n S n ( � X ) Macro X 2 X 1 � X = ( X 1 , X 2 , . . . , X n ) Micro Large deviation principle (LDP) p ( S = s ) n P ( S n = s ) ≈ e − nI ( s ) n = 500 • I ( s ) = rate function I ( s ) • Exponentially rare events n = 100 • Typical value: I ( s ∗ ) = 0 n = 10 • Concentration of probability µ s Hugo Touchette (NITheP) Large deviations April 2018 3 / 10 Coin tossing � — n = 5 H T T H T · · · — n = 25 � — n = 50 1 0 0 1 0 · · · � � ( � ) — n = 100 � · · · X 1 X 2 X 3 X 4 X 5 � � n ��� ��� ��� ��� ��� ��� S n = # heads = 1 � � X i n n ��� i =1 ��� ��� ��� � ( � ) ��� LDP ��� ��� ��� P ( S n = s ) ≈ e − nI ( s ) ��� ��� ��� ��� ��� ��� � Λ n I ( s ) = s log s + (1 − s ) log(1 − s ) + log 2 Typical sequences Typical # { � X : S n = 0 . 5 } ≈ 2 n Atypical Hugo Touchette (NITheP) Large deviations April 2018 4 / 10

Gaussian vectors and hyperspheres • Gaussian vector in n -dim: � X = ( X 1 , X 2 , . . . , X n ) • X i ∼ N (0 , 1) iid ∼ n 1 / 2 • Rescaled “radius”: ∼ n 1 / 4 n S n = 1 � X 2 i n i =1 LDP Typicality • Most points lie on surface P ( S n = s ) ≈ e − nI ( s ) • Asymptotically uniform • Typical value: S n → 1 • Volume concentrated near • Exponential concentration surface as n → ∞ Hugo Touchette (NITheP) Large deviations April 2018 5 / 10 Statistical physics • Total energy: N v 2 � N ∼ 10 23 i U N = 2 m , i =1 • Velocity distribution: L N ( v ) = # particles v i ∈ [ v , v + ∆ v ] N ∆ v LDP P ( L N = ρ ) ≈ e − NI ( ρ ) ρ ( v ) • Equilibrium distribution: − mv 2 ρ ∗ ( v ) = c v 2 e 2 kB T v • Maxwell’s distribution Hugo Touchette (NITheP) Large deviations April 2018 6 / 10

Spin glasses • Energy: � � H = − J ij σ i σ j + h σ i ���� ij i disorder • Find min energy (ground state) • Count number Ω N ( u ) local min with given energy LDPs Typicality • Exponentially many Ω N ( u ) ≈ e N Σ( u ) metastable states • Most critical points are • Σ( u ) = entropy saddles in high dim Hugo Touchette (NITheP) Large deviations April 2018 7 / 10 Problems from machine learning • Neurons: { σ i } • Weights: { w ij } • Cost function: C (input , output , { w ij } ) Practical Fundamental • Number of metastable states • Why ML works at all? • Typicality of data • Generic attractors • Overfitting fluctuations • Represent typical features Hugo Touchette (NITheP) Large deviations April 2018 8 / 10

Other applications Λ n • Information theory • Random graphs Typical • Markov process: X t , S T [ x ] • Signal analysis • Statistical physics Atypical • Phase transitions • Full space is huge • Quantum systems • Most states “look” the same • ... F. den Hollander, Large Deviations , AMS, 2000 H. Touchette, The large deviation approach to statistical mechanics, Physics Reports 478 , 2009 www.physics.sun.ac.za/~htouchette Hugo Touchette (NITheP) Large deviations April 2018 9 / 10 Current research: LDs of Markov processes • Process: { X t } T t =0 x H t L • Observable: A T [ x ] LDP t P ( A T = a ) ≈ e − TI ( a ) P H A T = a L Problems • Predict how rare fluctuations happen a • Effective process describing fluctuations x ( t ) • Related to non-Hermitian spectral problem t Hugo Touchette (NITheP) Large deviations April 2018 10 / 10