A Statistical Investigation of Long Memory in Language and Music - PowerPoint PPT Presentation

A Statistical Investigation of Long Memory in Language and Music Alexander Greaves-Tunnell and Zaid Harchaoui ICML 2019

Problem • How do we define long-range dependence ?

Problem • How do we define long-range dependence ? • How can it be estimated in modern sources of sequence data?

Problem • How do we define long-range dependence ? • How can it be estimated in modern sources of sequence data? • How can we evaluate if a model has captured this property?

Contributions • Introduce a framework for evaluation of long-range dependence anchored in the literature of long memory stochastic processes.

Contributions • Introduce a framework for evaluation of long-range dependence anchored in the literature of long memory stochastic processes. 1 . 0 AR(1) FI(d)-AR(1) 0 . 8 γ ( k ) (solid) 0 . 6 0 . 4 0 . 2 0 . 0 0 50 100 150 200 250 k

Contributions • Introduce a framework for evaluation of long-range dependence anchored in the literature of long memory stochastic processes. 1 . 0 AR(1) 50 FI(d)-AR(1) 0 . 8 � | γ ( k ) | (dashed) 40 γ ( k ) (solid) 0 . 6 30 0 . 4 20 0 . 2 10 0 . 0 0 0 50 100 150 200 250 k

Contributions • Introduce a framework for evaluation of long-range dependence anchored in the literature of long memory stochastic processes. 1 . 0 AR(1) 50 FI(d)-AR(1) 0 . 8 � | γ ( k ) | (dashed) 40 γ ( k ) (solid) 0 . 6 30 0 . 4 20 0 . 2 10 0 . 0 0 0 50 100 150 200 250 k • Adapt semiparametric statistical methods to define estimation and testing procedure for long memory in high dimensions.

Results: Language and Music Do language and music data have long memory?

Results: Language and Music Do language and music data have long memory? Visual heuristic from differing behavior of partial sums: � K ∞ long memory � | γ ( k ) | → short memory , as K → ∞ c < ∞ k =1

Results: Language and Music Do language and music data have long memory? Visual heuristic from differing behavior of partial sums: � K ∞ long memory � | γ ( k ) | → short memory , as K → ∞ c < ∞ k =1 Natural language Music 60 Penn TreeBank Miles Davis 4 . 0 Bible Oum Kalthoum 50 Facebook bAbI CBT J.S. Bach 3 . 5 40 � | γ ( k ) | � | γ ( k ) | 3 . 0 30 2 . 5 20 2 . 0 1 . 5 10 1 . 0 0 0 50 100 150 200 250 0 50 100 150 200 250 k k

Results: RNN Models Hypothesis test for long memory: anticipated result � �� � H 0 : d = 0 vs. H A : d > 0 .

Results: RNN Models Hypothesis test for long memory: experimental result � �� � H 0 : d = 0 vs. H A : d > 0 . Reject H 0 ? Model memory d p-value − 8 . 59 × 10 − 4 LSTM (trained) 0.583 X − 4 . 17 × 10 − 4 LSTM (untrained) 0.572 X − 5 . 96 × 10 − 4 Memory cell 0.552 X 2 . 37 × 10 − 3 SCRN 0.324 X

Results: RNN Models Hypothesis test for long memory: experimental result � �� � H 0 : d = 0 vs. H A : d > 0 . Reject H 0 ? Model memory d p-value − 8 . 59 × 10 − 4 LSTM (trained) 0.583 X − 4 . 17 × 10 − 4 LSTM (untrained) 0.572 X − 5 . 96 × 10 − 4 Memory cell 0.552 X 2 . 37 × 10 − 3 SCRN 0.324 X Come see our poster! Pacific Ballroom # 253 .