Data Mining Techniques CS 6220 - Section 3 - Fall 2016 Lecture 18: - PowerPoint PPT Presentation

Data Mining Techniques CS 6220 - Section 3 - Fall 2016 Lecture 18: Time Series Jan-Willem van de Meent ( credit: Aggarwal Chapter 14.3 )

Time Series Data http://www.capitalhubs.com/2012/08/the-correlation-between-apple-product.html

Time Series Data

Time Series Data • Time series forecasting is fundamentally hard • Rare events often play a big role in changing trends • Impossible to know how events will affects trends 
 (and often when such events will occur)

Time Series Data source: https://am241.wordpress.com/tag/time-series/ • In some cases there are clear trends 
 (here: seasonal effects + growth)

Autoregressive Models

Time Series Smoothing − − Moving Average Exponential 200 200 195 195 IBM STOCK PRICE IBM STOCK PRICE 190 190 185 185 180 180 175 175 ACTUAL VALUES ACTUAL VALUES 170 170 20 − DAY MOVING AVERAGE EXP. SMOOTHING ( α =0.1) 50 − DAY MOVING AVERAGE EXP. SMOOTHING ( α =0.05) 165 165 50 100 150 200 250 50 100 150 200 250 NUMBER OF TRADING DAYS NUMBER OF TRADING DAYS (a) Moving average smoothing (b) Exponential smoothing i = 1 P k � 1 y ′ i = α · y i + (1 − α ) · y ′ y 0 n =0 y i � n i − 1 k

Stationary Time Series E [ ✏ t ] = 0 y t = c + ✏ t Definition 14.3.1 (Strictly Stationary Time Series) A strictly stationary time series is one in which the probabilistic distribution of the values in any time interval [ a, b ] is identical to that in the shifted interval [ a + h, b + h ] for any value of the time shift h . Differencing y t - y t -1 Log differencing log y t - log y t -1 60 4.5 ORIGINAL SERIES ORIGINAL SERIES (LOG) LOGARITHM(PRICE VALUE) 4 DIFFERENCED SERIES DIFFERENCED SERIES (LOG) 50 3.5 PRICE VALUE 40 3 2.5 30 2 20 1.5 1 10 0.5 0 0 0 5 10 15 20 25 30 0 5 10 15 20 25 30 TIME INDEX TIME INDEX (a) Unscaled series (b) Logarithmic scaling

Auto-correlation IBM Stock Price Sine Wave 1 1 0.8 0.8 0.6 0.6 AUTOCORRELATION AUTOCORRELATION 0.4 0.4 0.2 0.2 0 0 − 0.2 − 0.2 − 0.4 − 0.4 − 0.6 − 0.6 − 0.8 − 0.8 − 1 − 1 0 50 100 150 200 250 0 100 200 300 400 500 600 700 800 900 1000 LAG LAG (DEGREES) Autocorrelation( L ) = Covariance t ( y t , y t + L ) . Variance t ( y t )

Autoregressive Models Autoregressive: AR(p) Moving-Average: MA(q) p q X � y t = b i · ϵ t − i + c + ϵ t y t = a i y t − i + c + ✏ t i =1 i =1 Autoregressive moving-average: ARMA(p,q) p q � � y t = a i · y t − i + b i · ϵ t − i + c + ϵ t i =1 i =1 Autoregressive integrated moving-average: ARIMA(p,d,q) p q y ( d ) a i y ( d ) X X b i ✏ t − i + c + ✏ t = t − i + t i =1 i =1 Do least-squares regression to estimate a,b,c

ARIMA on Airline Data ( p , d , q ) = (0,1,12) source: http://www.statsref.com/HTML/index.html?arima.html

Hidden Markov Models

Time Series with Distinct States

Can we use a Gaussian Mixture Model? Time Series Histogram Posterior on states Mixture

Can we use a Gaussian Mixture Model? Time Series Histogram Posterior on states Mixture

Hidden Markov Models Estimate from GMM Estimate from HMM • Idea: Mixture model + Markov chain for states • Can model correlation between subsequent states 
 (more likely to be in same state than different state) 


Reminder: Random Surfers in PageRank y/2 y a/2 y/2 m a m a/2 Model for random Surfer: • At time t = 0 pick a page at random • At each subsequent time t follow an 
 outgoing link at random (adapted from:: Mining of Massive Datasets, http://www.mmds.org)

Reminder: Random Surfers in PageRank y/2 y a/2 y/2 m a m a/2 (adapted from:: Mining of Massive Datasets, http://www.mmds.org)

Hidden Markov Models Gaussian Mixture Gaussian HMM A = M > z 1 ∼ Discrete( π ) z n ∼ Discrete( π ) z t +1 | z t = k ∼ Discrete( A k ) x n | z n = k ∼ Normal( µ k , σ k ) x t | z t = k ∼ Normal( µ k , σ k )

Review: Gaussian Mixtures Expectation Maximization 1. Update cluster probabilities tk = p ( z t = k | x t , θ i − 1 ) γ i p ( x t , z t = k | θ i − 1 ) = l p ( x t , z t = l | θ i − 1 ) P 2. Update parameters P T µ i t =1 γ i 1 k = tk x t N i k k ) 2 ⌘ 1 / 2 ⇣ P T z n ∼ Discrete( π ) σ i t =1 γ i tk ( x i t − µ i 1 k = N i k x n | z n = k ∼ Normal( µ k , σ k ) k = P T π i k = N i N i t =1 γ i k /N tk

Forward-backward Algorithm Expectation step for HMM γ t,k = p ( z t = k | x 1: T , θ ) = p ( x 1: t , z t ) p ( x t +1: T | z t ) p ( x 1: T ) ∝ α t,k β t,k α t,l := p ( x 1: t , z t ) X = p ( x t | µ l , σ l ) A kl α t − 1 ,k k z 1 ∼ Discrete( π ) β t,k := p ( x t +1: T | z t ) z t +1 | z t = k ∼ Discrete( A k ) X x t | z t = k ∼ Normal( µ k , σ k ) = β t +1 ,l p ( x t +1 | µ l , σ l ) A kl l

Other Examples for HMMs RNA splicing Handwritten Digits itten sam- hid- ained • State 1: Exon (relevant) • State 1: Sweeping arc • State 2: Splice site • State 2: Horizontal line • State 3: Intron (ignored)