Lecture 8 AR, MA, and ARMA Models 9/27/2018 1 AR models AR(p) - - PowerPoint PPT Presentation

Lecture 8

AR, MA, and ARMA Models

9/27/2018

1

AR models

AR(p) models

We can generalize from an AR(1) to an AR(p) model by simply adding additional autoregressive terms to the model.

𝐵𝑆(𝑞) ∶ 𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + 𝜚2 𝑧𝑢−2 + ⋯ + 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 = 𝜀 + 𝑥𝑢 +

𝑞

∑

𝑗=1

𝜚𝑗 𝑧𝑢−𝑗

What are the properities of 𝐵𝑆(𝑞),

• 1. Expected value?
• 2. Autocovariance / autocorrelation?
• 3. Stationarity conditions?

2

AR(p) models

We can generalize from an AR(1) to an AR(p) model by simply adding additional autoregressive terms to the model.

𝐵𝑆(𝑞) ∶ 𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + 𝜚2 𝑧𝑢−2 + ⋯ + 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 = 𝜀 + 𝑥𝑢 +

𝑞

∑

𝑗=1

𝜚𝑗 𝑧𝑢−𝑗

What are the properities of 𝐵𝑆(𝑞),

• 1. Expected value?
• 2. Autocovariance / autocorrelation?
• 3. Stationarity conditions?

2

Lag operator

The lag operator is convenience notation for writing out AR (and other) time series models. We define the lag operator 𝑀 as follows,

𝑀 𝑧𝑢 = 𝑧𝑢−1

this can be generalized where,

𝑀2𝑧𝑢 = 𝑀 (𝑀 𝑧𝑢) = 𝑀 𝑧𝑢−1 = 𝑧𝑢−2

therefore,

𝑀𝑙 𝑧𝑢 = 𝑧𝑢−𝑙

3

Lag operator

The lag operator is convenience notation for writing out AR (and other) time series models. We define the lag operator 𝑀 as follows,

𝑀 𝑧𝑢 = 𝑧𝑢−1

this can be generalized where,

𝑀2𝑧𝑢 = 𝑀 (𝑀 𝑧𝑢) = 𝑀 𝑧𝑢−1 = 𝑧𝑢−2

therefore,

𝑀𝑙 𝑧𝑢 = 𝑧𝑢−𝑙

3

Lag polynomial

Lets rewrite the 𝐵𝑆(𝑞) model using the lag operator,

𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + 𝜚2 𝑧𝑢−2 + … + 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 𝑧𝑢 = 𝜀 + 𝜚1 𝑀 𝑧𝑢 + 𝜚2 𝑀2 𝑧𝑢 + … + 𝜚𝑞 𝑀𝑞 𝑧𝑢 + 𝑥𝑢 𝑧𝑢 − 𝜚1 𝑀 𝑧𝑢 − 𝜚2 𝑀2 𝑧𝑢 − … − 𝜚𝑞 𝑀𝑞 𝑧𝑢 = 𝜀 + 𝑥𝑢 (1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − … − 𝜚𝑞 𝑀𝑞) 𝑧𝑢 = 𝜀 + 𝑥𝑢

This polynomial of lags

𝜚𝑞(𝑀) = (1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − ⋯ − 𝜚𝑞 𝑀𝑞)

is called the characteristic polynomial of the AR process.

4

Lag polynomial

Lets rewrite the 𝐵𝑆(𝑞) model using the lag operator,

𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + 𝜚2 𝑧𝑢−2 + … + 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 𝑧𝑢 = 𝜀 + 𝜚1 𝑀 𝑧𝑢 + 𝜚2 𝑀2 𝑧𝑢 + … + 𝜚𝑞 𝑀𝑞 𝑧𝑢 + 𝑥𝑢 𝑧𝑢 − 𝜚1 𝑀 𝑧𝑢 − 𝜚2 𝑀2 𝑧𝑢 − … − 𝜚𝑞 𝑀𝑞 𝑧𝑢 = 𝜀 + 𝑥𝑢 (1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − … − 𝜚𝑞 𝑀𝑞) 𝑧𝑢 = 𝜀 + 𝑥𝑢

This polynomial of lags

𝜚𝑞(𝑀) = (1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − ⋯ − 𝜚𝑞 𝑀𝑞)

is called the characteristic polynomial of the AR process.

4

Lag polynomial

Lets rewrite the 𝐵𝑆(𝑞) model using the lag operator,

𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + 𝜚2 𝑧𝑢−2 + … + 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 𝑧𝑢 = 𝜀 + 𝜚1 𝑀 𝑧𝑢 + 𝜚2 𝑀2 𝑧𝑢 + … + 𝜚𝑞 𝑀𝑞 𝑧𝑢 + 𝑥𝑢 𝑧𝑢 − 𝜚1 𝑀 𝑧𝑢 − 𝜚2 𝑀2 𝑧𝑢 − … − 𝜚𝑞 𝑀𝑞 𝑧𝑢 = 𝜀 + 𝑥𝑢 (1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − … − 𝜚𝑞 𝑀𝑞) 𝑧𝑢 = 𝜀 + 𝑥𝑢

This polynomial of lags

𝜚𝑞(𝑀) = (1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − ⋯ − 𝜚𝑞 𝑀𝑞)

is called the characteristic polynomial of the AR process.

4

Stationarity of 𝐵𝑆(𝑞) processes

Claim: An 𝐵𝑆(𝑞) process is stationary if the roots of the characteristic polynomial lay outside the complex unit circle If we define 𝜇 = 1/𝑀 then we can rewrite the characteristic polynomial as

(𝜇𝑞 − 𝜚1𝜇𝑞−1 − 𝜚2𝜇𝑞−2 − ⋯ − 𝜚𝑞−1𝜇 − 𝜚𝑞)

then as a corollary of our claim the 𝐵𝑆(𝑞) process is stationary if the roots

• f this new polynomial are inside the complex unit circle (i.e. |𝜇| < 1).

5

Stationarity of 𝐵𝑆(𝑞) processes

Claim: An 𝐵𝑆(𝑞) process is stationary if the roots of the characteristic polynomial lay outside the complex unit circle If we define 𝜇 = 1/𝑀 then we can rewrite the characteristic polynomial as

(𝜇𝑞 − 𝜚1𝜇𝑞−1 − 𝜚2𝜇𝑞−2 − ⋯ − 𝜚𝑞−1𝜇 − 𝜚𝑞)

then as a corollary of our claim the 𝐵𝑆(𝑞) process is stationary if the roots

• f this new polynomial are inside the complex unit circle (i.e. |𝜇| < 1).

5

Example AR(1)

6

Example AR(2)

7

AR(2) Stationarity Conditions

From Shumway&Stofer4thed.

8

Proof Sketch

We can rewrite the 𝐵𝑆(𝑞) model into an 𝐵𝑆(1) form using matrix notation

𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + 𝜚2 𝑧𝑢−2 + ⋯ + 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 𝝄𝑢 = 𝜺 + 𝐆 𝝄𝑢−1 + 𝐱𝑢

where

⎡ ⎢ ⎢ ⎢ ⎣ 𝑧𝑢 𝑧𝑢−1 𝑧𝑢−2 ⋮ 𝑧𝑢−𝑞+1 ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ 𝜀 ⋮ ⎤ ⎥ ⎥ ⎥ ⎦ + ⎡ ⎢ ⎢ ⎢ ⎣ 𝜚1 𝜚2 𝜚3 ⋯ 𝜚𝑞−1 𝜚𝑞 1 ⋯ 1 ⋯ ⋮ ⋮ ⋮ ⋯ ⋮ ⋮ ⋯ 1 ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 𝑧𝑢−1 𝑧𝑢−2 𝑧𝑢−3 ⋮ 𝑧𝑢−𝑞 ⎤ ⎥ ⎥ ⎥ ⎦ + ⎡ ⎢ ⎢ ⎢ ⎣ 𝑥𝑢 ⋮ ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ 𝜀 + 𝑥𝑢 + ∑

𝑞 𝑗=1 𝜚𝑗 𝑧𝑢−𝑗

𝑧𝑢−1 𝑧𝑢−2 ⋮ 𝑧𝑢−𝑞+1 ⎤ ⎥ ⎥ ⎥ ⎦

9

Proof sketch (cont.)

So just like the original 𝐵𝑆(1) we can expand out the autoregressive equation

𝝄𝑢 = 𝜺 + 𝐱𝑢 + 𝐆 𝝄𝑢−1 = 𝜺 + 𝐱𝑢 + 𝐆 (𝜺 + 𝐱𝑢−1) + 𝐆2 (𝜺 + 𝐱𝑢−2) + ⋯ + 𝐆𝑢−1 (𝜺 + 𝐱1) + 𝐆𝑢 (𝜺 + 𝐱0) = (

𝑢

∑

𝑗=0

𝐺 𝑗)𝜺 +

𝑢

∑

𝑗=0

𝐺 𝑗 𝑥𝑢−𝑗

and therefore we need lim

𝑢→∞𝐺 𝑢 → 0.

10

Proof sketch (cont.)

We can find the eigen decomposition such that 𝐆 = 𝐑𝚳𝐑−1 where the columns of 𝐑 are the eigenvectors of 𝐆 and 𝚳 is a diagonal matrix of the corresponding eigenvalues. A useful property of the eigen decomposition is that

𝐆𝑗 = 𝐑𝚳𝑗𝐑−1

Using this property we can rewrite our equation from the previous slide as

𝝄𝑢 = (

𝑢

∑

𝑗=0

𝐺 𝑗)𝜺 +

𝑢

∑

𝑗=0

𝐺 𝑗 𝑥𝑢−𝑗 = (

𝑢

∑

𝑗=0

𝐑𝚳𝑗𝐑−1)𝜺 +

𝑢

∑

𝑗=0

𝐑𝚳𝑗𝐑−1 𝑥𝑢−𝑗

11

Proof sketch (cont.)

We can find the eigen decomposition such that 𝐆 = 𝐑𝚳𝐑−1 where the columns of 𝐑 are the eigenvectors of 𝐆 and 𝚳 is a diagonal matrix of the corresponding eigenvalues. A useful property of the eigen decomposition is that

𝐆𝑗 = 𝐑𝚳𝑗𝐑−1

Using this property we can rewrite our equation from the previous slide as

𝝄𝑢 = (

𝑢

∑

𝑗=0

𝐺 𝑗)𝜺 +

𝑢

∑

𝑗=0

𝐺 𝑗 𝑥𝑢−𝑗 = (

𝑢

∑

𝑗=0

𝐑𝚳𝑗𝐑−1)𝜺 +

𝑢

∑

𝑗=0

𝐑𝚳𝑗𝐑−1 𝑥𝑢−𝑗

11

Proof sketch (cont.)

𝚳𝑗 = ⎡ ⎢ ⎢ ⎣ 𝜇𝑗

1

⋯ 𝜇𝑗

2

⋯ ⋮ ⋮ ⋱ ⋮ ⋯ 𝜇𝑗

𝑞

⎤ ⎥ ⎥ ⎦

Therefore,

lim

𝑢→∞𝐺 𝑢 → 0

when

lim

𝑢→∞Λ𝑢 → 0

which requires that

|𝜇𝑗| < 1

for all 𝑗

12

Proof sketch (cont.)

Eigenvalues are defined such that for 𝝁,

det(𝐆 − 𝝁 𝐉) = 0

based on our definition of 𝐆 our eigenvalues will therefore be the roots of

𝜇𝑞 − 𝜚1 𝜇𝑞−1 − 𝜚2 𝜇𝑞−2 − ⋯ − 𝜚𝑞1 𝜇1 − 𝜚𝑞 = 0

which if we multiply by 1/𝜇𝑞 where 𝑀 = 1/𝜇 gives

1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − ⋯ − 𝜚𝑞1 𝑀𝑞−1 − 𝜚𝑞 𝑀𝑞 = 0

13

Proof sketch (cont.)

Eigenvalues are defined such that for 𝝁,

det(𝐆 − 𝝁 𝐉) = 0

based on our definition of 𝐆 our eigenvalues will therefore be the roots of

𝜇𝑞 − 𝜚1 𝜇𝑞−1 − 𝜚2 𝜇𝑞−2 − ⋯ − 𝜚𝑞1 𝜇1 − 𝜚𝑞 = 0

which if we multiply by 1/𝜇𝑞 where 𝑀 = 1/𝜇 gives

1 − 𝜚1 𝑀 − 𝜚2 𝑀2 − ⋯ − 𝜚𝑞1 𝑀𝑞−1 − 𝜚𝑞 𝑀𝑞 = 0

13

Properties of 𝐵𝑆(2)

For a stationary 𝐵𝑆(2) process where 𝑥𝑢 has 𝐹(𝑥𝑢) = 0 and

𝑊 𝑏𝑠(𝑥𝑢) = 𝜏2

𝑥

14

Properties of 𝐵𝑆(𝑞)

For a stationary 𝐵𝑆(𝑞) process where 𝑥𝑢 has 𝐹(𝑥𝑢) = 0 and

𝑊 𝑏𝑠(𝑥𝑢) = 𝜏2

𝑥

𝐹(𝑍𝑢) = 𝜀 1 − 𝜚1 − 𝜚2 − ⋯ − 𝜚𝑞 𝑊 𝑏𝑠(𝑧𝑢) = 𝛿(0) = 𝜚1𝛿(1) + 𝜚2𝛿(2) + … + 𝜚𝑞𝛿(𝑞) + 𝜏2

𝑥

𝛿(ℎ) = 𝜚1𝛿(ℎ − 1) + 𝜚2𝛿(ℎ − 2) + … + 𝜚𝑞𝛿(ℎ − 𝑞) 𝜍(ℎ) = 𝜚1 𝜍(ℎ − 1) + 𝜚2 𝜍(ℎ − 2) + … + 𝜚𝑞 𝜍(ℎ − 𝑞)

15

Moving Average (MA) Processes

MA(1)

A moving average process is similar to an AR process, except that the autoregression is on the error term.

𝑁𝐵(1) ∶ 𝑧𝑢 = 𝜀 + 𝑥𝑢 + 𝜄 𝑥𝑢−1

Properties:

16

Time series

17

ACF

2 4 6 8 10 0.0 0.4 0.8 Lag ACF

θ=0.1

2 4 6 8 10 0.0 0.4 0.8 Lag ACF

θ=0.8

2 4 6 8 10 0.0 0.4 0.8 Lag ACF

θ=2.0

2 4 6 8 10 −0.10 0.00 0.10 Lag Partial ACF

θ=0.1

2 4 6 8 10 −0.2 0.2 Lag Partial ACF

θ=0.8

2 4 6 8 10 −0.2 0.0 0.2 0.4 Lag Partial ACF

θ=2.0 18

MA(q)

𝑁𝐵(𝑟) ∶ 𝑧𝑢 = 𝜀 + 𝑥𝑢 + 𝜄1 𝑥𝑢−1 + 𝜄2 𝑥𝑢−2 + ⋯ + 𝜄𝑟 𝑥𝑢−𝑟

Properties:

𝐹(𝑧𝑢) = 𝜀 𝛿(0) = (1 + 𝜄2

1 + 𝜄2 2 + ⋯ + 𝜄2 𝑟) 𝜏2 𝑥

𝛿(ℎ) = {−𝜄𝑙 + 𝜄1𝜄𝑙+1 + 𝜄2𝜄𝑙+2 + ⋯ + 𝜄𝑟+𝑙𝜄𝑟

if |𝑙| ∈ {1, … , 𝑟}

• therwise

19

Example series

..={−1.5, −1, 2} ..={−1.5, −1, 2, 3} ..={−1.5} ..={−1.5, −1} 25 50 75 100 25 50 75 100 −10 −5 5 10 −10 −5 5 10

t y 20

ACF

2 4 6 8 10 −0.4 0.4 Lag ACF

θ={−1.5}

2 4 6 8 10 −0.2 0.6 Lag ACF

θ={−1.5, −1}

2 4 6 8 10 −0.5 0.5 Lag ACF

θ={−1.5, −1, 2}

2 4 6 8 10 −0.2 0.6 Lag ACF

θ={−1.5, −1, 2, 3}

21

ARMA Model

ARMA Model

An ARMA model is a composite of AR and MA processes,

𝐵𝑆𝑁𝐵(𝑞, 𝑟): 𝑧𝑢 = 𝜀 + 𝜚1 𝑧𝑢−1 + ⋯ 𝜚𝑞 𝑧𝑢−𝑞 + 𝑥𝑢 + 𝜄1𝑥𝑢−1 + ⋯ + 𝜄𝑟𝑥𝑢𝑟 𝜚𝑞(𝑀)𝑧𝑢 = 𝜀 + 𝜄𝑟(𝑀)𝑥𝑢

Since all 𝑁𝐵 processes are stationary, we only need to examine the 𝐵𝑆 aspect to determine stationarity (roots of 𝜚𝑞(𝑀) lie outside the complex unit circle).

22

Time series

..={0.9}, ..={0.9} ..={−0.9}, ..={0.9} ..={0.9}, ..={−0.9} ..={−0.9}, ..={−0.9} ..={0.9}, ..={−} ..={−0.9}, ..={−} ..={−}, ..={0.9} ..={−}, ..={−0.9} 25 50 75 100 0 25 50 75 100 0 25 50 75 100 0 25 50 75 100 −5 5 10 −5 5 10

t y 23

𝜚 = 0.9, 𝜄 = 0

φ={0.9}, θ={0}

20 40 60 80 100 −2 2 5 10 15 20 −0.4 0.2 0.6 Lag ACF 5 10 15 20 −0.4 0.2 0.6 Lag PACF 24

𝜚 = 0, 𝜄 = 0.9

φ={0}, θ={0.9}

20 40 60 80 100 −3 2 4 5 10 15 20 −0.2 0.2 Lag ACF 5 10 15 20 −0.2 0.2 Lag PACF 25

𝜚 = 0.9, 𝜄 = 0.9

φ={0.9}, θ={0.9}

20 40 60 80 100 −5 5 5 10 15 20 −0.2 0.2 0.6 Lag ACF 5 10 15 20 −0.2 0.2 0.6 Lag PACF 26