Monitoring and data filtering II. Correlated data Advanced Herd - PowerPoint PPT Presentation

Monitoring and data filtering II. Correlated data Advanced Herd Management Anders Ringgaard Kristensen

The framework – as last lecture Farmer's utility Restraints function Goals Adjustments Planning Analysis Control Plan Implementation: Products Factors

Summary, previous lecture � Key figures k 1 , k 2 , … , k t are regarded as a time series. � Basic model (textbook pools the two errors): � The values k 1 , k 2 , … , k t were regarded as independent and identically distributed.

Summary, previous lecture � The distribution of k t is N( k , σ 2 ) – if errors are normally distributed. � The variance: σ 2 = V( e st ) + V( e ot ) � Under those assumptions various control charts were presented.

Relevant questions � Is it (always) reasonable to assume that the true underlying value k is constant over time: � Trends: k increases/decreases � Seasonality: k changes with season � Is it (always) reasonable to assume that the sample errors e s 1 , e s 2 , … , e st are independent? � Repeated measurements on same animal(s) � Temporary environmental effects � Is it (always) reasonable to assume that the observation errors e o 1 , e o 2 , … , e ot are independent? � Yes, very often, but it depends on the method of measurement.

Our main example – were we wrong? g 600 650 700 750 800 850 900 950 2. kvartal 97 3. kvartal 97 4. kvartal 97 1. kvartal 98 Daily gain, slaughter pigs 2. kvartal 98 3. kvartal 98 4. kvartal 98 1. kvartal 99 Period 2. kvartal 99 3. kvartal 99 4. kvartal 99 1. kvartal 00 2. kvartal 00 3. kvartal 00 4. kvartal 00 1. kvartal 01 2. kvartal 01

Were we wrong? � Is there a trend? � Is there a seasonal pattern? � Are the sample errors e s 1 , e s 2 , … , e st correlated? If yes, positively or negatively? � Are the observation errors e o 1 , e o 2 , … , e ot correlated? If yes, positively or negatively?

Check for autocorrelation � Present versus previous observation Cor r el ati on � Not obvious! 900 850 800 750 700 650 650 700 750 800 850 900 P r e v i o s

Check for autocorrelation � Present versus same quarter last year � Seems clear! Cor r el ati on � Short series 900 850 800 750 700 650 650 700 750 800 850 900 P r e v i o s

Sample autocorrelation 1 0,8 0,6 0,4 0,2 0 1 2 3 4 -0,2 -0,4 -0,6 -0,8 -1 Lag � Clear negative autocorrelation for lag 2 � Clear positive autocorrelation for lag 4 � Low autocorrelation for lags 1 & 3

A model for the autocorrelation � Assume that we are dealing with a pure seasonal effect: k t = µ + ρ 4 ( k t -4 - µ ) + ε t � Where µ = 762 g, ρ 4 = 0.68 and the residual ε t ∼ N(0, σ 2 (1- ρ 4 2 )) where σ = 7.4 g � Predicted value for k t +1 : � Forecast error:

Observed and predicted gain g 600 650 700 750 800 850 900 950 2. kvartal 97 3. kvartal 97 4. kvartal 97 1. kvartal 98 Daily gain, slaughter pigs 2. kvartal 98 Observed gain 3. kvartal 98 4. kvartal 98 1. kvartal 99 Period 2. kvartal 99 3. kvartal 99 Predicted gain 4. kvartal 99 1. kvartal 00 2. kvartal 00 3. kvartal 00 4. kvartal 00 1. kvartal 01 2. kvartal 01

Control chart – correlated data � Construct a model describing the correlation � Use the model to predict next observation � Calculate the forecast error (difference between the predicted and observed value) � Calculate the standard deviation of the forecast error � Create a usual control chart for the prediction error

Prediction error control chart, visual Daily gain, slaughter pigs 100 80 60 40 20 0 g -20 -40 -60 -80 -100 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Period Prediction error Upper control limit Lower control limit

Daily gain example, comments � Further options Daily gain, slaughter pigs 950 � Include information on 900 850 k t -2 (and perhaps even 800 g 750 700 k t -3 and k t-1 ) – 4th order 650 600 autoregressive time 2. kvartal 97 3. kvartal 97 4. kvartal 97 1. kvartal 98 2. kvartal 98 3. kvartal 98 4. kvartal 98 1. kvartal 99 2. kvartal 99 3. kvartal 99 4. kvartal 99 1. kvartal 00 2. kvartal 00 3. kvartal 00 4. kvartal 00 1. kvartal 01 2. kvartal 01 series: no marked improvement. Period Observed gain Predicted gain � Systematic seasonal effect. � Combine with what you know about animal production!

Dynamic linear models � West & Harrison, chapter 2. � Bayesian framework � Patterns � Self-calibrating – learning pattern from data � Only very simple versions in textbook � A more advanced version is presented here

A simple DLM � Observation equation: k t = µ t + e t , e t ∼ N(0, σ 2 ) � Like before: e t = e st + e os � The symbol µ t is the underlying true value at time t . � System equation: µ t = µ t -1 + w t , w t ∼ N(0, σ w 2 ) � The true value is not any longer assumed to be constant . � A fair assumption in animal production! � Basically, we wish to detect ”large” changes in µ t

A DLM with a trend � Observation equation: k t = µ t + e t , e t ∼ N(0, σ 2 ) � System equations: µ t = µ t -1 + β t -1 + w 1 t , w 1 t ∼ N(0, σ 1 w 2 ) β t = β t -1 + w 2 t , w 2 t ∼ N(0, σ 2 w 2 )

Matrix notation � Let K t = ( k 1 , … , k n )’ be a vector of key figures observed at time t . � Let θ t = ( θ 1 , … , θ m )’ be a vector of parameters describing the system at time t. � Observation equation K t = F t θ t + e t � System equation: θ t = G t θ t -1 + w t

Updating equations � See the textbook for details! � Each time an observation is made, the current estimates for the parameters are updating in a Bayesian framework.

Trend model in matrix notation

Seasonal pig gain model – 4 seasons

Seasonal pig gain DLM Daily gain, slaughter pigs 950 900 850 800 g 750 700 650 600 2. kvartal 97 3. kvartal 97 4. kvartal 97 1. kvartal 98 2. kvartal 98 3. kvartal 98 4. kvartal 98 1. kvartal 99 2. kvartal 99 3. kvartal 99 4. kvartal 99 1. kvartal 00 2. kvartal 00 3. kvartal 00 4. kvartal 00 1. kvartal 01 Period Observed gain Predicted gain

Control chart, forecast error Daily gain, slaughter pigs 100 80 60 40 20 0 g -20 -40 -60 -80 -100 7 7 7 8 8 8 8 9 9 9 9 0 0 0 0 1 9 9 9 9 9 9 9 9 9 9 9 0 0 0 0 0 l l l l l l l l l l l l l l l l a a a a a a a a a a a a a a a a t t t t t t t t t t t t t t t t r r r r r r r r r r r r r r r r a a a a a a a a a a a a a a a a v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k . . . . . . . . . . . . . . . . 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 Period Forecast error Lower limit Upper limit � Note that the standard deviation of the forecast error is calculated by the model.

Components – trend Daily gain, slaughter pigs 950 900 850 800 g 750 700 650 600 2. kvartal 97 3. kvartal 97 4. kvartal 97 1. kvartal 98 2. kvartal 98 3. kvartal 98 4. kvartal 98 1. kvartal 99 2. kvartal 99 3. kvartal 99 4. kvartal 99 1. kvartal 00 2. kvartal 00 3. kvartal 00 4. kvartal 00 1. kvartal 01 Period Observed gain Predicted gain Level Season 1 Season 2 Season 3 Season 4

Components – seasonal parts Daily gain, slaughter pigs 100 80 60 40 20 0 g -20 -40 -60 -80 -100 2. kvartal 97 3. kvartal 97 4. kvartal 97 1. kvartal 98 2. kvartal 98 3. kvartal 98 4. kvartal 98 1. kvartal 99 2. kvartal 99 3. kvartal 99 4. kvartal 99 1. kvartal 00 2. kvartal 00 3. kvartal 00 4. kvartal 00 1. kvartal 01 Period Season 1 Season 2 Season 3 Season 4

Seasonal effects � More sophisticated approaches exist � ”Seasonal” may just as well be a diurnal pattern. � Thomas Nejsum Madsen will present an approach based on sine functions.

DLM in production monitoring � Not necessarily as graphs – automatic alarms. � Variance components often ”guestimated” � Many handles to adjust – dangerous � Always combine with your knowledge on animal production. � Well suited for mandatory report