Biosens II Background → Biosens II → Subproject 2.3 Optimal replacement policies for dairy cows based on Overview daily yield measurements MDP intro Dairy HMDP Model Case example: Markov decision processes State space model Embedding the SSM ☞ Biosens II: Improved monitoring and management of dairy Lars R. Nielsen ∗ and Erik Jørgensen HMDP results production based on on-farm biosensors Dept. of Genetics and Biotechnology, University of Aarhus, Denmark Status/future work ☞ Goal: Better detection of oestrus and illnesses Søren Østergaard ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Dept. of Animal Health, Welfare and Nutrition, University of Aarhus, Denmark Anders R. Kristensen ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Dept. of Large Animal Sciences, University of Copenhagen, Denmark ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Commercial product Herd Navigator TM based on Biosens project ∗ Contact: lars@relund.dk ( www.research.relund.dk ) ( www.herdnavigator.com ) Life – Oct 16’th 2009 – 2 / 19 Project 2.3: Economic value of the dairy cow Problem, models and results Find optimal strategy for each cow w.r.t. replacement, treatment and reproduction Problem (economic point of view). Background ☞ Assign an economic value to a dairy cow during lactation Overview → Problem and model ☞ Many papers about the dairy cow replacement problem but limited use in pratice. ☞ Calculate the optimal replacement strategy based on the MDP intro Reasons could be: economic value Dairy HMDP Model ☞ Assume daily yield measurements available State space model - Often large and complex models. Embedding the SSM Models - Parameters in the model must be estimated, i.e. data collection frameworks HMDP results ☞ Use a state space model (SSM) for predicting daily milk yield at herd level must exist. Status/future work - Stage length: one month up to a year → no assistance when to inseminate, ☞ Use a Markov decision process (MDP) for calculating the optimal treat or cull the cow in the current month. strategy with the SSM embedded ☞ Bio-sensors and cow specific traits/interventions exists in modern dairy herds → Results parameters can be estimated on a daily basis. ☞ A strategy saying whether to replace or keep the cow given its current state Develop MDP with daily stages based on daily yield measurements. ☞ An economic value of the cow and forecast of the yield. Life – Oct 16’th 2009 – 3 / 19 Life – Oct 16’th 2009 – 4 / 19
What is an MDP? What is an MDP? Background 1 2 3 4 Background 1 2 3 4 Overview Overview MDP intro MDP intro → What is an MDP? → What is an MDP? S 1 S 2 S 3 S 1 S 2 S 3 S 4 S 4 → Hierarchical MDP → Hierarchical MDP Dairy HMDP Model Dairy HMDP Model s 1 , 1 s 1 , 2 s 1 , 3 s 1 , 4 s 1 , 1 s 1 , 2 s 1 , 3 s 1 , 4 State space model State space model Embedding the SSM Embedding the SSM p HMDP results HMDP results r Status/future work Status/future work s 2 , 1 s 2 , 2 s 2 , 3 s 2 , 4 s 2 , 1 s 2 , 2 s 2 , 3 s 2 , 4 s 3 , 1 s 3 , 2 s 3 , 3 s 3 , 4 s 3 , 1 s 3 , 2 s 3 , 3 s 3 , 4 Life – Oct 16’th 2009 – 5 / 19 Life – Oct 16’th 2009 – 5 / 19 What is an MDP? Hierarchical MDP (HMDP) Background 1 2 3 4 Background Level 0 child process Overview Overview MDP intro MDP intro child process → What is an MDP? → What is an MDP? S 1 S 2 S 3 S 4 → Hierarchical MDP → Hierarchical MDP Level 1 Dairy HMDP Model Dairy HMDP Model child processes s 1 , 1 s 1 , 2 s 1 , 3 s 1 , 4 State space model State space model child process child process Embedding the SSM Embedding the SSM HMDP results HMDP results Level 2 Status/future work Status/future work s 2 , 1 s 2 , 2 s 2 , 3 s 2 , 4 s 3 , 1 s 3 , 2 s 3 , 3 s 3 , 4 Life – Oct 16’th 2009 – 5 / 19 Life – Oct 16’th 2009 – 6 / 19
Lactation cycle of the cow Dairy model overview Calving ☞ Formulate a hierarchical MDP (HMDP) based on lactation cycles Background Background Overview Overview of the cow. Start insem MDP intro MDP intro Dry period ☞ Infinite time-horizon, Daily stages, 3 levels Dairy HMDP Model Dairy HMDP Model Oesteus → Lactation cycle → Lactation cycle ☞ Decisions Replace, Keep and Dry → Model overview → Model overview ☞ Maximize the discounted net present reward of the cow → State variables → State variables → Rewards and prob → Rewards and prob State space model State space model Level new cow new cow Pregnant Embedding the SSM Embedding the SSM cow 1 cow 2 cow 3 cow 4 cow 5 0 HMDP results HMDP results Status/future work Status/future work replace calving calving replace parity 1 parity 2 parity 3 parity 10 1 Pregnancy Lactation Slut insem dry insemination insemination starts ends 2 Life – Oct 16’th 2009 – 7 / 19 Life – Oct 16’th 2009 – 8 / 19 Rewards and transition probabilities SSM models Transition probabilities are a found using ☞ ... or dynamic linear models are models of phenomena evolving in Background Background Overview Overview time e.g. blood pressure and milk yield. ☞ The SSM milk yield model MDP intro MDP intro ☞ A reproduction model Dairy HMDP Model Dairy HMDP Model Y t-1 Y t Y t + 1 → Lactation cycle ☞ An IC model State space model → Model overview → SSM Formulation → State variables → Yield SSM → Rewards and prob The net reward is a combination of → Kalman filter State space model Embedding the SSM ☞ Milk yield Embedding the SSM HMDP results HMDP results ☞ The calf ☞ Latent process evolves as a first order Markov process. Status/future work Status/future work ☞ Beef θ t = Gθ t − 1 + ω t , ( θ t | θ t − 1 ) ∼ N ( Gθ t − 1 , W ) ☞ Feeding and treatment costs ☞ Y t are observations which we model as a function depending on θ t . Y t = F ′ θ t + ν t , ( Y t | θ t ) ∼ N � F ′ θ t , V � Life – Oct 16’th 2009 – 10 / 19 Life – Oct 16’th 2009 – 11 / 19
Milk yield SSM Milk yield SSM Observed milk yield intensity Observed milk yield intensity 50 100 150 200 250 50 100 150 200 250 Background Background 1 2 3 1 2 3 60 60 Overview Overview M tc = µ t + A c + X tc + ν tc M tc = µ t + A c + X tc + ν tc 50 50 MDP intro MDP intro 40 Subtract herd effect (remove index c ) 40 Dairy HMDP Model Dairy HMDP Model milk yield milk yield 30 30 State space model State space model Y t = M t − µ t = F ′ θ t + ν t → SSM Formulation → SSM Formulation 20 20 � A → Yield SSM → Yield SSM � 10 10 → Kalman filter → Kalman filter M t µ t = ( 1 1 ) + ν t M t µ t 0 X t 0 Embedding the SSM Embedding the SSM 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 t t HMDP results HMDP results θ t = Gθ t − 1 + ω t � 1 � 0 Status/future work Status/future work � � � � 0 A = + 0 ρ X t − 1 ǫ t where ( θ t | Y 0 , . . . , Y t ) ∼ N ( m t , C t ) Life – Oct 16’th 2009 – 12 / 19 Life – Oct 16’th 2009 – 12 / 19 Milk yield SSM Applying the Kalman filter Observed milk yield intensity D t − 1 : data up to time t − 1 . Fact: 50 100 150 200 250 Background Background 1 2 3 60 ( θ t − 1 | D t − 1 ) ∼ N ( m t − 1 , C t − 1 ) Overview M tc = µ t + A c + X tc + ν tc Overview 50 MDP intro MDP intro Subtract herd effect (remove index c ) 40 Dairy HMDP Model Dairy HMDP Model milk yield Data Y est. θ est. 30 State space model State space model Y t = M t − µ t = F ′ θ t + ν t → SSM Formulation → SSM Formulation 20 0.2 � A → Yield SSM → Yield SSM � 10 → Kalman filter → Kalman filter 0.0 = ( 1 1 ) + ν t M t µ t X t 0 Embedding the SSM Embedding the SSM −0.2 50 100 150 200 250 50 100 150 200 250 t HMDP results HMDP results θ t = Gθ t − 1 + ω t −0.4 50 100 150 200 250 � 1 � 0 Status/future work Status/future work 1 2 3 � � � � 30 0 A −0.6 = + 20 0 ρ X t − 1 ǫ t −0.8 10 residual milk yield where −1.0 0 −10 0 5 10 15 20 25 30 35 ( θ t | Y 0 , . . . , Y t ) ∼ N ( m t , C t ) −20 Y t E( Y t | D t − 1 ) E( A 3 | D t − 1 ) TFC 50 100 150 200 250 50 100 150 200 250 t Life – Oct 16’th 2009 – 12 / 19 Life – Oct 16’th 2009 – 13 / 19
More recommend
