Markov Decision Processes: Biosens II E. Jrgensen & Lars R. - PDF document

Markov Decision Processes: Biosens II E. Jørgensen & Lars R. Nielsen Department of Genetics and Biotechnology Faculty of Agricultural Sciences, University of Århus 10/10 2008

Background: Markov Decision Processes Background: Biosens project Examples • Electronic ID and recording at each milking • Robot milking • Dairy Replacement models • Yield, temperature • In-line laboratory • Sow Replacement models • Lactate DeHydrogenase (LDH) • Delivery strategy • Electric Conductivity (EC) • . . . • Somatic cell counts • Progesterone • . . . Biosens II Background: Biosens project Methods Background → Biosens II → Subproject 2.3 → MDPs in dairy MDP intro Dairy HMDP Model • Kalman-filter and DLM/State Space Models • Multiprocess: outlier and trend model ☞ Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors • Decisions via heuristic decision rules ☞ Goal: Better detection of oestrus and illnesses • Implemented in Herd navigator ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Commercial product Herd Navigator TM based on Biosens project ( www.herdnavigator.com ) OR50 – Sep. 11’th 2008 – 2 / 13 Biosens II Biosens II Background Background → Biosens II → Biosens II → Subproject 2.3 → Subproject 2.3 → MDPs in dairy → MDPs in dairy MDP intro MDP intro Dairy HMDP Model Dairy HMDP Model ☞ Biosens II: Improved monitoring and management of dairy ☞ Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors production based on on-farm biosensors ☞ Goal: Better detection of oestrus and illnesses ☞ Goal: Better detection of oestrus and illnesses ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Commercial product Herd Navigator TM based on Biosens project ☞ Commercial product Herd Navigator TM based on Biosens project ( www.herdnavigator.com ) ( www.herdnavigator.com ) OR50 – Sep. 11’th 2008 – 2 / 13 OR50 – Sep. 11’th 2008 – 2 / 13

Biosens II Biosens II Background Background → Biosens II → Biosens II → Subproject 2.3 → Subproject 2.3 → MDPs in dairy → MDPs in dairy MDP intro MDP intro Dairy HMDP Model Dairy HMDP Model ☞ Biosens II: Improved monitoring and management of dairy ☞ Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors production based on on-farm biosensors ☞ Goal: Better detection of oestrus and illnesses ☞ Goal: Better detection of oestrus and illnesses ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Commercial product Herd Navigator TM based on Biosens project ☞ Commercial product Herd Navigator TM based on Biosens project ( www.herdnavigator.com ) ( www.herdnavigator.com ) OR50 – Sep. 11’th 2008 – 2 / 13 OR50 – Sep. 11’th 2008 – 2 / 13 Biosens II Use decision theoretical for optimizing of decisions BioSens II Background → Biosens II → Subproject 2.3 → MDPs in dairy MDP intro • Base decisions on herd specific parameters Dairy HMDP Model • Optimize treatment strategies on cow level (economic point of view). ☞ Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors • Need model for predicting milk yield, degree of infection, ☞ Goal: Better detection of oestrus and illnesses oestrus → state space models (SSMs). ☞ Focus on biomarkers in milk (progesterone, LDH, yield, etc.) • Need model for calculating the optimal strategy → Markov decision processes (MDPs). ☞ Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) ☞ Five year project (2007-2011). Budget ≈ 5 mill EUR ☞ Commercial product Herd Navigator TM based on Biosens project ( www.herdnavigator.com ) OR50 – Sep. 11’th 2008 – 2 / 13 Use decision theoretical for optimizing of decisions Project Phases BioSens II Phases • Base decisions on herd specific parameters • Optimize treatment strategies on cow level (economic • Extend existing Dairy replacement models. point of view). • Include decisions and registrations related to reproduction • Need model for predicting milk yield, degree of infection, • Include decisions and registrations related to disease oestrus → state space models (SSMs). (Mastitis) • Need model for calculating the optimal strategy → Markov decision processes (MDPs).

MDPs applied to dairy Model components Phase 1 ☞ Many papers about the dairy cow replacement problem but limited Background → Biosens II use in pratice. Reasons could be: → Subproject 2.3 → MDPs in dairy - Often large and complex models with no friendly user MDP intro interface. Dairy HMDP Model • Daily yield - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. • Reproduction - Stage length: one month up to a year → no assistance when • Involuntary culling to inseminate, treat or cull the cow in the current month. • Growth and economic parameters ☞ Bio-sensors and cow specific traits/interventions exists in modern dairy herds → parameters can be estimated on a daily basis. OR50 – Sep. 11’th 2008 – 4 / 13 MDPs applied to dairy MDPs applied to dairy ☞ Many papers about the dairy cow replacement problem but limited ☞ Many papers about the dairy cow replacement problem but limited Background Background → Biosens II → Biosens II use in pratice. Reasons could be: use in pratice. Reasons could be: → Subproject 2.3 → Subproject 2.3 → MDPs in dairy → MDPs in dairy - Often large and complex models with no friendly user - Often large and complex models with no friendly user MDP intro MDP intro interface. interface. Dairy HMDP Model Dairy HMDP Model - Parameters in the model must be estimated, i.e. data - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. collection frameworks at herd level must exist. - Stage length: one month up to a year → no assistance when - Stage length: one month up to a year → no assistance when to inseminate, treat or cull the cow in the current month. to inseminate, treat or cull the cow in the current month. ☞ Bio-sensors and cow specific traits/interventions exists in modern ☞ Bio-sensors and cow specific traits/interventions exists in modern dairy herds → parameters can be estimated on a daily basis. dairy herds → parameters can be estimated on a daily basis. OR50 – Sep. 11’th 2008 – 4 / 13 OR50 – Sep. 11’th 2008 – 4 / 13 MDPs applied to dairy MDPs applied to dairy ☞ Many papers about the dairy cow replacement problem but limited ☞ Many papers about the dairy cow replacement problem but limited Background Background → Biosens II → Biosens II use in pratice. Reasons could be: use in pratice. Reasons could be: → Subproject 2.3 → Subproject 2.3 → MDPs in dairy → MDPs in dairy - Often large and complex models with no friendly user - Often large and complex models with no friendly user MDP intro MDP intro interface. interface. Dairy HMDP Model Dairy HMDP Model - Parameters in the model must be estimated, i.e. data - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. collection frameworks at herd level must exist. - Stage length: one month up to a year → no assistance when - Stage length: one month up to a year → no assistance when to inseminate, treat or cull the cow in the current month. to inseminate, treat or cull the cow in the current month. ☞ Bio-sensors and cow specific traits/interventions exists in modern ☞ Bio-sensors and cow specific traits/interventions exists in modern dairy herds → parameters can be estimated on a daily basis. dairy herds → parameters can be estimated on a daily basis. OR50 – Sep. 11’th 2008 – 4 / 13 OR50 – Sep. 11’th 2008 – 4 / 13