12/17/2019 Department of Veterinary and Animal Sciences - PDF document

12/17/2019 Department of Veterinary and Animal Sciences Hierarchical Markov decision processes Original slides by Anders Ringgaard Kristensen Presented by Dan Børge Jensen Department of Veterinary and Animal Sciences Outline Quick summary of Monday The markov property – revisited Graphical representation of models Hierarchical models Multi-level models Decisions on multiple time scale Markov chain simulation Advanced Quantitative Methods in Herd Management Slide 2 Department of Veterinary and Animal Sciences Summary from Monday The Markov property: only the current state affects future states! • Optimization goal: find the best policy (decision strategy) Different objective functions • • Sum of (discounted) rewards over time • Average reward per time unit • Average reward per unit of product Optimization methods: Value iteration • • Exact for finite time horizons • Non-exact for infinite time horizons Policy iteration • • Exact for infinite time horizon • Can not handle very large state spaces Advanced Quantitative Methods in Herd Management Slide 3 1

12/17/2019 Department of Veterinary and Animal Sciences The Markov property Let i n be the state at stage n The Markov property is satisfied if, and only if, • P( i n+ 1 | i n , i n- 1 , … , i 1 ) = P ( i n+ 1 | i n ) • In words: The distribution of the state at next stage depends only on the present state – previous states are not relevant. This property is crucial in Markov decision processes. Advanced Quantitative Methods in Herd Management Slide 4 Department of Veterinary and Animal Sciences Markov property: Example Litter size in sows: • Litter size in sows may be represented as a multi dimensional normal distribution from previous exercise. • We wish to predict litter size of parity n • How shall we define the state space in order to fulfill the Markov property? Slide 5 Department of Veterinary and Animal Sciences Markovian prediction of litter size I Straight forward solution: • Include previous litter sizes as part of the current state • For a sow in parity 8 this means e.g. 15 8 = 2.5 x 10 9 state combinations. • Prohibitive! • Trick most often used in practice: • Only include the 2 – 3 most recent litter size results. Slide 6 2

12/17/2019 The model tree Department of Veterinary and Animal Sciences Graphical representation of MDPs Recall the structure of the simple dairy cow replacement model: Stage: • 1 lactation cycle State: • i= 1: Low milk yield • i= 2: Average milk yield • i= 3: High milk yield Action: • d =1: Keep the cow • d =2: Replace the cow at the end of the stage The structure may be displayed graphically as a model tree We will model 10 stages (finite horizon) Advanced Quantitative Methods in Herd Management Slide 8 Department of Veterinary and Animal Sciences The model displayed as a tree We have a nested structure: • The root of the model is the process itself • The process holds 10 stages (the time horizon) • Each stage holds 3 states (Low, Average, High) • Each state holds 2 actions (Keep, Replace) The parameters: • Each action has a set of parameters attached: • A reward • A probability distribution (to the states at next stage). Implemented in the MLHMP software system Advanced Quantitative Methods in Herd Management Slide 9 3

12/17/2019 Department of Veterinary and Animal Sciences The MLHMP software – the model tree window Advanced Quantitative Methods in Herd Management Slide 10 Department of Veterinary and Animal Sciences Summary of the model tree The nested structure of an MDP is shown directly Each value (stage, state and action) is displayed as an icon of a certain type. A label is (optionally) attached to each value in order to ease the (human) interpretation of the values. Asymmetric models are easily handled (and displayed) Advanced Quantitative Methods in Herd Management Slide 11 5-10 Minute Break 4

12/17/2019 The curse of dimensionality Department of Veterinary and Animal Sciences Age dependency of milk yield 6000 5800 5600 5400 5200 5000 Kg ECM 4800 4600 4400 1st 2nd 3rd 4th Parity Parity Parity Parity Advanced Quantitative Methods in Herd Management Slide 14 Department of Veterinary and Animal Sciences An extended model, I State variables • Age • Parity 1 • Parity 2 • Parity 3 • Parity 4 • Relative milk yield • Low • Average • High Advanced Quantitative Methods in Herd Management Slide 15 5

12/17/2019 Department of Veterinary and Animal Sciences An extended model, II Advanced Quantitative Methods in Herd Management Slide 16 Department of Veterinary and Animal Sciences An extended model, III Advanced Quantitative Methods in Herd Management Slide 17 Department of Veterinary and Animal Sciences An extended model, IV Advanced Quantitative Methods in Herd Management Slide 18 6

12/17/2019 Department of Veterinary and Animal Sciences Let us take a look at the model tree Advanced Quantitative Methods in Herd Management Slide 19 Department of Veterinary and Animal Sciences Age and genotype dependency 7000 6000 Low genetic merit 5000 4000 Average genetic 3000 merit 2000 High genetic merit 1000 0 Par. 1 Par. 2 Par. 3 Par. 4 Advanced Quantitative Methods in Herd Management Slide 20 Department of Veterinary and Animal Sciences A further extended model State variables • Genetic merit: • Low, • Average, • High • Age: • Parity 1, • Parity 2, • Parity 3, • Parity 4 • (Relative) milk yield: • Low, • Average, • High 7

12/17/2019 Department of Veterinary and Animal Sciences Rewards and output Department of Veterinary and Animal Sciences Transition probabilities, Keep Department of Veterinary and Animal Sciences Transition probabilities, Replace 8

12/17/2019 Department of Veterinary and Animal Sciences We shall again take a look at the graphical display Department of Veterinary and Animal Sciences An example: Houben et al. (1994) State variables: • Age (monthly intervals, 204 levels) • Milk yield, present lactation (15 levels) • Milk yield, previous lactation (15 levels) • Length of calving interval (8 levels) • Mastitis, present lactation (4 levels) • Mastitis, previous lactation (4 levels) • Clinical mastitis (yes/no) Total state space 6,821,724 states Houben, E. P. H., R. B. M. Huirne, A. A. Dijkhuizen & A. R. Kristensen. 1994. Optimal replacement of mastitis cows determined by a hierarchic Markov process. Journal of Dairy Science 77 , 2975-2993. Department of Veterinary and Animal Sciences The curse of dimensionality If • state variables are represented at a realistic number of levels • all relevant state variables are included in the model then • the state space grows to prohibitive dimensions Solution: • Hierarchical models Advanced Quantitative Methods in Herd Management Slide 27 9

12/17/2019 5-10 Minute Break Hierachical Markov Decision Processes Department of Veterinary and Animal Sciences Important observations, transition matrix Most elements are zero because • Age is included as a state variable • Some state variables are constant within animal • Some state variables are constant over several stages If state numbers are defined appropriately the non-zero elements are arranged in a certain pattern This can be utilized for a hierarchical organisation of the state space! Advanced Quantitative Methods in Herd Management Slide 30 10

12/17/2019 Department of Veterinary and Animal Sciences Illustration of the hierarchy for the example Genetic merit Founder Cow 1 Cow 2 Cow 3 Dummy (no action) Relative milk yield Child 3 4 3 4 1 2 1 2 1 2 Keep/Replace Optimization technique • Policy iteration in the founder process (exact) • Value iteration in the child processes (exact) The positive properties of both techniques are combined into a very efficient and exact hierarchic technique Department of Veterinary and Animal Sciences The dairy cow replacement model as a hierarchical process Founder process: • Stage : Life time of a cow • State : Genetic merit • Action : Dummy Child process: • Stage : A lactation cycle • State : Milk yield (relative to genetic merit and lactation) • Action : Keep/Replace Benefits: • The age of the cow is known from the child level stage • The size of the transition matrices are reduced to 3 x 3 (as compared to 36 x 36 in the original model) Advanced Quantitative Methods in Herd Management Slide 32 Department of Veterinary and Animal Sciences Multi-level processes The hierarchy may be extended to several levels Curse of dimensionality circumvented Simultaneous optimization of decisions at different levels (time horizon) Advanced Quantitative Methods in Herd Management Slide 33 11