Markov Limid processes for representing and solving renewal problems

In this paper a new tool for simultaneous optimisation of decisions on multiple time scales is presented. The tool combines the dynamic properties of Markov decision processes with the flexible and compact state space representation of LImited Memory Influence Diagrams (Limids). A temporal version of Limids, TemLimids, is defined by adding time-related functions to utility nodes. As a result, expected discounted utility, as well as expected relative utility might be used as optimisation criteria in TemLimids. Optimisation proceeds as in ordinary Limids. A sequence of such TemLimids can be used to model a Markov Limid Process, where each TemLimid represents a macro action. Algorithms are presented to find optimal plans for a sequence of such macro actions. Use of algorithms is illustrated based on an extended version of an example from pig production originally used to introduce the Limid concept.

[1]  Anders Ringgaard Kristensen,et al.  Hierarchic Markov processes and their applications in replacement models , 1988 .

[2]  A R Kristensen,et al.  Optimal replacement of mastitic cows determined by a hierarchic Markov process. , 1994, Journal of dairy science.

[3]  Anders L. Madsen,et al.  Solving Influence Diagrams using HUGIN, Shafer-Shenoy and Lazy Propagation , 2001, UAI.

[4]  Ross D. Shachter,et al.  Dynamic programming and influence diagrams , 1990, IEEE Trans. Syst. Man Cybern..

[5]  Michael Höhle,et al.  Methods for evaluating Decision Problems with Limited Information , 2005 .

[6]  Peter Green,et al.  Highly Structured Stochastic Systems , 2003 .

[7]  Leslie Pack Kaelbling,et al.  Planning and Acting in Partially Observable Stochastic Domains , 1998, Artif. Intell..

[8]  Anders Ringgaard Kristensen.  A general software system for Markov decision processes in herd management applications , 2003 .

[9]  Dina Kvl,et al.  Multi-level hierarchic Markov processes as a framework for herd management support , 1996 .

[10]  Jim Q. Smith.  Influence diagrams for Bayesian decision analysis , 1989 .

[11]  Ruud B.M. Huirne,et al.  Economic optimization of dairy heifer management decisions , 1999 .

[12]  R. M. Oliver,et al.  Influence diagrams, belief nets and decision analysis , 1992 .

[13]  Michael I. Jordan,et al.  Probabilistic Networks and Expert Systems , 1999 .

[14]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[15]  T. Larsen,et al.  A model for detection of individual cow mastitis based on an indicator measured in milk. , 2006, Journal of dairy science.

[16]  Ronald A. Howard,et al.  Readings on the Principles and Applications of Decision Analysis , 1989 .

[17]  James C. Cox,et al.  Quantifying the Effects of Sow‐Herd Management Information Systems on Farmers' Decision Making Using Experimental Economics , 1998 .

[18]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[19]  Søren Højsgaard,et al.  Embedding a state space model into a Markov decision process , 2011, Ann. Oper. Res..

[20]  Anders Ringgaard Kristensen,et al.  A model for monitoring the condition of young pigs by their drinking behaviour , 2005 .

[21]  Gudbrand Lien,et al.  Optimal length of leys in an area with winter damage problems , 2003 .

[22]  Erik Jørgensen,et al.  Multi‐level hierarchic Markov processes as a framework for herd management support , 2000, Ann. Oper. Res..

[23]  Anders Ringgaard Kristensen.  Bayesian updating in hierarchic Markov processes applied to the animal replacement problem , 1993 .

[24]  A. Kristensen,et al.  A sow replacement model using Bayesian updating in a 3-level hierarchic Markov process I . Biological model , 2001 .

[25]  Steffen L. Lauritzen,et al.  Evaluating Influence Diagrams using LIMIDs , 2000, UAI.

[26]  Steffen L. Lauritzen,et al.  Representing and Solving Decision Problems with Limited Information , 2001, Manag. Sci..

[27]  W Ouweltjes,et al.  Detection model for mastitis in cows milked in an automatic milking system. , 2001, Preventive veterinary medicine.

[28]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[29]  Frank Jensen,et al.  From Influence Diagrams to junction Trees , 1994, UAI.

[30]  George R. Terre.  Influence Diagrams, Belief Nets, and Decision Analysis , 2012 .

[31]  Prakash P. Shenoy,et al.  Valuation-Based Systems for Bayesian Decision Analysis , 1992, Oper. Res..

[32]  Steffen L. Lauritzen,et al.  Some modern applications of graphical models , 2003 .

[33]  Anders Ringgaard Kristensen,et al.  A sow replacement model using Bayesian updating in a three-level hierarchic Markov process: II. Optimization model , 2004 .

[34]  W. Lovejoy.  A survey of algorithmic methods for partially observed Markov decision processes , 1991 .

[35]  Ross D. Shachter.  Evaluating Influence Diagrams , 1986, Oper. Res..

[36]  Scott M. Olmsted.  On representing and solving decision problems , 1983 .