A modeling framework for replacing medical therapies

A common application of Markov Decision Processes (MDPs) is to determine when to replace machines that stochastically degrade over time. Typical assumptions are that there exists an infinite supply of identical replacements, each of which, upon installation, immediately renews the system to the best state, from which stochastic deterioration proceeds. This paper considers a situation for which these assumptions no longer apply: the replacement of medical therapies so as to maximize a patient's expected lifetime (or quality-adjusted lifetime). For example, upon taking HIV therapy, levels of the virus gradually decrease; however, viral resistance accrues over time, leading to an eventual increase of viral levels. This prompts a switch in therapy; however, there are only a finite number of effective therapies. A model is presented that addresses these challenges to a stationary MDP framework and a general algorithm for scheduling therapies is discussed.

[1]  Gordon Johnston,et al.  Statistical Models and Methods for Lifetime Data , 2003, Technometrics.

[2]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[3]  J. Ben Atkinson,et al.  Modeling and Analysis of Stochastic Systems , 1996 .

[4]  A. N,et al.  Dynamic Multidrug Therapies for HIV : A Control Theoretic Approach , 1997 .

[5]  Michael Pinedo,et al.  Scheduling: Theory, Algorithms, and Systems , 1994 .

[6]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[7]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[8]  Wallace J. Hopp,et al.  Technical Note - Identifying Forecast Horizons in Nonhomogeneous Markov Decision Processes , 1989, Oper. Res..

[9]  Wallace J. Hopp,et al.  Machine Maintenance with Multiple Maintenance Actions , 1990 .

[10]  Andrew J. Schaefer,et al.  The Optimal Time to Initiate HIV Therapy Under Ordered Health States , 2008, Oper. Res..

[11]  J. J. Henning,et al.  Guidelines for the Use of Antiretroviral Agents in HIV-Infected Adults and Adolescents, January 28, 2000 , 1998, HIV clinical trials.

[12]  G. J. Lieberman,et al.  On the Use of Replacements to Extend System Life , 1984, Oper. Res..

[13]  Uri Yechiali,et al.  OPTIMALITY OF CONTROL LIMIT MAINTENANCE POLICIES UNDER NONSTATIONARY DETERIORATION , 1999, Probability in the Engineering and Informational Sciences.

[14]  Andrew J. Schaefer,et al.  Determining the Acceptance of Cadaveric Livers Using an Implicit Model of the Waiting List , 2007, Oper. Res..

[15]  M. Klein.  Inspection—Maintenance—Replacement Schedules Under Markovian Deterioration , 1962 .

[16]  Jerald F. Lawless,et al.  Statistical Models and Methods for Lifetime Data. , 1983 .

[17]  I. Langner.  Survival Analysis: Techniques for Censored and Truncated Data , 2006 .

[18]  C. Bes,et al.  Concepts of Forecast and Decision Horizons: Applications to Dynamic Stochastic Optimization Problems , 1986, Math. Oper. Res..

[19]  Andrew J. Schaefer,et al.  Modeling Medical Treatment Using Markov Decision Processes , 2005 .

[20]  Robert L. Smith,et al.  Conditions for the Existence of Planning Horizons , 1984, Math. Oper. Res..

[21]  Z. A. Lomnicki,et al.  Mathematical Theory of Reliability , 1966 .

[22]  Andrew J. Schaefer,et al.  The Optimal Timing of Living-Donor Liver Transplantation , 2004, Manag. Sci..

[23]  Robert L. Smith,et al.  A New Optimality Criterion for Nonhomogeneous Markov Decision Processes , 1987, Oper. Res..