Developing Adaptive Personalized Therapy for Cystic Fibrosis using Reinforcement Learning

Optimal clinical management of inherited chronic diseases, such as Cystic Fibrosis (CF), requires a dynamic approach which updates treatments to cope with the evolving course of illness and to tailor medicines and dosages for individual patient. In this paper, we examine the problem of computing optimal adaptive personalized therapy for CF patients. A temporal difference reinforcement learning method called fitted Q-iteration is utilized to discover the optimal treatment regimen directly from clinical data. We conduct a simulation study of virtual cystic fibrosis patients with Pseudomonas aeruginosa infection and antibiotic therapy with parameters tuned to approximately match published data from CF patients. Our simulation results indicate that reinforcement learning can be an effective tool in developing personalized therapy which optimises the benefit-risk trade off in multi-stage decision making and improves long term outcomes in chronic diseases.

[1]  Karen A Robinson,et al.  Pulmonary Perspective Cystic Fibrosis Pulmonary Guidelines Treatment of Pulmonary Exacerbations , 2009 .

[2]  M. Hodson,et al.  Antibiotic therapy against Pseudomonas aeruginosa in cystic fibrosis: a European consensus. , 2000, The European respiratory journal.

[3]  S. Murphy,et al.  An experimental design for the development of adaptive treatment strategies , 2005, Statistics in medicine.

[4]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[5]  H Putter,et al.  Tutorial in biostatistics: competing risks and multi‐state models , 2007, Statistics in medicine.

[6]  M. Kosorok,et al.  Reinforcement learning design for cancer clinical trials , 2009, Statistics in medicine.

[7]  M. Kosorok,et al.  Acceleration of lung disease in children with cystic fibrosis after Pseudomonas aeruginosa acquisition , 2001, Pediatric pulmonology.

[8]  R. Kronmal,et al.  Advancing outcome measures for the new era of drug development in cystic fibrosis. , 2007, Proceedings of the American Thoracic Society.

[9]  J. Kalbfleisch,et al.  The Analysis of Panel Data under a Markov Assumption , 1985 .

[10]  Karen A Robinson,et al.  Cystic fibrosis pulmonary guidelines: chronic medications for maintenance of lung health. , 2007, American journal of respiratory and critical care medicine.

[11]  J. Emerson,et al.  Early pulmonary infection, inflammation, and clinical outcomes in infants with cystic fibrosis * , 2001, Pediatric pulmonology.

[12]  James M. Robins,et al.  Optimal Structural Nested Models for Optimal Sequential Decisions , 2004 .

[13]  R. Bellman Dynamic programming. , 1957, Science.

[14]  J. Emerson,et al.  Longitudinal assessment of Pseudomonas aeruginosa in young children with cystic fibrosis. , 2001, The Journal of infectious diseases.

[15]  M. Kosorok,et al.  Reinforcement Learning Strategies for Clinical Trials in Nonsmall Cell Lung Cancer , 2011, Biometrics.

[16]  V. Waters,et al.  Combination antimicrobial susceptibility testing for acute exacerbations in chronic infection of Pseudomonas aeruginosa in cystic fibrosis. , 2008, The Cochrane database of systematic reviews.

[17]  A. Smyth,et al.  Antibiotic strategies for eradicating Pseudomonas aeruginosa in people with cystic fibrosis. , 2006, The Cochrane database of systematic reviews.

[18]  Joelle Pineau,et al.  Constructing evidence-based treatment strategies using methods from computer science. , 2007, Drug and alcohol dependence.

[19]  Carmen Cadarso-Suárez,et al.  Multi-state models for the analysis of time-to-event data , 2009, Statistical methods in medical research.

[20]  Margaret Rosenfeld,et al.  Early anti-pseudomonal acquisition in young patients with cystic fibrosis: rationale and design of the EPIC clinical trial and observational study'. , 2009, Contemporary clinical trials.

[21]  S. Rowe,et al.  mechanisms of disease Cystic Fibrosis , 2005 .

[22]  N. Høiby,et al.  Early intervention and prevention of lung disease in cystic fibrosis: a European consensus. , 2004, Journal of cystic fibrosis : official journal of the European Cystic Fibrosis Society.

[23]  Meenu Singh,et al.  Nebulised anti-pseudomonal antibiotics for cystic fibrosis. , 2003, The Cochrane database of systematic reviews.

[24]  P. McCray,et al.  Pathogenesis of Early Lung Disease in Cystic Fibrosis: A Window of Opportunity To Eradicate Bacteria , 2005, Annals of Internal Medicine.

[25]  J. Dankert-Roelse,et al.  Newborn screening for cystic fibrosis. , 2009, The Cochrane database of systematic reviews.

[26]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[27]  Susan A. Murphy,et al.  A Generalization Error for Q-Learning , 2005, J. Mach. Learn. Res..

[28]  R. Kronmal,et al.  Association between pulmonary function and sputum biomarkers in cystic fibrosis. , 2007, American journal of respiratory and critical care medicine.

[29]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[30]  C. Watkins Learning from delayed rewards , 1989 .

[31]  R. Gibson,et al.  Approach to eradication of initial Pseudomonas aeruginosa infection in children with cystic fibrosis , 2007, Pediatric pulmonology.

[32]  Massimiliano Mascherini,et al.  Early eradication therapy against Pseudomonas aeruginosa in cystic fibrosis patients , 2005, European Respiratory Journal.

[33]  P. Sly,et al.  Acquisition and eradication of P. aeruginosa in young children with cystic fibrosis , 2008, European Respiratory Journal.

[34]  R. Gibson,et al.  Pathophysiology and management of pulmonary infections in cystic fibrosis. , 2003, American journal of respiratory and critical care medicine.

[35]  Joelle Pineau,et al.  Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning , 2008, AAAI.

[36]  Louis Wehenkel,et al.  Clinical data based optimal STI strategies for HIV: a reinforcement learning approach , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[37]  N. Høiby,et al.  Prevention of chronic Pseudomonas aeruginosa colonisation in cystic fibrosis by early treatment , 1991, The Lancet.

[38]  M. Kosorok,et al.  Longitudinal development of mucoid Pseudomonas aeruginosa infection and lung disease progression in children with cystic fibrosis. , 2005, JAMA.

[39]  Alexander J. Smola,et al.  Support Vector Method for Function Approximation, Regression Estimation and Signal Processing , 1996, NIPS.

[40]  G. Retsch-Bogart Update on new pulmonary therapies , 2009, Current opinion in pulmonary medicine.

[41]  A. Prince Biofilms, antimicrobial resistance, and airway infection. , 2002, The New England journal of medicine.

[42]  F. Ratjen,et al.  Effect of inhaled tobramycin on early Pseudomonas aeruginosa colonisation in patients with cystic fibrosis , 2001, The Lancet.

[43]  J M Robins,et al.  Marginal Mean Models for Dynamic Regimes , 2001, Journal of the American Statistical Association.

[44]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.

[45]  J. Emerson,et al.  Significant microbiological effect of inhaled tobramycin in young children with cystic fibrosis. , 2003, American journal of respiratory and critical care medicine.

[46]  M S Pepe,et al.  Intermittent administration of inhaled tobramycin in patients with cystic fibrosis. Cystic Fibrosis Inhaled Tobramycin Study Group. , 1999, The New England journal of medicine.

[47]  P. Farrell,et al.  Longitudinal evaluation of bronchopulmonary disease in children with cystic fibrosis , 2003, Pediatric pulmonology.

[48]  S. Murphy,et al.  Developing adaptive treatment strategies in substance abuse research. , 2007, Drug and alcohol dependence.

[49]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[50]  J. Emerson,et al.  Diagnostic accuracy of oropharyngeal cultures in infants and young children with cystic fibrosis , 1999, Pediatric pulmonology.

[51]  N. Høiby,et al.  Diagnosis of chronic Pseudomonas aeruginosa infection in cystic fibrosis by enzyme-linked immunosorbent assay , 1987, Journal of clinical microbiology.

[52]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[53]  J. Carlin,et al.  Bronchoalveolar lavage or oropharyngeal cultures to identify lower respiratory pathogens in infants with cystic fibrosis , 1996, Pediatric pulmonology.

[54]  J. Elborn,et al.  Clinical trials in cystic fibrosis. , 2007, Journal of cystic fibrosis : official journal of the European Cystic Fibrosis Society.

[55]  H. Sung,et al.  Selecting Therapeutic Strategies Based on Efficacy and Death in Multicourse Clinical Trials , 2002 .