Performance evaluation of Emergency Department patient arrivals forecasting models by including meteorological and calendar information: A comparative study

The volume of daily patient arrivals at Emergency Departments (EDs) is unpredictable and is a significant reason of ED crowding in hospitals worldwide. Timely forecast of patients arriving at ED can help the hospital management in early planning and avoiding of overcrowding. Many different ED patient arrivals forecasting models have previously been proposed by using time series analysis methods. Even though the time series methods such as Linear and Logistic Regression, Autoregressive Integrated Moving Average (ARIMA), Seasonal ARIMA (SARIMA), Exponential Smoothing (ES), and Artificial Neural Network (ANN) have been explored extensively for the ED forecasting model development, the few significant limitations of these methods associated in the analysis of time series data make the models inadequate in many practical situations. Therefore, in this paper, Machine Learning (ML)-based Random Forest (RF) regressor, and Deep Neural Network (DNN)-based Long Short-Term Memory (LSTM) and Convolutional Neural network (CNN) methods, which have not been explored to the same extent as the other time series techniques, are implemented by incorporating meteorological and calendar parameters for the development of forecasting models. The performances of the developed three models in forecasting ED patient arrivals are evaluated. Among the three models, CNN outperformed for short-term (3 days in advance) patient arrivals prediction with Mean Absolute Percentage Error (MAPE) of 9.24% and LSTM performed better for moderate-term (7 days in advance) patient arrivals prediction with MAPE of 8.91% using weather forecast information. Whereas, LSTM model outperformed with MAPE of 8.04% compared to 9.53% by CNN and 10.10% by RF model for current day prediction of patient arrivals using 3 days past weather information. Thus, for short-term ED patient arrivals forecasting, DNN-based model performed better compared to RF regressor ML-based model.

[1]  K. Davidson,et al.  The association of emergency department crowding during treatment for acute coronary syndrome with subsequent posttraumatic stress disorder symptoms. , 2013, JAMA Internal Medicine.

[2]  M. Bech,et al.  Increasing emergency hospital activity in Denmark, 2005–2016: a nationwide descriptive study , 2020, BMJ Open.

[3]  Benjamin C. Sun,et al.  Emergency Department Crowding Predicts Admission Length-of-Stay But Not Mortality in a Large Health System , 2014, Medical care.

[4]  Ronald K. Klimberg,et al.  Forecasting performance measures – what are their practical meaning? , 2010 .

[5]  J. Díaz,et al.  A model for forecasting emergency hospital admissions: effect of environmental variables. , 2001, Journal of environmental health.

[6]  Ward Whitt,et al.  Forecasting arrivals and occupancy levels in an emergency department , 2019, Operations Research for Health Care.

[7]  S. Hajat,et al.  Forecasting daily emergency department visits using calendar variables and ambient temperature readings. , 2013, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[8]  M. Lavieri,et al.  Predicting emergency department volume using forecasting methods to create a "surge response" for noncrisis events. , 2012, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[9]  Cindy Lim,et al.  Forecasting Emergency Department Admissions for Pneumonia in Tropical Singapore , 2018, Online Journal of Public Health Informatics.

[10]  D. Richardson,et al.  Increase in patient mortality at 10 days associated with emergency department overcrowding , 2006, The Medical journal of Australia.

[11]  Karin V Rhodes,et al.  A conceptual model of emergency department crowding. , 2003, Annals of emergency medicine.

[12]  A. Ciampi,et al.  Increases in emergency department occupancy are associated with adverse 30-day outcomes. , 2014, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[13]  Kwai-Sang Chin,et al.  Modeling daily patient arrivals at Emergency Department and quantifying the relative importance of contributing variables using artificial neural network , 2013, Decis. Support Syst..

[14]  A. D. de Craen,et al.  Early prediction of hospital admission for emergency department patients: a comparison between patients younger or older than 70 years , 2017, Emergency Medicine Journal.

[15]  M. Wallis,et al.  Predicting emergency department admissions , 2011, Emergency Medicine Journal.

[16]  Peter Szolovits,et al.  Clinical Intervention Prediction and Understanding with Deep Neural Networks , 2017, MLHC.

[17]  Melik Koyuncu,et al.  Emergency Department Capacity Planning: A Recurrent Neural Network and Simulation Approach , 2019, Comput. Math. Methods Medicine.

[18]  Ali Fuat Guneri,et al.  Planning the future of emergency departments: Forecasting ED patient arrivals by using regression and neural network models , 2016 .

[19]  J. Healy,et al.  Excess winter mortality in Europe: a cross country analysis identifying key risk factors , 2003, Journal of epidemiology and community health.

[20]  Lionel Amodeo,et al.  Forecasting the Emergency Department Patients Flow , 2016, Journal of Medical Systems.

[21]  N. Menke,et al.  A retrospective analysis of the utility of an artificial neural network to predict ED volume. , 2014, The American journal of emergency medicine.

[22]  Jon Pearson,et al.  Forecasting Demand of Emergency Care , 2002, Health Care Management Science.

[23]  Xingyu Zhang,et al.  Prediction of Emergency Department Hospital Admission Based on Natural Language Processing and Neural Networks , 2017, Methods of Information in Medicine.

[24]  Paul Robert Harper,et al.  A hierarchical Bayesian model for improving short‐term forecasting of hospital demand by including meteorological information , 2014 .

[25]  Moslem Yousefi,et al.  Patient visit forecasting in an emergency department using a deep neural network approach , 2019, Kybernetes.

[26]  Ofir Ben-Assuli,et al.  ED Revisits Forecasting: Utilizing Latent Models , 2019, IntelliSys.

[27]  Avishek Choudhury Forecasting Hourly Emergency Department Arrival Using Time Series Analysis , 2019, British Journal of Healthcare Management.

[28]  Sion Jo,et al.  Emergency department occupancy ratio is associated with increased early mortality. , 2014, The Journal of emergency medicine.

[29]  Wen-Hsien Ho,et al.  Long-Term Prediction of Emergency Department Revenue and Visitor Volume Using Autoregressive Integrated Moving Average Model , 2011, Comput. Math. Methods Medicine.

[30]  Alberto Mozo,et al.  Forecasting short-term data center network traffic load with convolutional neural networks , 2018, PloS one.

[31]  A. Asheim,et al.  Real-time forecasting of emergency department arrivals using prehospital data , 2019, BMC Emergency Medicine.

[32]  Jesse M Pines,et al.  Emergency department crowding is associated with poor care for patients with severe pain. , 2008, Annals of emergency medicine.

[33]  Manop Phankokkruad,et al.  An Application of Convolutional Neural Network-Long Short-Term Memory Model for Service Demand Forecasting , 2019, 2019 International Conference on Information and Communications Technology (ICOIACT).

[34]  S. McMillan,et al.  Predicting patient visits to an urgent care clinic using calendar variables. , 2001, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[35]  Haley S Hunter-Zinck,et al.  Predicting emergency department orders with multilabel machine learning techniques and simulating effects on length of stay , 2019, J. Am. Medical Informatics Assoc..

[36]  Steven L Bernstein,et al.  The effect of emergency department crowding on clinically oriented outcomes. , 2009, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[37]  Elif Akçali,et al.  Forecasting Emergency Department Arrivals: A Tutorial for Emergency Department Directors , 2013, Hospital topics.

[38]  Farid Kadri,et al.  Time Series Modelling and Forecasting of Emergency Department Overcrowding , 2014, Journal of Medical Systems.

[39]  Abdellatif El Afia,et al.  Forecasting of weekly patient visits to emergency department: real case study , 2019, Procedia Computer Science.

[40]  Jochen Bergs,et al.  Knowing what to expect, forecasting monthly emergency department visits: A time-series analysis. , 2014, International emergency nursing.

[41]  Li-Jung Liang,et al.  Effect of emergency department crowding on outcomes of admitted patients. , 2013, Annals of emergency medicine.

[42]  Frances S. Shofer,et al.  The effect of emergency department crowding on analgesia in patients with back pain in two hospitals. , 2010, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[43]  E. Ionides,et al.  Forecasting models of emergency department crowding. , 2009, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[44]  K. T. Madavan Nambiar,et al.  Studying the Variability in Patient Inflow and Staffing Trends on Sundays versus Other Days in the Academic Emergency Department , 2017, Journal of emergencies, trauma, and shock.

[45]  Seamus O’Reilly,et al.  Can an Emergency Department-based Clinical Decision Unit successfully utilize alternatives to emergency hospitalization? , 2010, European journal of emergency medicine : official journal of the European Society for Emergency Medicine.

[46]  Yaniv Kerem,et al.  Emergency Department Crowding is Associated with Reduced Satisfaction Scores in Patients Discharged from the Emergency Department , 2013, The western journal of emergency medicine.

[47]  W. Cha,et al.  Association between ED crowding and delay in resuscitation effort. , 2013, The American journal of emergency medicine.

[48]  Flavio S. Fogliatto,et al.  Forecasting Daily Volume and Acuity of Patients in the Emergency Department , 2016, Comput. Math. Methods Medicine.

[49]  Ceren Ocal Tasar,et al.  Modeling and Forecasting the Daily Number of Emergency Department Visits Using Hybrid Models , 2020 .

[50]  Scott L. Zeger,et al.  Predicting Emergency Department Length of Stay Using Quantile Regression , 2009, 2009 International Conference on Management and Service Science.

[51]  Chao-Tung Yang,et al.  Influenza-like illness prediction using a long short-term memory deep learning model with multiple open data sources , 2020, The Journal of Supercomputing.

[52]  R. Campbell,et al.  Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory , 2018, bioRxiv.

[53]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[54]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[55]  S. Zeger,et al.  The challenge of predicting demand for emergency department services. , 2008, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[56]  Filipe de Sá-Soares,et al.  Assessment of forecasting models for patients arrival at Emergency Department , 2017, Operations Research for Health Care.

[57]  Hamid Allaoui,et al.  A stochastic model to minimize patient waiting time in an emergency department , 2018, Operations Research for Health Care.

[58]  Junliang Liu,et al.  Convolutional neural networks for time series classification , 2017 .

[59]  Young-Jin Kim,et al.  An architecture for emergency event prediction using LSTM recurrent neural networks , 2018, Expert Syst. Appl..

[60]  Erkan Celik,et al.  A multi-method patient arrival forecasting outline for hospital emergency departments , 2018, International Journal of Healthcare Management.

[61]  S. Schneider,et al.  Emergency department crowding: a point in time. , 2003, Annals of emergency medicine.

[62]  Brian H Rowe,et al.  The role of a rapid assessment zone/pod on reducing overcrowding in emergency departments: a systematic review , 2011, Emergency Medicine Journal.

[63]  D. Aronsky,et al.  Systematic review of emergency department crowding: causes, effects, and solutions. , 2008, Annals of emergency medicine.

[64]  William K. Mallon,et al.  Financial Impact of Emergency Department Crowding , 2011, The western journal of emergency medicine.

[65]  Ali Fuat Guneri,et al.  Forecasting patient length of stay in an emergency department by artificial neural networks , 2015 .

[66]  Morten Hertzum,et al.  Forecasting Hourly Patient Visits in the Emergency Department to Counteract Crowding , 2017 .

[67]  Raymond Bond,et al.  Using Data Mining to Predict Hospital Admissions From the Emergency Department , 2018, IEEE Access.

[68]  M. Wargon,et al.  From model to forecasting: a multicenter study in emergency departments. , 2010, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[69]  Qi Tian,et al.  DisturbLabel: Regularizing CNN on the Loss Layer , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  L. Rose,et al.  Emergency nurse responsibilities for mechanical ventilation: a national survey. , 2013, Journal of emergency nursing: JEN : official publication of the Emergency Department Nurses Association.

[71]  Peter J. Haug,et al.  A multivariate time series approach to modeling and forecasting demand in the emergency department , 2009, J. Biomed. Informatics.

[72]  HwaMin Lee,et al.  Air Pollution Prediction Using Long Short-Term Memory (LSTM) and Deep Autoencoder (DAE) Models , 2020 .

[73]  Patrick Aboagye-Sarfo,et al.  A comparison of multivariate and univariate time series approaches to modelling and forecasting emergency department demand in Western Australia , 2015, J. Biomed. Informatics.

[74]  E. Kulstad,et al.  Overcrowding is associated with delays in percutaneous coronary intervention for acute myocardial infarction , 2009, International journal of emergency medicine.

[75]  D. Erickson,et al.  Predicting trauma admissions: the effect of weather, weekday, and other variables. , 2009, Minnesota medicine.

[76]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[77]  Frank M Sanfilippo,et al.  Predicting the number of emergency department presentations in Western Australia: A population‐based time series analysis , 2015, Emergency medicine Australasia : EMA.

[78]  Xun Gong,et al.  A Hybrid Method for Traffic Flow Forecasting Using Multimodal Deep Learning , 2018, Int. J. Comput. Intell. Syst..

[79]  Philipp Probst,et al.  To tune or not to tune the number of trees in random forest? , 2017, J. Mach. Learn. Res..

[80]  B. Rechel,et al.  Health Systems in Transition , 2021, Health Management 2.0.

[81]  Ozgur M. Araz,et al.  Using Google Flu Trends data in forecasting influenza-like-illness related ED visits in Omaha, Nebraska. , 2014, The American journal of emergency medicine.

[82]  Ling Tang,et al.  Forecasting Patient Visits to Hospitals using a WD&ANN-based Decomposition and Ensemble Model , 2017 .

[83]  Chuan Zhou,et al.  Forecasting emergency department crowding: a prospective, real-time evaluation. , 2009, Journal of the American Medical Informatics Association : JAMIA.

[84]  Wang-Chuan Juang,et al.  Application of time series analysis in modelling and forecasting emergency department visits in a medical centre in Southern Taiwan , 2017, BMJ Open.

[85]  Grazziela P. Figueredo,et al.  Short and Long term predictions of Hospital emergency department attendances , 2019, Int. J. Medical Informatics.

[86]  S. Hajat,et al.  Heat-related and cold-related deaths in England and Wales: who is at risk? , 2006, Occupational and Environmental Medicine.

[87]  Max Kuhn,et al.  Applied Predictive Modeling , 2013 .

[88]  Jie Li,et al.  Wavelet Decomposition and Convolutional LSTM Networks Based Improved Deep Learning Model for Solar Irradiance Forecasting , 2018, Applied Sciences.

[89]  Ricardo Navares,et al.  Deep learning architecture to predict daily hospital admissions , 2020, Neural Computing and Applications.

[90]  Rae Woong Park,et al.  Prediction of Daily Patient Numbers for a Regional Emergency Medical Center using Time Series Analysis , 2010, Healthcare informatics research.

[91]  Woo Suk Hong,et al.  Predicting hospital admission at emergency department triage using machine learning , 2018, PloS one.

[92]  E. Seow,et al.  Forecasting daily attendances at an emergency department to aid resource planning , 2009, BMC emergency medicine.

[93]  Igi Ardiyanto,et al.  Deep Learning-Based Patient Visits Forecasting Using Long Short Term Memory , 2019, 2019 International Conference of Artificial Intelligence and Information Technology (ICAIIT).

[94]  Shari J. Welch,et al.  Forecasting daily patient volumes in the emergency department. , 2008, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[95]  L. Kinsman,et al.  Emergency department crowding: A systematic review of causes, consequences and solutions , 2018, PloS one.

[96]  Robert Champion,et al.  Forecasting emergency department presentations. , 2007, Australian health review : a publication of the Australian Hospital Association.

[97]  Gad Abraham,et al.  Short-Term Forecasting of Emergency Inpatient Flow , 2009, IEEE Transactions on Information Technology in Biomedicine.

[98]  Wei Jiang,et al.  A Hybrid Approach for Forecasting Patient Visits in Emergency Department , 2016, Qual. Reliab. Eng. Int..