Evaluation of Event Impact on Key Performance Indicators

Monitoring and understanding the performance of equipment and operations is a real and challenging problem in industrial and commercial settings. Although many operations have a set of agreed-upon Key Performance Indicators (KPIs) that have been defined by domain experts, most of these KPIs suffer from many issues including but not limited to sensitivity to seasonality and load conditions, approximate and inaccurate calculations, and measurement noise. This makes further analysis of these measurements a real challenge. This paper focuses on answering a simple yet crucial question in regards to KPI monitoring: "Have a given event made an impact on the performance of an operation?". The paper presents an approach to answer this question in a systematic and statistically-sound way while alleviating the aforementioned problems with the provided KPIs. The key idea behind the proposed approach is to predict the would-be performance as if the event has not happened. The approach then applies a three-way hypothesis testing on the difference between the would-be and actual performance to determine whether the event resulted in improvement, degradation or no-change in performance. Results from simulation study and experiments on real use cases demonstrate the effectiveness and usability of the proposed approach.

[1]  Ali Hortaçsu,et al.  Winner's Curse, Reserve Prices and Endogenous Entry: Empirical Insights from Ebay Auctions , 2003 .

[2]  J. Mbugua,et al.  Impact of user charges on vulnerable groups: the case of Kibwezi in rural Kenya. , 1995, Social science & medicine.

[3]  Florian Haller,et al.  Equivalence test in quantitative reverse transcription polymerase chain reaction: confirmation of reference genes suitable for normalization. , 2004, Analytical biochemistry.

[4]  Ryan P. Adams,et al.  Bayesian Online Changepoint Detection , 2007, 0710.3742.

[5]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[6]  Abhinav Saxena,et al.  Damage propagation modeling for aircraft engine run-to-failure simulation , 2008, 2008 International Conference on Prognostics and Health Management.

[7]  Guoqiang Peter Zhang,et al.  Time series forecasting using a hybrid ARIMA and neural network model , 2003, Neurocomputing.

[8]  George E. P. Box,et al.  Intervention Analysis with Applications to Economic and Environmental Problems , 1975 .

[9]  Chetan Gupta,et al.  Mining traffic incidents to forecast impact , 2012, UrbComp '12.

[10]  V. Ediger,et al.  ARIMA forecasting of primary energy demand by fuel in Turkey , 2007 .

[11]  E. Vuori,et al.  Alcohol tax cuts and increase in alcohol-positive sudden deaths: a time-series intervention analysis. , 2007, Addiction.

[12]  Chetan Gupta,et al.  What Maintenance is Worth the Money? A Data-Driven Answer , 2018, 2018 IEEE 16th International Conference on Industrial Informatics (INDIN).

[13]  Daniela Jarušková,et al.  Some Problems with Application of Change-Point Detection Methods to Environmental Data , 1997 .

[14]  Chetan Gupta,et al.  Remaining Useful Life Estimation Using Functional Data Analysis , 2019, 2019 IEEE International Conference on Prognostics and Health Management (ICPHM).

[15]  Hubert Cardot,et al.  A new boosting algorithm for improved time-series forecasting with recurrent neural networks , 2008, Inf. Fusion.

[16]  Harald Heinzl,et al.  Combining difference and equivalence test results in spatial maps , 2011, International journal of health geographics.

[17]  Stergios B. Fotopoulos,et al.  Introduction to Modern Nonparametric Statistics , 2004, Technometrics.

[18]  Mylene Lagarde,et al.  How to do (or not to do) ... Assessing the impact of a policy change with routine longitudinal data. , 2012, Health policy and planning.

[19]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[20]  Jürgen Schmidhuber,et al.  Applying LSTM to Time Series Predictable through Time-Window Approaches , 2000, ICANN.

[21]  Wolfgang Jank,et al.  Explaining and Forecasting Online Auction Prices and Their Dynamics Using Functional Data Analysis , 2008 .

[22]  R. Keith Mobley,et al.  An introduction to predictive maintenance , 1989 .

[23]  M. Melnik,et al.  Does a Seller's Ecommerce Reputation Matter? Evidence from Ebay Auctions , 2003 .

[24]  E. Lesaffre.  Superiority, equivalence, and non-inferiority trials. , 2008, Bulletin of the NYU hospital for joint diseases.