A Real-time Framework for Detecting Efficiency Regressions in a Globally Distributed Codebase

Multiple teams at Facebook are tasked with monitoring compute and memory utilization metrics that are important for managing the efficiency of the codebase. An efficiency regression is characterized by instances where the CPU utilization or query per second (QPS) patterns of a function or endpoint experience an unexpected increase over its prior baseline. If the code changes responsible for these regressions get propagated to Facebook's fleet of web servers, the impact of the inefficient code will get compounded over billions of executions per day, carrying potential ramifications to Facebook's scaling efforts and the quality of the user experience. With a codebase ingesting in excess of 1,000 diffs across multiple pushes per day, it is important to have a real-time solution for detecting regressions that is not only scalable and high in recall, but also highly precise in order to avoid overrunning the remediation queue with thousands of false positives. This paper describes the end-to-end regression detection system designed and used at Facebook. The main detection algorithm is based on sequential statistics supplemented by signal processing transformations, and the performance of the algorithm was assessed with a mixture of online and offline tests across different use cases. We compare the performance of our algorithm against a simple benchmark as well as a commercial anomaly detection software solution.

[1]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[2]  Anonymous Cvpr submission Anonymous,et al.  Robust Online Change-point Detection in Video Sequences , 2006, CVPR 2006.

[3]  Zheng Shao,et al.  Data warehousing and analytics infrastructure at facebook , 2010, SIGMOD Conference.

[4]  Kenji Yamanishi,et al.  A unifying framework for detecting outliers and change points from non-stationary time series data , 2002, KDD.

[5]  Parameswaran Ramanathan,et al.  Real-time computing: a new discipline of computer science and engineering , 1994, Proc. IEEE.

[6]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[7]  Dimitris N. Metaxas,et al.  Robust Online Change-point Detection in Video Sequences , 2006 .

[8]  Tony Savor,et al.  Continuous deployment of mobile software at facebook (showcase) , 2016, SIGSOFT FSE.

[9]  Beveridge-Nelson-Stock-Watson Decomposition,et al.  Structural Time Series Models , 2004 .

[10]  Diane J. Cook,et al.  A survey of methods for time series change point detection , 2017, Knowledge and Information Systems.

[11]  Stefan Fremdt,et al.  Page's sequential procedure for change-point detection in time series regression , 2013, 1308.1237.

[12]  Arun Kejariwal,et al.  On the Runtime-Efficacy Trade-off of Anomaly Detection Techniques for Real-Time Streaming Data , 2017, ArXiv.

[13]  J. Yosinski,et al.  Time-series Extreme Event Forecasting with Neural Networks at Uber , 2017 .

[14]  Hui Ding,et al.  TAO: Facebook's Distributed Data Store for the Social Graph , 2013, USENIX Annual Technical Conference.

[15]  Idris A. Eckley,et al.  changepoint: An R Package for Changepoint Analysis , 2014 .

[16]  Xing Xie,et al.  Learning transportation mode from raw gps data for geographic applications on the web , 2008, WWW.

[17]  Komal Shringare,et al.  Apache Hadoop Goes Realtime at Facebook , 2015 .

[18]  Avery Ching,et al.  One Trillion Edges: Graph Processing at Facebook-Scale , 2015, Proc. VLDB Endow..

[19]  Mikhail Bautin,et al.  Storage Infrastructure Behind Facebook Messages: Using HBase at Scale , 2012, IEEE Data Eng. Bull..

[20]  Victor Konev,et al.  Quickest Detection of Parameter Changes in Stochastic Regression: Nonparametric CUSUM , 2017, IEEE Transactions on Information Theory.