Test-case prioritization for configuration testing

Configuration changes are among the dominant causes of failures of large-scale software system deployment. Given the velocity of configuration changes, typically at the scale of hundreds to thousands of times daily in modern cloud systems, checking these configuration changes is critical to prevent failures due to misconfigurations. Recent work has proposed configuration testing, Ctest, a technique that tests configuration changes together with the code that uses the changed configurations. Ctest can automatically generate a large number of ctests that can effectively detect misconfigurations, including those that are hard to detect by traditional techniques. However, running ctests can take a long time to detect misconfigurations. Inspired by traditional test-case prioritization (TCP) that aims to reorder test executions to speed up detection of regression code faults, we propose to apply TCP to reorder ctests to speed up detection of misconfigurations. We extensively evaluate a total of 84 traditional and novel ctest-specific TCP techniques. The experimental results on five widely used cloud projects demonstrate that TCP can substantially speed up misconfiguration detection. Our study provides guidelines for applying TCP to configuration testing in practice.

[1]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[2]  Gregg Rothermel,et al.  Techniques for improving regression testing in continuous integration development environments , 2014, SIGSOFT FSE.

[3]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[4]  Haryadi S. Gunawi,et al.  Why Does the Cloud Stop Computing?: Lessons from Hundreds of Service Outages , 2016, SoCC.

[5]  Yuhua Qi,et al.  Efficient Automated Program Repair through Fault-Recorded Testing Prioritization , 2013, 2013 IEEE International Conference on Software Maintenance.

[6]  Lu Zhang,et al.  Optimizing test prioritization via test distribution analysis , 2018, ESEC/SIGSOFT FSE.

[7]  Michael D. Ernst,et al.  Reducing wasted development time via continuous testing , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[8]  Lingming Zhang,et al.  Empirically revisiting and enhancing IR-based test-case prioritization , 2020, ISSTA.

[9]  Dan Hao,et al.  How Does Regression Test Selection Affect Program Repair? An Extensive Study on 2 Million Patches , 2021, ArXiv.

[10]  Long Jin,et al.  Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software , 2015, ESEC/SIGSOFT FSE.

[11]  Sarfraz Khurshid,et al.  Faster mutation testing inspired by test prioritization and reduction , 2013, ISSTA.

[12]  Tianyin Xu,et al.  Mining Container Image Repositories for Software Configuration and Beyond , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[13]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines , 2014, IEEE Transactions on Software Engineering.

[14]  Gregg Rothermel,et al.  Bridging the gap between the total and additional test-case prioritization strategies , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[15]  Stephen E. Robertson,et al.  Experimentation as a way of life: Okapi at TREC , 2000, Inf. Process. Manag..

[16]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[17]  Robert Karl,et al.  Holistic configuration management at Facebook , 2015, SOSP.

[18]  Lingming Zhang,et al.  Practical program repair via bytecode mutation , 2018, ISSTA.

[19]  Runxiang Cheng,et al.  Testing Configuration Changes in Context to Prevent Production Failures , 2020, OSDI.

[20]  Shanshan Li,et al.  An Evolutionary Study of Configuration Design and Implementation in Cloud Systems , 2021, 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE).

[21]  J. Tukey Comparing individual means in the analysis of variance. , 1949, Biometrics.

[22]  Mark Harman,et al.  Regression Testing Minimisation, Selection and Prioritisation - A Survey , 2009 .

[23]  Mark Harman,et al.  Empirical evaluation of pareto efficient multi-objective regression test case prioritisation , 2015, ISSTA.

[24]  Randy H. Katz,et al.  How Hadoop Clusters Break , 2013, IEEE Software.

[25]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[26]  Yves Le Traon,et al.  Comparing White-Box and Black-Box Test Prioritization , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[27]  Tianyin Xu,et al.  EnCore: exploiting system environment and correlation information for misconfiguration detection , 2014, ASPLOS.

[28]  Lu Zhang,et al.  How Does Regression Test Prioritization Perform in Real-World Software Evolution? , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[29]  Salman Baset,et al.  Usable declarative configuration specification and validation for applications, systems, and cloud , 2017, Middleware '17.

[30]  Darko Marinov,et al.  DeFlaker: Automatically Detecting Flaky Tests , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[31]  Junfeng Yang,et al.  Context-based Online Configuration-Error Detection , 2011, USENIX Annual Technical Conference.

[32]  Sarfraz Khurshid,et al.  An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[33]  Xiao Ma,et al.  An empirical study on configuration errors in commercial and open source systems , 2011, SOSP.

[34]  Sven Apel,et al.  A Comparison of 10 Sampling Algorithms for Configurable Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[35]  Lorenzo Keller,et al.  ConfErr: A tool for assessing resilience to human configuration errors , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[36]  Rui Abreu,et al.  An Empirical Study on the Use of Defect Prediction for Test Case Prioritization , 2019, 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST).

[37]  Helen J. Wang,et al.  Friends Troubleshooting Network: Towards Privacy-Preserving, Automatic Troubleshooting , 2004, IPTPS.

[38]  Myra B. Cohen,et al.  Reducing Field Failures in System Configurable Software: Cost-Based Prioritization , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[39]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[40]  Helen J. Wang,et al.  Automatic Misconfiguration Troubleshooting with PeerPressure , 2004, OSDI.

[41]  Ben Maurer Fail at Scale , 2015, ACM Queue.

[42]  Steven D. Gribble,et al.  Configuration Debugging as Search: Finding the Needle in the Haystack , 2004, OSDI.

[43]  M. Stone Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[44]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[45]  Shankar Pasupathy,et al.  PracExtractor: Extracting Configuration Good Practices from Manuals to Detect Server Misconfigurations , 2020, USENIX Annual Technical Conference.

[46]  Parthasarathy Ranganathan,et al.  The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition , 2018, The Datacenter as a Computer.

[47]  Mona Attariyan,et al.  X-ray: Automating Root-Cause Diagnosis of Performance Anomalies in Production Software , 2012, OSDI.

[48]  Thorsten Berger,et al.  Tackling Combinatorial Explosion: A Study of Industrial Needs and Practices for Analyzing Highly Configurable Systems , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[49]  Michael D. Ernst,et al.  Automated diagnosis of software configuration errors , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[50]  Tianyin Xu,et al.  Systems Approaches to Tackling Configuration Errors , 2015, ACM Comput. Surv..

[51]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[52]  Shanshan Li,et al.  Understanding and discovering software configuration dependencies in cloud and datacenter systems , 2020, ESEC/SIGSOFT FSE.

[53]  Ruzica Piskac,et al.  Probabilistic Automated Language Learning for Configuration Files , 2016, CAV.

[54]  Yuanyuan Zhou,et al.  Early Detection of Configuration Errors to Reduce Failure Damage , 2016, USENIX Annual Technical Conference.

[55]  Randy H. Katz,et al.  Precomputing possible configuration error diagnoses , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[56]  Yuanyuan Zhou,et al.  Do not blame users for misconfigurations , 2013, SOSP.

[57]  Ran Wolff,et al.  Mining for misconfigured machines in grid systems , 2006, KDD '06.

[58]  Archana Ganapathi,et al.  Why Do Internet Services Fail, and What Can Be Done About It? , 2002, USENIX Symposium on Internet Technologies and Systems.

[59]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[60]  Qi Luo,et al.  How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects , 2018, IEEE Transactions on Software Engineering.

[61]  Arjan J. C. van Gemund,et al.  Prioritizing Tests for Software Fault Localization , 2010, 2010 10th International Conference on Quality Software.

[62]  Gemaennes Encore , 1989 .

[63]  Navendu Jain,et al.  ConfSeer: Leveraging Customer Support Knowledge Bases for Automated Misconfiguration Detection , 2015, Proc. VLDB Endow..

[64]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[65]  Ruzica Piskac,et al.  Synthesizing configuration file specifications with association rule learning , 2017, Proc. ACM Program. Lang..

[66]  Helen J. Wang,et al.  Privacy-Preserving Friends Troubleshooting Network , 2005, NDSS.

[67]  Mona Attariyan,et al.  Automating Configuration Troubleshooting with Dynamic Information Flow Analysis , 2010, OSDI.

[68]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[69]  Ranjita Bhagwan,et al.  Rex: Preventing Bugs and Misconfiguration in Large Services Using Correlated Change Analysis , 2020, NSDI.

[70]  Michael D. Ernst,et al.  Which configuration option should I change? , 2014, ICSE.

[71]  Richard P. Martin,et al.  Understanding and Dealing with Operator Mistakes in Internet Services , 2004, OSDI.

[72]  Yuanyuan Zhou,et al.  Towards Continuous Access Control Validation and Forensics , 2019, CCS.

[73]  Peng Huang,et al.  ConfValley: a systematic configuration validation framework for cloud services , 2015, EuroSys.

[74]  Tianyin Xu,et al.  Configuration Testing: Testing Configuration Values as Code and with Code. , 2019, 1905.12195.

[75]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[76]  Sarfraz Khurshid,et al.  SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems , 2013, ESEC/FSE 2013.

[77]  C. Isci,et al.  ConfEx : An Analytics Framework for Text-Based Software Configurations in the Cloud , 2018 .

[78]  Joel Wein,et al.  ACMS: the Akamai configuration management system , 2005, NSDI.