Peng (

Studies and many incidents in the headlines suggest misconfigurations remain a major cause of unavailability in large systems despite the large amount of work put into detecting, diagnosing and repairing them. In part, this is because many of the solutions are either post-mortem or too expensive to use in production cloud-scale systems. Configuration validation is the process of explicitly defining specifications and proactively checking configurations against those specifications to prevent misconfigurations from entering production. We propose a generic framework, ConfValley, to make configuration validation easy, systematic and efficient, and to allow configuration validation as an ordinary part of system deployment. ConfValley consists of a declarative language for practitioners to express configuration specifications, an inference engine that automatically generates specifications, and a checker that determines if a given configuration obeys its specifications. Our specification language expressed the configuration validation code from Microsoft Azure in 10x fewer lines, many of which were automatically inferred. Using expert-written and inferred specifications, we detected a number of configuration errors in the latest configurations deployed in Microsoft Azure.

[1]  Dan Boneh,et al.  Who killed my battery?: analyzing mobile browser energy consumption , 2012, WWW.

[2]  Giovanni Denaro,et al.  Early performance testing of distributed software applications , 2004, WOSP '04.

[3]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[4]  Ranjita Bhagwan,et al.  Baaz: A System for Detecting Access Control Misconfigurations , 2010, USENIX Security Symposium.

[5]  Brad Calder,et al.  Automatically characterizing large scale program behavior , 2002, ASPLOS X.

[6]  Gregg Rothermel,et al.  An empirical comparison of dynamic impact analysis algorithms , 2004, Proceedings. 26th International Conference on Software Engineering.

[7]  Alvin R. Lebeck,et al.  Power aware page allocation , 2000, SIGP.

[8]  Marin Litoiu,et al.  Model-based performance testing: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[9]  Arun Venkataramani,et al.  Energy consumption in mobile phones: a measurement study and implications for network applications , 2009, IMC '09.

[10]  Andreas Zeller,et al.  Mining Version Histories to Guide Software Changes , 2004 .

[11]  Jakob Engblom,et al.  The worst-case execution-time problem—overview of methods and survey of tools , 2008, TECS.

[12]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[13]  Steven D. Gribble,et al.  Configuration Debugging as Search: Finding the Needle in the Haystack , 2004, OSDI.

[14]  Jason Flinn,et al.  quFiles: The right file at the right time , 2010, TOS.

[15]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[16]  Mithun Acharya,et al.  Practical change impact analysis based on static program slicing for industrial software systems , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[17]  Samuel P. Midkiff,et al.  What is keeping my phone awake?: characterizing and detecting no-sleep energy bugs in smartphone apps , 2012, MobiSys '12.

[18]  R. A. Powers Batteries for low power electronics , 1995, Proc. IEEE.

[19]  Ramesh Govindan,et al.  Energy-delay tradeoffs in smartphone applications , 2010, MobiSys '10.

[20]  Mun Choon Chan,et al.  PGTP: Power aware game transport protocol for multi-player mobile games , 2011, 2011 International Conference on Communications and Signal Processing.

[21]  Ming Zhang,et al.  Bootstrapping energy debugging on smartphones: a first look at energy bugs in mobile devices , 2011, HotNets-X.

[22]  J. Widmer,et al.  On the impact of 2G and 3G network usage for mobile phones' battery life , 2009, 2009 European Wireless Conference.

[23]  Prioritizing test cases for regression testing , 2000, ISSTA.

[24]  James M. Bieman Editorial: Is Anyone Listening? , 2005, Software Quality Journal.

[25]  John DeTreville Making System Configuration More Declarative , 2005, HotOS.

[26]  Ranjit Jhala,et al.  Finding latent performance bugs in systems implementations , 2010, FSE '10.

[27]  Gregg Rothermel,et al.  Whole program path-based dynamic impact analysis , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[28]  Paramvir Bahl,et al.  Fine-grained power modeling for smartphones using system call tracing , 2011, EuroSys '11.

[29]  Alec Wolman,et al.  MAUI: making smartphones last longer with code offload , 2010, MobiSys '10.

[30]  Zhen Wang,et al.  Reflex: using low-power processors in smartphones without knowing them , 2012, ASPLOS XVII.

[31]  Lin Zhong,et al.  Chameleon: A Color-Adaptive Web Browser for Mobile OLED Displays , 2012, IEEE Transactions on Mobile Computing.

[32]  Sandhya Dwarkadas,et al.  Characterizing and predicting program behavior and its variability , 2003, 2003 12th International Conference on Parallel Architectures and Compilation Techniques.

[33]  Jason Flinn,et al.  Self-Tuning Wireless Network Power Management , 2003, MobiCom '03.

[34]  Brad Calder,et al.  Phase tracking and prediction , 2003, ISCA '03.

[35]  Brad Calder,et al.  Motivation for Variable Length Intervals and Hierarchical Phase Behavior , 2005, IEEE International Symposium on Performance Analysis of Systems and Software, 2005. ISPASS 2005..

[36]  Lei Yang,et al.  Accurate online power estimation and automatic battery behavior based power model generation for smartphones , 2010, 2010 IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS).

[37]  Carlos Maltzahn,et al.  Using Comprehensive Analysis for Performance Debugging in Distributed Storage Systems , 2007, 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007).

[38]  D.C. Schmidt,et al.  Reliable Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems , 2007, IEEE Transactions on Software Engineering.

[39]  Mona Attariyan,et al.  AutoBash: improving configuration management with operating system causality analysis , 2007, SOSP.

[40]  Mun Choon Chan,et al.  Adaptive display power management for mobile games , 2011, MobiSys '11.

[41]  Randy H. Katz,et al.  Static extraction of program configuration options , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[42]  Helen J. Wang,et al.  Strider: a black-box, state-based approach to change and configuration management and support , 2003, Sci. Comput. Program..

[43]  Albert G. Greenberg,et al.  Configuration management at massive scale: system design and experience , 2007, IEEE Journal on Selected Areas in Communications.

[44]  Vikram Bhatt,et al.  The GreenDroid Mobile Application Processor: An Architecture for Silicon's Dark Future , 2011, IEEE Micro.

[45]  Xu Chen,et al.  Declarative configuration management for complex and dynamic networks , 2010, CoNEXT.

[46]  Shawn A. Bohner,et al.  Impact analysis-Towards a framework for comparison , 1993, 1993 Conference on Software Maintenance.

[47]  Ming Zhong,et al.  I/O system performance debugging using model-driven anomaly characterization , 2005, FAST'05.

[48]  Gregory Fox Performance engineering as a part of the development life cycle for large-scale software systems , 1989, ICSE '89.

[49]  Ying Zou,et al.  Mining Performance Regression Testing Repositories for Automated Performance Analysis , 2010, 2010 10th International Conference on Quality Software.

[50]  Jason Flinn,et al.  Ghosts in the machine: interfaces for better power management , 2004, MobiSys '04.

[51]  Gokhan Memik,et al.  Into the wild: Studying real user activity patterns to guide power optimizations for mobile architectures , 2009, 2009 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[52]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[53]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[54]  Atanas Rountev,et al.  Uncovering performance problems in Java applications with reference propagation profiling , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[55]  Liviu Iftode,et al.  Security versus energy tradeoffs in host-based mobile malware detection , 2011, MobiSys '11.

[56]  Gernot Heiser,et al.  An Analysis of Power Consumption in a Smartphone , 2010, USENIX Annual Technical Conference.

[57]  Robert C. Martin Agile Software Development, Principles, Patterns, and Practices , 2002 .

[58]  Mona Attariyan,et al.  X-ray: Automating Root-Cause Diagnosis of Performance Anomalies in Production Software , 2012, OSDI.

[59]  Reinhold Heckmann,et al.  Worst case execution time prediction by static program analysis , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[60]  Frank Bellosa,et al.  Cooperative I / O-- A Novel I / O Semantics for Energy-Aware Applications , 2003 .

[61]  Yuanyuan Zhou,et al.  Do not blame users for misconfigurations , 2013, SOSP.

[62]  Robert S. Arnold,et al.  Software Change Impact Analysis , 1996 .

[63]  Michael C. Huang,et al.  Positional adaptation of processors: application to energy reduction , 2003, ISCA '03.

[64]  Marcos K. Aguilera,et al.  Performance debugging for distributed systems of black boxes , 2003, SOSP '03.

[65]  Mikkel Baun Kjærgaard,et al.  EnTracked: energy-efficient robust position tracking for mobile devices , 2009, MobiSys '09.

[66]  Laurie A. Williams,et al.  Empirical Software Change Impact Analysis using Singular Value Decomposition , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[67]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[68]  Helen J. Wang,et al.  Automatic Misconfiguration Troubleshooting with PeerPressure , 2004, OSDI.

[69]  Chen Fu,et al.  Automatically finding performance problems with feedback-directed learning software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[70]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[71]  Dongmei Zhang,et al.  Performance debugging in the large via mining millions of stack traces , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[72]  Frank Tip,et al.  A survey of program slicing techniques , 1994, J. Program. Lang..

[73]  Justin Manweiler,et al.  Avoiding the Rush Hours: WiFi Energy Management via Traffic Isolation , 2011, IEEE Transactions on Mobile Computing.

[74]  Mona Attariyan,et al.  Automating Configuration Troubleshooting with Dynamic Information Flow Analysis , 2010, OSDI.

[75]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[76]  Dina Katabi,et al.  Enabling Configuration-Independent Automation by Non-Expert Users , 2010, OSDI.

[77]  Ion Stoica,et al.  Declarative routing: extensible routing with declarative queries , 2005, SIGCOMM '05.

[78]  King Chun Foo AUTOMATED DISCOVERY OF PERFORMANCE REGRESSIONS IN ENTERPRISE APPLICATIONS , 2011 .

[79]  Malcolm Munro,et al.  An early impact analysis technique for software maintenance , 1994, J. Softw. Maintenance Res. Pract..

[80]  Ramachandran Ramjee,et al.  Bartendr: a practical approach to energy-aware cellular data scheduling , 2010, MobiCom.

[81]  Hong Mei,et al.  An experimental comparison of four test suite reduction techniques , 2006, ICSE.

[82]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[83]  Lujo Bauer,et al.  Detecting and resolving policy misconfigurations in access-control systems , 2008, SACMAT '08.

[84]  Romit Roy Choudhury,et al.  EnLoc: Energy-Efficient Localization for Mobile Phones , 2009, IEEE INFOCOM 2009.

[85]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[86]  Yi-Min Wang,et al.  Discovering correctness constraints for self-management of system configuration , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[87]  Ming Zhang,et al.  Where is the energy spent inside my app?: fine grained energy accounting on smartphones with Eprof , 2012, EuroSys '12.

[88]  Amin Vahdat,et al.  ECOSystem: managing energy as a first class operating system resource , 2002, ASPLOS X.

[89]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[90]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[91]  Alessandro Orso,et al.  Leveraging field data for impact analysis and regression testing , 2003, ESEC/FSE-11.

[92]  Junfeng Yang,et al.  Context-based Online Configuration-Error Detection , 2011, USENIX Annual Technical Conference.

[93]  A. Orso,et al.  Efficient and precise dynamic impact analysis using execute-after sequences , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[94]  Lin Zhong,et al.  Self-constructive high-rate system energy modeling for battery-powered mobile systems , 2011, MobiSys '11.

[95]  Shawn A. Bohner,et al.  Extending software change impact analysis into COTS components , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[96]  Comparing Program Phase Detection Techniques , 2003, MICRO.

[97]  Matthias Hauswirth,et al.  Catch me if you can: performance bug detection in the wild , 2011, OOPSLA '11.

[98]  Philip Levis,et al.  Energy management in mobile devices with the cinder operating system , 2011, EuroSys '11.

[99]  Matthew B. Dwyer,et al.  Automatic generation of load tests , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[100]  Feng Zhao,et al.  Energy-accuracy trade-off for continuous mobile device location , 2010, MobiSys '10.

[101]  Mona Attariyan,et al.  Using Causality to Diagnose Configuration Bugs , 2008, USENIX Annual Technical Conference.

[102]  Romit Roy Choudhury,et al.  Micro-Blog: sharing and querying content through mobile phones and social participation , 2008, MobiSys '08.

[103]  Ian Molyneaux The Art of Application Performance Testing - Help for Programmers and Quality Assurance , 2009 .

[104]  Brad Calder,et al.  Picking statistically valid and early simulation points , 2003, 2003 12th International Conference on Parallel Architectures and Compilation Techniques.

[105]  Kang G. Shin,et al.  Detecting energy-greedy anomalies and mobile malware variants , 2008, MobiSys '08.

[106]  Elaine J. Weyuker,et al.  Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study , 2000, IEEE Trans. Software Eng..

[107]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[108]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[109]  Petr Tuma,et al.  Automated detection of performance regressions: the mono experience , 2005, 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.