Faster bug detection for software product lines with incomplete feature models

A software product line (SPL) is a family of programs that are differentiated by features --- increments in functionality. Systematically testing an SPL is challenging because it requires running each test of a test suite against a combinatorial number of programs. Feature models capture dependencies among features and can (1) reduce the space of programs to test and (2) enable accurate categorization of failing tests as failures of programs or the tests themselves, not as failures due to illegal combinations of features. In practice, sadly, feature models are not always available. We introduce SPLif, the first approach for testing SPLs that does not require the a priori availability of feature models. Our insight is to use a profile of passing and failing test runs to quickly identify failures that are indicative of real problems in test or code rather than specious failures due to illegal feature combinations. Experimental results on five SPLs and one large configurable system (GCC) demonstrate the effectiveness of our approach. SPLif enabled the discovery of five news bugs in GCC, three of which have already been fixed.

[1]  Andreas Zeller,et al.  Lightweight Defect Localization for Java , 2005, ECOOP.

[2]  Michael D. Ernst,et al.  Which configuration option should I change? , 2014, ICSE.

[3]  Sarfraz Khurshid,et al.  Reducing Configurations to Monitor in a Software Product Line , 2010, RV.

[4]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[5]  Myra B. Cohen,et al.  Failure Avoidance in Configurable Systems through Feature Locality , 2013, Assurances for Self-Adaptive Systems.

[6]  Christoph Pohl,et al.  An Exploratory Study of Information Retrieval Techniques in Domain Analysis , 2008, 2008 12th International Software Product Line Conference.

[7]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[8]  Alex Groce,et al.  Taming compiler fuzzers , 2013, ACM-SIGPLAN Symposium on Programming Language Design and Implementation.

[9]  Marcelo d'Amorim,et al.  Entropy-based test generation for improved fault localization , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[10]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[11]  Mathieu Acher,et al.  Feature model extraction from large collections of informal product descriptions , 2013, ESEC/FSE 2013.

[12]  Hung Viet Nguyen,et al.  Exploring variability-aware execution for testing plugin-based web applications , 2014, ICSE.

[13]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[14]  Sarfraz Khurshid,et al.  Reducing combinatorics in testing product lines , 2011, AOSD '11.

[15]  Myra B. Cohen,et al.  Feature Interaction Faults Revisited: An Exploratory Study , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[16]  Alexander Egyed,et al.  Reverse Engineering Feature Models with Evolutionary Algorithms: An Exploratory Study , 2012, SSBSE.

[17]  Krzysztof Czarnecki,et al.  Reverse engineering feature models , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[18]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[19]  Sarfraz Khurshid,et al.  Efficient specification-based testing using incremental techniques , 2008 .

[20]  Marcelo d'Amorim,et al.  Delta Execution for Efficient State-Space Exploration of Object-Oriented Programs , 2007, IEEE Transactions on Software Engineering.

[21]  Don S. Batory,et al.  Feature-oriented programming and the AHEAD tool suite , 2004, Proceedings. 26th International Conference on Software Engineering.

[22]  Myra B. Cohen,et al.  Analysis, Test and Verification in The Presence of Variability (Dagstuhl Seminar 13091) , 2013, Dagstuhl Reports.

[23]  Klaus Pohl,et al.  Software Product Line Engineering - Foundations, Principles, and Techniques , 2005 .

[24]  Alexander Egyed,et al.  On Extracting Feature Models from Sets of Valid Feature Combinations , 2013, FASE.

[25]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[26]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[27]  Ruzanna Chitchyan,et al.  A framework for constructing semantically composable feature models from natural language requirements , 2009, SPLC.

[28]  Sarfraz Khurshid,et al.  SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems , 2013, ESEC/FSE 2013.

[29]  Sven Apel,et al.  Feature cohesion in software product lines: an exploratory study , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[30]  Klaus Pohl,et al.  Software Product Line Engineering , 2005 .

[31]  Sven Apel,et al.  Strategies for product-line verification: Case studies and experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[32]  Peter Zoeteweij,et al.  Spectrum-Based Multiple Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[33]  Sarfraz Khurshid,et al.  Shared Execution for Efficiently Testing Product Lines , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[34]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[35]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[36]  Myra B. Cohen,et al.  Improving the Testing and Testability of Software Product Lines , 2010, SPLC.

[37]  Myra B. Cohen,et al.  Integration Testing of Software Product Lines Using Compositional Symbolic Execution , 2012, FASE.

[38]  Jacques Klein,et al.  Towards automated testing and fixing of re-engineered Feature Models , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[39]  Myra B. Cohen,et al.  Beyond the rainbow: self-adaptive failure avoidance in configurable systems , 2014, SIGSOFT FSE.

[40]  Sven Apel,et al.  Toward variability-aware testing , 2012, FOSD '12.

[41]  Jian Zhang,et al.  Characterizing failure-causing parameter interactions by adaptive testing , 2011, ISSTA '11.

[42]  Gunter Saake,et al.  Similarity-based prioritization in software product-line testing , 2014, SPLC.

[43]  Roberto Erick Lopez-Herrejon,et al.  A Standard Problem for Evaluating Product-Line Methodologies , 2001, GCSE.

[44]  Randy H. Katz,et al.  Static extraction of program configuration options , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[45]  Michael D. Ernst,et al.  Automated diagnosis of software configuration errors , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[46]  Krzysztof Czarnecki,et al.  Generative programming - methods, tools and applications , 2000 .

[47]  Sergio Segura,et al.  Automated Test Data Generation on the Analyses of Feature Models: A Metamorphic Testing Approach , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[48]  Hanspeter Mössenböck,et al.  Where has all my memory gone?: determining memory characteristics of product variants using virtual-machine-level monitoring , 2014, VaMoS '14.

[49]  Krzysztof Czarnecki,et al.  Efficient synthesis of feature models , 2012, SPLC '12.

[50]  Mathieu Acher,et al.  On extracting feature models from product descriptions , 2012, VaMoS.

[51]  Jeff Yu Lei,et al.  Fault localization based on failure-inducing combinations , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[52]  Marcelo d'Amorim,et al.  Delta Execution for Efficient State-Space Exploration of Object-Oriented Programs , 2008, IEEE Trans. Software Eng..

[53]  Krzysztof Czarnecki,et al.  Feature Diagrams and Logics: There and Back Again , 2007, 11th International Software Product Line Conference (SPLC 2007).

[54]  Yuanyuan Zhou,et al.  Do not blame users for misconfigurations , 2013, SOSP.

[55]  Adam A. Porter,et al.  iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees , 2014, IEEE Transactions on Software Engineering.