SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems

Many programs can be configured through dynamic and/or static selection of configuration variables. A software product line (SPL), for example, specifies a family of programs where each program is defined by a unique combination of features. Systematically testing SPL programs is expensive as it can require running each test against a combinatorial number of configurations. Fortunately, a test is often independent of many configuration variables and need not be run against every combination. Configurations that are not required for a test can be pruned from execution. This paper presents SPLat, a new way to dynamically prune irrelevant configurations: the configurations to run for a test can be determined during test execution by monitoring accesses to configuration variables. SPLat achieves an optimal reduction in the number of configurations and is lightweight compared to prior work that used static analysis and heavyweight dynamic execution. Experimental results on 10 SPLs written in Java show that SPLat substantially reduces the total test execution time in many cases. Moreover, we demonstrate the scalability of SPLat by applying it to a large industrial code base written in Ruby on Rails.

[1]  Thomas H. Austin,et al.  Multiple facets for dynamic information flow , 2012, POPL '12.

[2]  Ondrej Lhoták,et al.  Scaling Java Points-to Analysis Using SPARK , 2003, CC.

[3]  Sven Apel,et al.  Toward variability-aware testing , 2012, FOSD '12.

[4]  Emina Torlak,et al.  Angelic debugging , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[5]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[6]  Benjamin Livshits,et al.  Rozzle: De-cloaking Internet Malware , 2012, 2012 IEEE Symposium on Security and Privacy.

[7]  Yuanyuan Zhou,et al.  Efficient online validation with delta execution , 2009, ASPLOS.

[8]  Elaine J. Weyuker,et al.  Theories of Program Testing and the Application of Revealing Subdomains , 1980, IEEE Transactions on Software Engineering.

[9]  William R. Cook,et al.  Safe composition of product lines , 2007, GPCE '07.

[10]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[11]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[12]  Mark Ryan,et al.  Feature integration using a feature construct , 2001, Sci. Comput. Program..

[13]  Pierre-Yves Schobbens,et al.  Symbolic model checking of software product lines , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[14]  Harald C. Gall,et al.  Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu , HI, USA, May 21-28, 2011 , 2011, ICSE.

[15]  Cristian Cadar,et al.  Safe software updates via multi-version execution , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[16]  Pierre-Yves Schobbens,et al.  Model checking lots of systems: efficient verification of temporal properties in software product lines , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[17]  Sarfraz Khurshid,et al.  Reducing Configurations to Monitor in a Software Product Line , 2010, RV.

[18]  Myra B. Cohen,et al.  Improving the Testing and Testability of Software Product Lines , 2010, SPLC.

[19]  Sven Apel,et al.  Strategies for product-line verification: Case studies and experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[20]  Robert J. Hall,et al.  Fundamental Nonmodularity in Electronic Mail , 2004, Automated Software Engineering.

[21]  Darko Marinov,et al.  On test repair using symbolic execution , 2010, ISSTA '10.

[22]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[23]  Sven Apel,et al.  Introducing Binary Decision Diagrams in the explicit-state verification of Java code , 2011 .

[24]  Sarfraz Khurshid,et al.  Reducing combinatorics in testing product lines , 2011, AOSD '11.

[25]  Kyo Chul Kang,et al.  Feature-Oriented Domain Analysis (FODA) Feasibility Study , 1990 .

[26]  John D. McGregor,et al.  Testing a Software Product Line , 2001, PSSE.

[27]  Sarfraz Khurshid,et al.  Shared Execution for Efficiently Testing Product Lines , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[28]  Myra B. Cohen,et al.  Coverage and adequacy in software product line testing , 2006, ROSATEA '06.

[29]  Sven Apel,et al.  Feature cohesion in software product lines: an exploratory study , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[30]  Morris Sloman,et al.  CONIC: an integrated approach to distributed computer control systems , 1983 .

[31]  Roberto Erick Lopez-Herrejon,et al.  A Standard Problem for Evaluating Product-Line Methodologies , 2001, GCSE.

[32]  Claus Brabrand,et al.  Intraprocedural dataflow analysis for software product lines , 2012, AOSD.

[33]  Marcelo d'Amorim,et al.  Delta Execution for Efficient State-Space Exploration of Object-Oriented Programs , 2007, IEEE Transactions on Software Engineering.

[34]  Paul Grünbacher,et al.  Using regression testing to analyze the impact of changes to variability models on products , 2012, SPLC '12.

[35]  Nikolai Tillmann,et al.  Unit tests reloaded: parameterized unit testing with symbolic execution , 2006, IEEE Software.