Regression Testing in Software Product Line Engineering

Software product line engineering is an approach to cost-efficiently derive tailored products to markets and customers, utilizing common components and services in a planned manner. Product lines have been applied to other engineering fields for decades, while being quite recently introduced in software engineering. For software product lines, productivity gains are mostly related to the development process. Especially, software product line testing faces challenges in the vast number of versions and variants of software products to be tested, originating from a software product line, and consequently the risk for redundant testing. The testing challenges resemble those of regression testing in one-off software development, although adding the complexity of parallel variants. Ongoing research provide some support for software product line test selection, although they are too small-scale and require more formalism than generally available in practice. We propose a visualization approach to help test managers improve communication about testing in order to utilize test resources efficiently.

[1]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[2]  Timothy W. Simpson,et al.  A market-driven approach to product family design , 2009 .

[3]  Ian Sommerville,et al.  Testing in the Wild: The Social and Organisational Dimensions of Real World Practice , 2009, Computer Supported Cooperative Work (CSCW).

[4]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[5]  Jean-Pierre Corriveau,et al.  On Extracting Tests from a Testable Model in the Context of Domain Engineering , 2008, 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008).

[6]  Gregg Rothermel,et al.  Regression test selection for C++ software , 2000 .

[7]  Per Runeson,et al.  A systematic review on regression test selection techniques , 2010, Inf. Softw. Technol..

[8]  Laurie Williams,et al.  Agile Software Development Methodologies and Practices , 2010, Adv. Comput..

[9]  Klaus Pohl,et al.  Software product line testing , 2006, CACM.

[10]  Per Runeson,et al.  A survey of unit testing practices , 2006, IEEE Software.

[11]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[12]  Birgit Geppert,et al.  Enabling Verifiable Conformance for Product Lines , 2008, 2008 12th International Software Product Line Conference.

[13]  Bogdan Korel,et al.  Model based regression test reduction using dependence analysis , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[14]  John D. McGregor,et al.  A systematic mapping study of software product lines testing , 2011, Inf. Softw. Technol..

[15]  Antonia Bertolino,et al.  Use case-based testing of product lines , 2003, ESEC/FSE-11.

[16]  Alexander Verbraeck,et al.  Using Simulation to Design an Automated Underground System for Transporting Freight Around Schiphol Airport , 2002, Interfaces.

[17]  Hau L. Lee,et al.  Hewlett-Packard Gains Control of Inventory and Service through Design for Localization , 1993 .

[18]  Klaus Pohl,et al.  Integration Testing in Software Product Line Engineering: A Model-Based Technique , 2007, FASE.

[19]  René L. Krikhaar,et al.  Modeling Variability and Testability Interaction in Software Product Line Engineering , 2008, Seventh International Conference on Composition-Based Software Systems (ICCBSS 2008).

[20]  Juha Taina,et al.  Product family testing: a survey , 2004, SOEN.

[21]  Shigeo Kato,et al.  Variation Management for Software Product Lines with Cumulative Coverage of Feature Interactions , 2011, 2011 15th International Software Product Line Conference.

[22]  Paul Clements,et al.  Software product lines - practices and patterns , 2001, SEI series in software engineering.

[23]  Nashat Mansour,et al.  A comparative study of five regression testing algorithms , 1997, Proceedings of Australian Software Engineering Conference ASWEC 97.

[24]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[25]  Jacques Klein,et al.  Pairwise testing for software product lines: comparison of two approaches , 2012, Software Quality Journal.

[26]  Per Runeson,et al.  A Qualitative Survey of Regression Testing Practices , 2010, PROFES.

[27]  Hareton Leung,et al.  A study of integration testing and software regression at the integration level , 1990, Proceedings. Conference on Software Maintenance 1990.

[28]  Per Runeson,et al.  Software Product Line Testing -- A 3D Regression Testing Problem , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[29]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[30]  Myra B. Cohen,et al.  Coverage and adequacy in software product line testing , 2006, ROSATEA '06.

[31]  Klaus Pohl,et al.  Software Product Line Engineering - Foundations, Principles, and Techniques , 2005 .

[32]  Gregg Rothermel,et al.  A safe, efficient algorithm for regression test selection , 1993, 1993 Conference on Software Maintenance.

[33]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[34]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[35]  Klaus Pohl,et al.  Model-Based System Testing of Software Product Families , 2005, CAiSE.

[36]  Dirk Muthig,et al.  Making testing product lines more efficient by improving the testability of product line architectures , 2006, ROSATEA '06.

[37]  B. Wilhelm,et al.  Platform and Modular Concepts at Volkswagen — Their Effects on the Assembly Process , 1997 .

[38]  Andy Schürr,et al.  A Survey of Model-Based Software Product Lines Testing , 2011, Model-Based Testing for Embedded Systems.

[39]  Henry Muccini,et al.  Towards Testing Product Line Architectures , 2003, TACoS.

[40]  Xiaodong Liu,et al.  A product line based aspect-oriented generative unit testing approach to building quality components , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[41]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[42]  Timothy W. Simpson,et al.  Product platform design and customization: Status and promise , 2004, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[43]  Mark Harman,et al.  Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge , 2009, ISSTA.

[44]  Arlene Fink,et al.  The Survey Handbook , 1995 .

[45]  Kristen Nygaard,et al.  The development of the SIMULA languages , 1978, SIGP.

[46]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[47]  Greg Paula,et al.  Reinventing a core product line , 1997 .

[48]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[49]  Alessandro Orso,et al.  Scaling regression testing to large software systems , 2004, SIGSOFT '04/FSE-12.

[50]  Antonia Bertolino,et al.  Product Line Use Cases: Scenario-Based Specification and Testing of Requirements , 2006, Software Product Lines.

[51]  Per Runeson,et al.  Test Benchmarks -- what is the question? , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[52]  Lianping Chen,et al.  Variability management in software product lines: a systematic review , 2009, SPLC.

[53]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[54]  Charles W. Krueger.  Towards a Taxonomy for Software Product Lines , 2003, PFE.

[55]  Laurie A. Williams,et al.  An initial study of a lightweight process for change identification and regression test selection when source code is not available , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[56]  Birgit Geppert,et al.  Reuse Execution Traces to Reduce Testing of Product Lines , 2007, SPLC.

[57]  Hareton K. N. Leung,et al.  A cost model to compare regression test strategies , 1991, Proceedings. Conference on Software Maintenance 1991.

[58]  Clémentine Nebut,et al.  A Requirement-Based Approach to Test Product Families , 2003, PFE.

[59]  Jan Bosch,et al.  Design and use of software architectures - adopting and evolving a product-line approach , 2000 .

[60]  John D. McGregor,et al.  The Implications of Variation for Testing in a Software Product Line , 2007, SPLC.

[61]  M. Meyer,et al.  Revitalize Your Product Lines Through Continuous Platform Renewal , 1997 .

[62]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[63]  Klaus Pohl,et al.  The ScenTED Method for Testing Software Product Lines , 2006, Software Product Lines.

[64]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[65]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[66]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[67]  James Milne Neighbors,et al.  Software construction using components , 1980 .

[68]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[69]  Timothy W. Simpson,et al.  Henry Ford and the Model T: lessons for product platforming and mass customization , 2009 .

[70]  Myra B. Cohen,et al.  Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces , 2006, IEEE Trans. Software Eng..

[71]  Per Runeson,et al.  Test overlay in an emerging software product line - An industrial case study , 2013, Inf. Softw. Technol..

[72]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[73]  E. B. Swanson,et al.  Software maintenance management , 1980 .

[74]  Meir M. Lehman.  Programs, life cycles, and laws of software evolution , 1980 .

[75]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[76]  Øystein Haugen,et al.  A Survey of Empirics of Strategies for Software Product Line Testing , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[77]  John D. McGregor,et al.  Testing a Software Product Line , 2001, PSSE.

[78]  L. Erlikh,et al.  Leveraging legacy system dollars for e-business , 2000 .

[79]  Mark Harman,et al.  Test prioritization using system models , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[80]  David M. Weiss,et al.  Automatic Integration Test Generation from Unit Tests of eXVantage Product Family , 2007, SPLC.

[81]  A. S. M. Sajeev,et al.  Regression test selection based on version changes of components , 2003, Tenth Asia-Pacific Software Engineering Conference, 2003..

[82]  S. Sanderson,et al.  Managing product families: The case of the Sony Walkman , 1995 .

[83]  Even-André Karlsson,et al.  Software reuse: a holistic approach , 1995 .

[84]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[85]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[86]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[87]  Per Runeson,et al.  Improving Regression Testing Transparency and Efficiency with History-Based Prioritization -- An Industrial Case Study , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[88]  T. Simpson.  A concept exploration method for product family design , 1998 .

[89]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[90]  Antonia Bertolino,et al.  PLUTO: A Test Methodology for Product Families , 2003, PFE.

[91]  Per Runeson,et al.  An Empirical Evaluation of Regression Testing Based on Fix-Cache Recommendations , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[92]  Frank Klawonn,et al.  Visualisation of Test Coverage for Conformance Tests of Low Level Communication Protocols , 2010, KES.