RECONTEST: Effective Regression Testing of Concurrent Programs

Concurrent programs proliferate as multi-core technologies advance. The regression testing of concurrent programs often requires running a failing test for weeks before catching a faulty interleaving, due to the myriad of possible interleavings of memory accesses arising from concurrent program executions. As a result, the conventional approach that selects a sub-set of test cases for regression testing without considering interleavings is insufficient. In this paper we present RECONTEST to address the problem by selecting the new interleavings that arise due to code changes. These interleavings must be explored in order to uncover regression bugs. RECONTEST efficiently selects new interleavings by first identifying shared memory accesses that are affected by the changes, and then exploring only those problematic interleavings that contain at least one of these accesses. We have implemented RECONTEST as an automated tool and evaluated it using 13 real-world concurrent program subjects. Our results show that RECONTEST can significantly reduce the regression testing cost without missing any faulty interleavings induced by code changes.

[1]  W. K. Chan,et al.  MagicFuzzer: Scalable deadlock detection for large-scale applications , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[2]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[3]  Alessandro Orso,et al.  Regression test selection for Java software , 2001, OOPSLA '01.

[4]  Harald C. Gall,et al.  Change Distilling:Tree Differencing for Fine-Grained Source Code Change Extraction , 2007, IEEE Transactions on Software Engineering.

[5]  Robert S. Arnold,et al.  Software Change Impact Analysis , 1996 .

[6]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[7]  Suzette Person,et al.  Regression Verification Using Impact Summaries , 2013, SPIN.

[8]  Koushik Sen,et al.  CalFuzzer: An Extensible Active Testing Framework for Concurrent Programs , 2009, CAV.

[9]  Xiangyu Zhang,et al.  Identifying execution points for dynamic analyses , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[10]  Gregg Rothermel,et al.  Regression model checking , 2009, 2009 IEEE International Conference on Software Maintenance.

[11]  Jeff Huang,et al.  Persuasive prediction of concurrency access anomalies , 2011, ISSTA '11.

[12]  Alessandro Orso,et al.  MINTS: A general framework and tool for supporting test-suite minimization , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[13]  Yuanyuan Zhou,et al.  CTrigger: exposing atomicity violation bugs from their hiding places , 2009, ASPLOS.

[14]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[15]  Jeff Huang,et al.  Scaling predictive analysis of concurrent programs by removing trace redundancy , 2013, TSEM.

[16]  A. Orso,et al.  Efficient and precise dynamic impact analysis using execute-after sequences , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[17]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[18]  Koushik Sen,et al.  A randomized dynamic program analysis technique for detecting real deadlocks , 2009, PLDI '09.

[19]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[20]  Patrice Godefroid,et al.  Dynamic partial-order reduction for model checking software , 2005, POPL '05.

[21]  Hareton K. N. Leung,et al.  A survey of code‐based change impact analysis techniques , 2013, Softw. Test. Verification Reliab..

[22]  Julian Dolby,et al.  Marathon: Detecting Atomic-Set Serializability Violations with Conflict Graphs , 2011, RV.

[23]  Azadeh Farzan,et al.  The Complexity of Predicting Atomicity Violations , 2009, TACAS.

[24]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[25]  Koushik Sen,et al.  Race directed random testing of concurrent programs , 2008, PLDI '08.

[26]  Sarfraz Khurshid,et al.  Directed incremental symbolic execution , 2011, PLDI '11.

[27]  Xiangyu Zhang,et al.  Efficient program execution indexing , 2008, PLDI '08.

[28]  Mahesh Viswanathan,et al.  Incremental state-space exploration for programs with dynamically allocated data , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[29]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[30]  Friedemann Mattern,et al.  Virtual Time and Global States of Distributed Systems , 2002 .

[31]  Mayur Naik,et al.  A dynamic evaluation of the precision of static heap abstractions , 2010, OOPSLA.

[32]  Gregg Rothermel,et al.  The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments , 2010, IEEE Transactions on Software Engineering.

[33]  Francesco Sorrentino,et al.  PENELOPE: weaving threads to expose atomicity violations , 2010, FSE '10.

[34]  Dirk Beyer,et al.  Precision reuse for efficient regression verification , 2013, ESEC/FSE 2013.

[35]  Shing-Chi Cheung,et al.  Detecting atomic-set serializability violations in multithreaded programs through active randomized testing , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[36]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multi-threaded programs , 1997, TOCS.

[37]  Satish Narayanasamy,et al.  Maple: a coverage-driven testing tool for multithreaded programs , 2012, OOPSLA '12.

[38]  Shan Lu,et al.  A study of interleaving coverage criteria , 2007, ESEC-FSE companion '07.

[39]  Yuanyuan Zhou,et al.  Learning from mistakes: a comprehensive study on real world concurrency bug characteristics , 2008, ASPLOS.

[40]  Frank Tip,et al.  Associating synchronization constraints with data in an object-oriented language , 2006, POPL '06.

[41]  Eitan Farchi,et al.  Concurrent bug patterns and how to test them , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[42]  Darko Marinov,et al.  Change-aware preemption prioritization , 2011, ISSTA '11.

[43]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[44]  Koushik Sen,et al.  Detecting Errors in Multithreaded Programs by Generalized Predictive Analysis of Executions , 2005, FMOODS.

[45]  Olin Shivers,et al.  Control flow analysis in scheme , 1988, PLDI '88.

[46]  Rajiv Gupta,et al.  Dynamic recognition of synchronization operations for improved data race detection , 2008, ISSTA '08.

[47]  Zhiqiang Ma,et al.  Ad Hoc Synchronization Considered Harmful , 2010, OSDI.

[48]  Gregg Rothermel,et al.  SimRT: an automated framework to support regression testing for data races , 2014, ICSE.

[49]  Shan Lu,et al.  Efficient concurrency-bug detection across inputs , 2013, OOPSLA.

[50]  Shan Lu,et al.  Understanding the interleaving-space overlap across inputs and software versions , 2012, HotPar'12.