Research in concurrent software testing: a systematic review

The current increased demand for distributed applications in domains such as web services and cloud computing has significantly increased interest in concurrent programming. This demand in turn has resulted in new testing methodologies for such systems, which take account of the challenges necessary to test these applications. This paper presents a systematic review of the published research related to concurrent testing approaches, bug classification and testing tools. A systematic review is a process of collection, assessment and interpretation of the published papers related to a specific search question, designed to provide a background for further research. The results include information about the research relationships and research teams that are working in the different areas of concurrent programs testing.

[1]  Kazuo Ushijima,et al.  Test-case generation for concurrent programs with the testing criteria using interaction sequences , 1999, Proceedings Sixth Asia Pacific Software Engineering Conference (ASPEC'99) (Cat. No.PR00509).

[2]  Yuanyuan Zhou,et al.  CTrigger: exposing atomicity violation bugs from their hiding places , 2009, ASPLOS.

[3]  Yu Lei,et al.  Reachability Graph-Based Test Sequence Generation for Concurrent Programs , 2008, Int. J. Softw. Eng. Knowl. Eng..

[4]  Lori L. Pollock,et al.  An algorithm for all-du-path testing coverage of shared memory parallel programs , 1997, Proceedings Sixth Asian Test Symposium (ATS'97).

[5]  Hong-Fa Ho,et al.  Branch testing of concurrent programs using Petri net models , 1990, Comput. Syst. Sci. Eng..

[6]  David Chenho Kung,et al.  State-based reproducible testing for CORBA applications , 1999, 1999 Proceedings International Symposium on Software Engineering for Parallel and Distributed Systems.

[7]  Dan Grossman,et al.  Composable specifications for structured shared-memory communication , 2010, OOPSLA.

[8]  Madan Musuvathi,et al.  Iterative context bounding for systematic testing of multithreaded programs , 2007, PLDI '07.

[9]  Wei Shi,et al.  A self-adaptive test framework for concurrent programs , 2009, MEDES.

[10]  Shmuel Ur,et al.  Multi-threaded Testing with AOP Is Easy, and It Finds Bugs! , 2005, Euro-Par.

[11]  Zvonimir Rakamaric.  STORM: static unit checking of concurrent programs , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[12]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[13]  Zuohua Ding,et al.  A rigorous approach towards test case generation , 2008, Inf. Sci..

[14]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[15]  Konstantin Serebryany,et al.  ThreadSanitizer: data race detection in practice , 2009, WBIA '09.

[16]  Scott D. Stoller,et al.  Testing Concurrent Java Programs using Randomized Scheduling , 2002, RV@FLoC.

[17]  Christopher Collins,et al.  TIE: an interactive visualization of thread interleavings , 2010, SOFTVIS '10.

[18]  Katherine A. Yelick,et al.  Enforcing Textual Alignment of Collectives Using Dynamic Checks , 2009, LCPC.

[19]  Krishna M. Kavi,et al.  Modeling Multithreaded Applications Using Petri Nets , 2002, International Journal of Parallel Programming.

[20]  Scott D. Stoller,et al.  Runtime analysis for atomicity for multi-threaded programs , 2005 .

[21]  Kai Xu,et al.  Formally defining a graphical language for monitoring and checking object interactions , 2007, MODELS'07.

[22]  R. H. Carver,et al.  Integrating formal methods and testing for concurrent programs , 1995, COMPASS '95 Proceedings of the Tenth Annual Conference on Computer Assurance Systems Integrity, Software Safety and Process Security'.

[23]  Matthew B. Dwyer,et al.  A Flexible Framework for the Estimation of Coverage Metrics in Explicit State Software Model Checking , 2004, CASSIS.

[24]  Scott A. Mahlke,et al.  Gadara: Dynamic Deadlock Avoidance for Multithreaded Programs , 2008, OSDI.

[25]  Marek Olszewski,et al.  Kendo: efficient deterministic multithreading in software , 2009, ASPLOS.

[26]  John Chapin,et al.  Systematic Testing of Multithreaded Programs , 2000 .

[27]  Koushik Sen,et al.  A Race-Detection and Flipping Algorithm for Automated Testing of Multi-threaded Programs , 2006, Haifa Verification Conference.

[28]  T. H. Tse,et al.  A scheme for dynamic detection of concurrent execution of object-oriented software , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[29]  Francesco Sorrentino,et al.  PENELOPE: weaving threads to expose atomicity violations , 2010, FSE '10.

[30]  David Lie,et al.  Kivati: fast detection and prevention of atomicity violations , 2010, EuroSys '10.

[31]  Richard H. Carver,et al.  Distributed reachability testing of concurrent programs , 2010 .

[32]  Michael Hicks,et al.  Dynamic inference of polymorphic lock types , 2005, Sci. Comput. Program..

[33]  A. Smith.  Dynamic Datarace Detection for Object-Oriented Programs , 2002 .

[34]  R. Carver,et al.  Deterministic execution testing of concurrent Ada programs , 1989, TRI-Ada '89.

[35]  Darko Marinov,et al.  Mutation Operators for Actor Systems , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[36]  A. Petrenko,et al.  Dynamic analysis of java applications for multithreaded antipatterns , 2005, WODA '05.

[37]  Ganesh Gopalakrishnan,et al.  GEM: Graphical Explorer of MPI Programs , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[38]  Satish Narayanasamy,et al.  A case for an interleaving constrained shared-memory multi-processor , 2009, ISCA '09.

[39]  Yuanyuan Zhou,et al.  Learning from mistakes: a comprehensive study on real world concurrency bug characteristics , 2008, ASPLOS.

[40]  Bixin Li,et al.  On Testing Multi-threaded Java Programs , 2007 .

[41]  Andrew H. Sung.  Testing shared-memory parallel programs , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[42]  Simone do Rocio Senger de Souza,et al.  ValiPVM - A Graphical Tool for Structural Testing of PVM Programs , 2008, PVM/MPI.

[43]  Richard H. Carver,et al.  A combinatorial testing strategy for concurrent programs , 2007 .

[44]  Lori L. Pollock,et al.  All-du-path coverage for parallel programs , 1998, ISSTA '98.

[45]  Zijiang Yang,et al.  Trace-Driven Verification of Multithreaded Programs , 2010, ICFEM.

[46]  Bernhard K. Aichernig,et al.  Conformance Testing of Distributed Concurrent Systems with Executable Designs , 2008, FMCO.

[47]  Ren-Dar Yang,et al.  The analysis of infeasible concurrent paths of concurrent Ada programs , 1990, Proceedings., Fourteenth Annual International Computer Software and Applications Conference.

[48]  Lori Pollock,et al.  Program-based, structural testing of shared memory parallel programs , 1999 .

[49]  X. Cai,et al.  Control of nondeterminism in testing distributed multithreaded programs , 2000, Proceedings First Asia-Pacific Conference on Quality Software.

[50]  Li Shu,et al.  Timing-sequence testing of parallel programs , 2008, Journal of Computer Science and Technology.

[51]  Richard H. Carver,et al.  Incremental Integration Testing of Concurrent Programs , 2002, IEEE Trans. Software Eng..

[52]  Wei Sun,et al.  A Graph-Search Based Approach to BPEL4WS Test Generation , 2006, 2006 International Conference on Software Engineering Advances (ICSEA'06).

[53]  Zijiang Yang.  SMT-based Symbolic Model Checking for Multi-Threaded Programs , 2008 .

[54]  Kuo-Chung Tai,et al.  An incremental approach to structural testing of concurrent software , 1996, ISSTA '96.

[55]  Armin Biere,et al.  Enforcer - Efficient Failure Injection , 2006, FM.

[56]  Francisco Vilar Brasileiro,et al.  Improving Automated Testing of Multi-threaded Software , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[57]  Kuo-Chung Tai,et al.  Automated test sequence generation using sequencing constraints for concurrent programs , 1999, 1999 Proceedings International Symposium on Software Engineering for Parallel and Distributed Systems.

[58]  Kuo-Chung Tai,et al.  On Godefroid's state-less search technique for testing concurrent programs , 2001, Proceedings 5th International Symposium on Autonomous Decentralized Systems.

[59]  Pravesh Kothari,et al.  A randomized scheduler with probabilistic guarantees of finding bugs , 2010, ASPLOS XV.

[60]  Timothy K. Shih,et al.  Software testing and metrics for concurrent computation through task decomposition , 1997, 1997 IEEE International Conference on Intelligent Processing Systems (Cat. No.97TH8335).

[61]  Zengo Furukawa,et al.  Event interactions graph for test-case generations of concurrent programs , 1995, Proceedings 1995 Asia Pacific Software Engineering Conference.

[62]  William Pugh,et al.  Unit testing concurrent software , 2007, ASE.

[63]  Michael M. Resch,et al.  MPI Application Development Using the Analysis Tool MARMOT , 2004, International Conference on Computational Science.

[64]  Bernhard K. Aichernig,et al.  Modeling and Testing Multi-Threaded Asynchronous Systems with Creol , 2009, Electron. Notes Theor. Comput. Sci..

[65]  Eric Mercer,et al.  A Meta Heuristic for Effectively Detecting Concurrency Errors , 2008, Haifa Verification Conference.

[66]  Christian Trefftz,et al.  An Investigation on Testing of Parallelized Code with OpenMP , 2007, IWOMP.

[67]  Richard N. Taylor,et al.  A Concurrency Analysis Tool Suite: Rationale, Design, and Preliminary Experience , 1992 .

[68]  Ronald A. Olsson.  Reproducible execution of SR programs , 1999 .

[69]  Koushik Sen,et al.  CalFuzzer: An Extensible Active Testing Framework for Concurrent Programs , 2009, CAV.

[70]  Bogdan WiszniewskiFaculty,et al.  A Method for Determining Testing Scenarios for Parallel and Distributed Software , 1996 .

[71]  Richard N. Taylor,et al.  Structural Testing of Concurrent Programs , 1992, IEEE Trans. Software Eng..

[72]  Shaz Qadeer,et al.  CHESS: A Systematic Testing Tool for Concurrent Software , 2007 .

[73]  Jessica Chen.  On Verifying Distributed Multithreaded Java Programs , 2004, Software Quality Journal.

[74]  Eitan Farchi,et al.  Concurrent bug patterns and how to test them , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[75]  James C. Corbett,et al.  Bandera: a source-level interface for model checking Java programs , 2000, ICSE '00.

[76]  Scott D. Stoller,et al.  Runtime analysis of atomicity for multithreaded programs , 2006, IEEE Transactions on Software Engineering.

[77]  Yong-Kee Jun,et al.  MPIRace-Check: Detection of Message Races in MPI Programs , 2007, GPC.

[78]  Krishnendu Chatterjee,et al.  Analyzing the Impact of Change in Multi-threaded Programs , 2010, FASE.

[79]  Baowen Xu,et al.  An approach to analyzing dependency of concurrent programs , 2000, Proceedings First Asia-Pacific Conference on Quality Software.

[80]  Hai-Yin Xu,et al.  A Feasible Strategy for Reachability Testing of Internet-Based Concurrent Programs , 2008, 2008 IEEE International Conference on Networking, Sensing and Control.

[81]  Gwan-Hwan Hwang,et al.  Reachability testing: an approach to testing concurrent software , 1994, Proceedings of 1st Asia-Pacific Software Engineering Conference.

[82]  Grigore Rosu,et al.  Parametric and Sliced Causality , 2007, CAV.

[83]  Richard W. Vuduc,et al.  Improving distributed memory applications testing by message perturbation , 2006, PADTAD '06.

[84]  Jehad Al Dallal,et al.  Anomaly detection in concurrent Java programs using dynamic data flow analysis , 2001, Inf. Softw. Technol..

[85]  Yu Xia Sun,et al.  A framework of reachability testing for Java multithread programs , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[86]  Zengo Furukawa,et al.  A prototype of a concurrent behavior monitoring tool for testing of concurrent programs , 1996, Proceedings 1996 Asia-Pacific Software Engineering Conference.

[87]  Konstantinos Sagonas,et al.  Detection of Asynchronous Message Passing Errors Using Static Analysis , 2011, PADL.

[88]  Zengo Furukawa,et al.  Design and implementation of test-case generation for concurrent programs , 1998, Proceedings 1998 Asia Pacific Software Engineering Conference (Cat. No.98EX240).

[89]  Phil McMinn,et al.  Search-based failure discovery using testability transformations to generate pseudo-oracles , 2009, GECCO.

[90]  Eitan Farchi,et al.  Framework for testing multi‐threaded Java programs , 2003, Concurr. Comput. Pract. Exp..

[91]  G. Gopalakrishnan,et al.  Inspect : A Runtime Model Checker for Multithreaded C Programs , 2008 .

[92]  Richard H. Carver,et al.  A General Model for Reachability Testing of Concurrent Programs , 2004, ICFEM.

[93]  Eitan Farchi,et al.  Applications of synchronization coverage , 2005, PPoPP.

[94]  Tomás Vojnar,et al.  A platform for search-based testing of concurrent software , 2010, PADTAD '10.

[95]  Wei Wang,et al.  Testing Path Generation Algorithm with Network Performance Constraints for Nondeterministic Parallel Programs , 2006, 2006 Seventh International Conference on Web-Age Information Management Workshops.

[96]  Armin Eberlein,et al.  Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing , 2009, Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing.

[97]  Timothy K. Shih,et al.  Software testing and metrics for concurrent computation , 1996, Proceedings 1996 Asia-Pacific Software Engineering Conference.

[98]  J. Dingel,et al.  Mutation Operators for Concurrent Java (J2SE 5.0) , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[99]  Chao Wang,et al.  Contessa: Concurrency Testing Augmented with Symbolic Analysis , 2010, CAV.

[100]  Jianjun Zhao,et al.  A Lightweight and Portable Approach to Making Concurrent Failures Reproducible , 2010, FASE.

[101]  Henryk Krawczyk,et al.  Classification of software defects in parallel programs , 2007 .

[102]  Tomás Vojnar,et al.  Healing data races on-the-fly , 2007, PADTAD '07.

[103]  Robert Cartwright,et al.  ConcJUnit: unit testing for concurrent programs , 2009, PPPJ '09.

[104]  Brandon Lucia,et al.  DMP: deterministic shared memory multiprocessing , 2009, IEEE Micro.

[105]  Richard H. Carver,et al.  A Stateful Approach to Testing Monitors in Multithreaded Programs , 2010, 2010 IEEE 12th International Symposium on High Assurance Systems Engineering.

[106]  R.-D. Yang,et al.  Path analysis testing of concurrent programs , 1992, Inf. Softw. Technol..

[107]  Mounir Benabdenbi,et al.  Distributed online software monitoring of manycore architectures , 2010, 2010 IEEE 16th International On-Line Testing Symposium.

[108]  Shan Lu,et al.  ConMem: detecting severe concurrency bugs through an effect-oriented approach , 2010, ASPLOS XV.

[109]  Richard H. Carver,et al.  Reachability testing of concurrent programs , 2006, IEEE Transactions on Software Engineering.

[110]  Alexandre Petrenko,et al.  Multiplexing of Partially Ordered Events , 2005, TestCom.

[111]  Grigore Rosu,et al.  IMUnit: improved multithreaded unit testing , 2010, IWMSE '10.

[112]  Jessica Chen,et al.  A study on static analysis in network of synchronizing FSMs , 2000, Proceedings Seventh Asia-Pacific Software Engeering Conference. APSEC 2000.

[113]  Sebastian Burckhardt,et al.  Deconstructing concurrency heisenbugs , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[114]  Maria Luisa Villani,et al.  Model Checking Multithreaded Programs by Means of Reduced Models , 2004, LDTA@ETAPS.

[115]  João Lourenço,et al.  Testing patterns for software transactional memory engines , 2007, PADTAD '07.

[116]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[117]  Cosimo Antonio Prete,et al.  A tool for testing of parallel and distributed programs in message-passing environments , 1998, MELECON '98. 9th Mediterranean Electrotechnical Conference. Proceedings (Cat. No.98CH36056).

[118]  Zengo Furukawa,et al.  Coverage Based Testing for Concurrent Software , 2008, 2008 The 28th International Conference on Distributed Computing Systems Workshops.

[119]  Stephen N. Freund,et al.  Exploiting Purity for Atomicity , 2005, IEEE Trans. Software Eng..

[120]  Yoshiaki Kakuda,et al.  A model for concurrent states and its coverage criteria , 2009, 2009 International Symposium on Autonomous Decentralized Systems.

[121]  Lori L. Pollock,et al.  IDENTIFYING REDUNDANT TEST CASESFOR TESTING PARALLEL LANGUAGE CONSTRUCTS , 1997 .

[122]  Richard H. Carver,et al.  Reachability testing of semaphore-based programs , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[123]  Simone do Rocio Senger de Souza,et al.  Structural testing criteria for message‐passing parallel programs , 2008, Concurr. Comput. Pract. Exp..

[124]  Zuohua Ding,et al.  Test Case Generation of Concurrent Programs Based on Event Graph , 2008, 2009 Fifth International Joint Conference on INC, IMS and IDC.

[125]  Timothy K. Shih,et al.  Task decomposition testing and metrics for concurrent programs , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[126]  Yong Rae Kwon,et al.  Generating Test Sequences from Statecharts for Concurrent Program Testing , 2006, IEICE Trans. Inf. Syst..

[127]  Yuanyuan Zhou,et al.  AVIO: Detecting Atomicity Violations via Access-Interleaving Invariants , 2007, IEEE Micro.

[128]  Jürgen Dingel,et al.  Automating comprehensive safety analysis of concurrent programs using verisoft and TXL , 2004, SIGSOFT '04/FSE-12.

[129]  Jeff Lei.  Non-deterministic Testing of Concurrent Programs , .

[130]  Richard H. Carver,et al.  Replay and testing for concurrent programs , 1991, IEEE Software.

[131]  Willem Visser,et al.  Efficient Testing of Concurrent Programs with Abstraction-Guided Symbolic Execution , 2009, SPIN.

[132]  D. Kung,et al.  A combinatorial testing strategy for concurrent programs , 2007, Softw. Test. Verification Reliab..

[133]  Roy Patrick Tan,et al.  Automated Black Box Testing Tool for a Parallel Programming Library , 2009, 2009 International Conference on Software Testing Verification and Validation.

[134]  Simone R. S. Souza,et al.  Concurrent Software Testing : A Systematic Review ? , 2010 .

[135]  Jun Chen,et al.  Testing concurrent programs using value schedules , 2007, ASE.

[136]  Sebastian Burckhardt,et al.  Preemption Sealing for Efficient Concurrency Testing , 2010, TACAS.

[137]  Xiao Ma,et al.  Effective generation of test sequences for structural testing of concurrent programs , 2005, 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'05).

[138]  Madan Musuvathi,et al.  Fair stateless model checking , 2008, PLDI '08.

[139]  Simone do Rocio Senger de Souza,et al.  Structural Testing for Semaphore-Based Multithread Programs , 2008, ICCS.

[140]  Byeong Man Kim,et al.  The design and implementation of automata-based testing environment for Java multi-thread programs , 2001, Proceedings Eighth Asia-Pacific Software Engineering Conference.

[141]  Assaf Schuster,et al.  Efficient on-the-fly data race detection in multithreaded C++ programs , 2003, PPoPP '03.

[142]  Magdy S. Abadir,et al.  Coverage metrics for verification of concurrent SystemC designs using mutation testing , 2010, 2010 IEEE International High Level Design Validation and Test Workshop (HLDVT).

[143]  Robert Cartwright,et al.  A framework for testing concurrent programs , 2011 .

[144]  Cyrille Artho.  Finding faults in multi-threaded programs , 2001 .

[145]  Jiannong Cao,et al.  A robust monitor construct with runtime fault detection , 2006, Concurr. Comput. Pract. Exp..

[146]  Zijiang Yang,et al.  HAVE: Detecting Atomicity Violations via Integrated Dynamic and Static Analysis , 2009, FASE.

[147]  Wang Dingxing,et al.  Analyzing nondeterminacy of message passing programs , 1996 .

[148]  R.-D. Yang,et al.  A path analysis approach to concurrent program testing , 1990, Ninth Annual International Phoenix Conference on Computers and Communications. 1990 Conference Proceedings.

[149]  Matthew B. Dwyer,et al.  Saturation-based testing of concurrent programs , 2009, ESEC/FSE '09.

[150]  Jianjun Zhao,et al.  Multithreaded dependence graphs for concurrent Java programs , 1999, 1999 Proceedings International Symposium on Software Engineering for Parallel and Distributed Systems.

[151]  Darko Marinov,et al.  MuTMuT: Efficient Exploration for Mutation Testing of Multithreaded Code , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[152]  Jiannong Cao,et al.  Run-time fault detection in monitor based concurrent programming , 2001, 2001 International Conference on Dependable Systems and Networks.

[153]  Lin Yan,et al.  Correctness Analysis Based on Testing and Checking for OpenMP Programs , 2009, 2009 Fourth ChinaGrid Annual Conference.

[154]  Koushik Sen,et al.  Effective random testing of concurrent programs , 2007, ASE.

[155]  Shan Lu,et al.  A study of interleaving coverage criteria , 2007, ESEC-FSE '07.

[156]  Lori L. Pollock,et al.  All‐uses testing of shared memory parallel programs , 2003, Softw. Test. Verification Reliab..

[157]  Michael Oberhuber,et al.  Elimination of Nondeterminacy for Testing and Debugging Parallel Programs , 1995, AADEBUG.

[158]  Zvonimir Rakamaric,et al.  Delay-bounded scheduling , 2011, POPL '11.

[159]  Jong-Deok Choi,et al.  Isolating failure-inducing thread schedules , 2002, ISSTA '02.

[160]  Jun Chen,et al.  Towards a better collaboration of static and dynamic analyses for testing concurrent programs , 2008, PADTAD '08.