A Survey on Data-Flow Testing

Data-flow testing (DFT) is a family of testing strategies designed to verify the interactions between each program variable’s definition and its uses. Such a test objective of interest is referred to as a def-use pair. DFT selects test data with respect to various test adequacy criteria (i.e., data-flow coverage criteria) to exercise each pair. The original conception of DFT was introduced by Herman in 1976. Since then, a number of studies have been conducted, both theoretically and empirically, to analyze DFT’s complexity and effectiveness. In the past four decades, DFT has been continuously concerned, and various approaches from different aspects are proposed to pursue automatic and efficient data-flow testing. This survey presents a detailed overview of data-flow testing, including challenges and approaches in enforcing and automating it: (1) it introduces the data-flow analysis techniques that are used to identify def-use pairs; (2) it classifies and discusses techniques for data-flow-based test data generation, such as search-based testing, random testing, collateral-coverage-based testing, symbolic-execution-based testing, and model-checking-based testing; (3) it discusses techniques for tracking data-flow coverage; (4) it presents several DFT applications, including software fault localization, web security testing, and specification consistency checking; and (5) it summarizes recent advances and discusses future research directions toward more practical data-flow testing.

[1]  Hasan Ural,et al.  Test generation based on control and data dependencies within system specifications in SDL , 2000, Comput. Commun..

[2]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[3]  Stephan Merz,et al.  Model Checking , 2000 .

[4]  Ahmed S. Ghiduk A New Software Data-Flow Testing Approach via Ant Colony Algorithms , 2010 .

[5]  Sriram K. Rajamani,et al.  The SLAM project: debugging system software via static analysis , 2002, POPL '02.

[6]  Hasan Ural,et al.  Dependence Testing: Extending Data Flow Testing with Control Dependence , 2005, TestCom.

[7]  Elaine J. Weyuker,et al.  The Cost of Data Flow Testing: An Empirical Study , 1990, IEEE Trans. Software Eng..

[8]  Zuohua Ding,et al.  Test Data Generation for Derived Types in C Program , 2009, 2009 Third IEEE International Symposium on Theoretical Aspects of Software Engineering.

[9]  A. Jefferson Offutt,et al.  The dynamic domain reduction procedure for test data generation , 1999, Softw. Pract. Exp..

[10]  Raúl A. Santelices,et al.  Subsumption of program entities for efficient coverage and monitoring , 2006, SOQUA '06.

[11]  Marc Roper,et al.  Investigating data-flow coverage of classes using evolutionary algorithms , 2007, GECCO '07.

[12]  Giovanni Denaro,et al.  Enhancing structural software coverage by incrementally computing branch executability , 2011, Software Quality Journal.

[13]  Alessandro Orso,et al.  Interclass testing of object oriented software , 2002, Eighth IEEE International Conference on Engineering of Complex Computer Systems, 2002. Proceedings..

[14]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[15]  Alex Groce,et al.  Modular verification of software components in C , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[16]  Jon Edvardsson,et al.  A Survey on Automatic Test Data Generation , 2002 .

[17]  Elaine J. Weyuker,et al.  More Experience with Data Flow Testing , 1993, IEEE Trans. Software Eng..

[18]  Moheb R. Girgis Automatic Test Data Generation for Data Flow Testing Using a Genetic Algorithm , 2005, J. Univers. Comput. Sci..

[19]  Raúl A. Santelices,et al.  Efficiently monitoring data-flow test coverage , 2007, ASE.

[20]  Ilona Bluemke,et al.  Dataflow Approach to Testing Java Programs , 2009, 2009 Fourth International Conference on Dependability of Computer Systems.

[21]  Barbara G. Ryder,et al.  Interprocedural Def-Use Associations for C Systems with Single Level Pointers , 1994, IEEE Trans. Software Eng..

[22]  Qian Yang,et al.  A survey of coverage based testing tools , 2006, AST '06.

[23]  Elaine J. Weyuker,et al.  Asset: A System to Select and Evaluate Tests , 2019 .

[24]  Andreas Stefik,et al.  Testing coupling relationships in object‐oriented programs , 2010, Softw. Test. Verification Reliab..

[25]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[26]  Giuliano Antoniol,et al.  MC/DC automatic test input data generation , 2009, GECCO '09.

[27]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[28]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[29]  Marcos Lordello Chaim,et al.  Data-Flow Testing in the Large , 2014, ICST.

[30]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[31]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[32]  Mauro Baluda Automatic structural testing with abstraction refinement and coarsening , 2011, ESEC/FSE '11.

[33]  Akbar Siami Namin,et al.  The influence of size and coverage on test suite effectiveness , 2009, ISSTA.

[34]  Alessandra Gorla,et al.  Contextual Integration Testing of Classes , 2008, FASE.

[35]  Alex Groce,et al.  Modular verification of software components in C , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[36]  Suresh Gyan,et al.  A Hybrid PSO Approach to Automate Test Data Generation for Data Flow Coverage with Dominance Concepts , 2011 .

[37]  W. Eric Wong,et al.  Coverage testing of Java programs and components , 2005, Sci. Comput. Program..

[38]  Dirk Beyer,et al.  CPAchecker: A Tool for Configurable Software Verification , 2009, CAV.

[39]  Michael Hicks,et al.  Directed Symbolic Execution , 2011, SAS.

[40]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[41]  Gregor Snelting,et al.  Efficient path conditions in dependence graphs , 2002, ICSE '02.

[42]  Rajiv Gupta,et al.  Refining data flow information using infeasible paths , 1997, ESEC '97/FSE-5.

[43]  Jianjun Zhao,et al.  A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[44]  Zhendong Su,et al.  Steering symbolic execution to less traveled paths , 2013, OOPSLA.

[45]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[46]  Phyllis G. Frankl The use of Data Flow Information for the Selection and Evaluation of Software Test Data , 2015 .

[47]  David Chenho Kung,et al.  Object-based data flow testing of web applications , 2000, Proceedings First Asia-Pacific Conference on Quality Software.

[48]  Bin Fang,et al.  Automated Coverage-Driven Test Data Generation Using Dynamic Symbolic Execution , 2014, 2014 Eighth International Conference on Software Security and Reliability.

[49]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[50]  James H. Andrews,et al.  Comparing Multi-Point Stride Coverage and dataflow coverage , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[51]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[52]  Xiao Yu,et al.  A Parallel Approach to Concolic Testing with Low-cost Synchronization , 2011, Electron. Notes Theor. Comput. Sci..

[53]  Rajiv Gupta,et al.  A demand-driven analyzer for data flow testing at the integration level , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[54]  Stuart H. Zweben,et al.  A study of the effectiveness of control and data flow testing strategies , 1993, J. Syst. Softw..

[55]  Insup Lee,et al.  Data flow testing as model checking , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[56]  Elaine J. Weyuker,et al.  Data flow-based test adequacy analysis for languages with pointers , 1991, TAV4.

[57]  Doo-Hwan Bae,et al.  A test sequence selection method for statecharts , 2000, Softw. Test. Verification Reliab..

[58]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[59]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[60]  Rajiv Gupta,et al.  A practical framework for demand-driven interprocedural data flow analysis , 1997, TOPL.

[61]  M.J. Harrold Performing data flow testing in parallel , 1994, Proceedings of 8th International Parallel Processing Symposium.

[62]  John Cocke,et al.  A program data flow analysis procedure , 1976, CACM.

[63]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[64]  Luciano Baresi,et al.  TestFul: automatic unit-test generation for Java classes , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[65]  Joseph Robert Horgan,et al.  Fault localization using execution slices and dataflow tests , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[66]  Alessandra Gorla,et al.  Search-based data-flow test generation , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[67]  Mary Jean Harrold,et al.  Using Genetic Algorithms to Aid Test-Data Generation for Data-Flow Coverage , 2007, 14th Asia-Pacific Software Engineering Conference (APSEC'07).

[68]  Marc Roper,et al.  Hybridizing Evolutionary Testing with Artificial Immune Systems and Local Search , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[69]  Alessandro Orso,et al.  Automated Testing of Classes , 2000, ISSTA '00.

[70]  Nahid Shahmehri,et al.  Interprocedural dynamic slicing applied to interprocedural data flow testing , 1993, 1993 Conference on Software Maintenance.

[71]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[72]  Luciano Baresi,et al.  TestFul: An Evolutionary Test Approach for Java , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[73]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[74]  A. Jefferson Offutt,et al.  The Dynamic Domain Reduction Procedure for Test Data Generation: Design and Algorithms , 1994 .

[75]  Giovanni Denaro,et al.  On the Right Objectives of Data Flow Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[76]  Elaine J. Weyuker,et al.  A data flow testing tool , 1985 .

[77]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[78]  Zhendong Su,et al.  Combining Symbolic Execution and Model Checking for Data Flow Testing , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[79]  P. M. Herman,et al.  A Data Flow Analysis Approach to Program Testing , 1976, Aust. Comput. J..

[80]  Ahmed S. Ghiduk,et al.  Automatic Generation of Data Flow Test Paths using a Genetic Algorithm , 2014 .

[81]  M.L. Soffa,et al.  Demand-driven structural testing with dynamic instrumentation , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[82]  Thomas A. Henzinger,et al.  SYNERGY: a new algorithm for property checking , 2006, SIGSOFT '06/FSE-14.

[83]  Alessandra Gorla,et al.  DaTeC: Contextual data flow testing of java classes , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[84]  Giuliano Antoniol,et al.  A static measure of a subset of intra-procedural data flow testing coverage based on node coverage , 1999, CASCON.

[85]  Durga Prasad Mohapatra,et al.  Automatic Test Data Generation for Data Flow Testing Using Particle Swarm Optimization , 2010, IC3.

[86]  Lee Copeland,et al.  A Practitioner's Guide to Software Test Design , 2003 .

[87]  Gordon Fraser,et al.  Testing with model checkers: a survey , 2009, Softw. Test. Verification Reliab..

[88]  Reem Bahgat,et al.  Automatic Test Data Generation Using Data Flow Information , 2000 .

[89]  J. C. Huang,et al.  Detection of Data Flow Anomaly Through Program Instrumentation , 1979, IEEE Transactions on Software Engineering.

[90]  Lori L. Pollock,et al.  The Construction of Contextual Def-Use Associations for Object-Oriented Systems , 2003, IEEE Trans. Software Eng..

[91]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[92]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[93]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[94]  Robert E. Tarjan,et al.  A fast algorithm for finding dominators in a flowgraph , 1979, TOPL.

[95]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (2nd, extended ed.) , 1994 .

[96]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[97]  Elaine J. Weyuker,et al.  Data flow analysis techniques for test data selection , 2015, ICSE '82.

[98]  Mattia Vivanti Dynamic data-flow testing , 2014, ICSE Companion.

[99]  Thomas A. Henzinger,et al.  Lazy abstraction , 2002, POPL '02.

[100]  Antonia Bertolino,et al.  Unconstrained duals and their use in achieving all-uses coverage , 1996, ISSTA '96.

[101]  Mary Lou Soffa,et al.  Efficient computation of interprocedural definition-use chains , 1994, TOPL.

[102]  Sigrid Eldh Software Testing Techniques , 2007 .

[103]  Rohit Ghatol,et al.  An Introduction to Data-Flow Testing , 2006 .

[104]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[105]  Janusz W. Laski,et al.  A Data Flow Oriented Program Testing Strategy , 1983, IEEE Transactions on Software Engineering.

[106]  Jianjun Zhao,et al.  Data-flow-based unit testing of aspect-oriented programs , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[107]  David Chenho Kung,et al.  An agent-based data-flow testing approach for Web applications , 2006, Inf. Softw. Technol..

[108]  T. H. Tse,et al.  Data flow testing of service-oriented workflow applications , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[109]  I. K. Mak,et al.  Adaptive Random Testing , 2004, ASIAN.

[110]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[111]  T. H. Tse,et al.  Data flow testing of service choreography , 2009, ESEC/FSE '09.

[112]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[113]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[114]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[115]  Norbert Oster Automated Generation and Evaluation of Dataflow-Based Test Data for Object-Oriented Software , 2005, QoSA/SOQUA.

[116]  Bin Gu,et al.  Towards Scalable Compositional Test Generation , 2009, 2009 Ninth International Conference on Quality Software.

[117]  Gordon Fraser,et al.  Sound empirical evidence in software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[118]  Ilona Bluemke,et al.  Dataflow Testing of Java Programs with DFC , 2009, CEE-SET.

[119]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[120]  Arnaud Gotlieb,et al.  Symbolic Path-Oriented Test Data Generation for Floating-Point Programs , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[121]  George Candea,et al.  Execution synthesis: a technique for automated software debugging , 2010, EuroSys '10.

[122]  Giovanni Denaro,et al.  Dynamic Data Flow Testing of Object Oriented Systems , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[123]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[124]  Nicos Malevris,et al.  The collateral coverage of data flow criteria when branch testing , 2006, Inf. Softw. Technol..

[125]  Moheb R. Girgis Using symbolic execution and data flow criteria to aid test data selection , 1993, Softw. Test. Verification Reliab..

[126]  Mark Harman,et al.  A multi-objective approach to search-based test data generation , 2007, GECCO '07.

[127]  John A. Clark,et al.  Automatic Test Data Generation for Multiple Condition and MCDC Coverage , 2009, 2009 Fourth International Conference on Software Engineering Advances.

[128]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[129]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[130]  Lori A. Clarke,et al.  A Formal Evaluation of Data Flow Path Selection Criteria , 1989, IEEE Trans. Software Eng..

[131]  Gregg Rothermel,et al.  Performing data flow testing on classes , 1994, SIGSOFT '94.

[132]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[133]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[134]  Marcos Lordello Chaim,et al.  An efficient bitwise algorithm for intra-procedural data-flow testing coverage , 2013, Inf. Process. Lett..

[135]  Giovanni Denaro,et al.  Quantifying the complexity of dataflow testing , 2013, 2013 8th International Workshop on Automation of Software Test (AST).

[136]  Lionel C. Briand,et al.  Adaptive random testing: an illusion of effectiveness? , 2011, ISSTA '11.

[137]  Mark Harman,et al.  Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[138]  Brian A. Malloy,et al.  Data flow testing of parallelized code , 1992, Proceedings Conference on Software Maintenance 1992.

[139]  Bertrand Meyer,et al.  ARTOO , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[140]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[141]  Giovanni Denaro,et al.  Structural coverage of feasible code , 2010, AST '10.

[142]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[143]  Mark Harman,et al.  State aware test case regeneration for improving web application test suite coverage and fault detection , 2012, ISSTA 2012.

[144]  Thomas A. Henzinger,et al.  The software model checker B last : Applications to software engineering , 2007 .

[145]  Mark Harman,et al.  Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet? , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[146]  Robert J. Simmons,et al.  Proofs from Tests , 2008, IEEE Transactions on Software Engineering.

[147]  K. Liaskos,et al.  Automatic Test-Data Generation: An Immunological Approach , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[148]  Yuanyuan Zhang,et al.  Achievements, Open Problems and Challenges for Search Based Software Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[149]  Rong Chen,et al.  Automatic test data generation model by combining dataflow analysis with genetic algorithm , 2009, 2009 Joint Conferences on Pervasive Computing (JCPC).

[150]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[151]  Alessandra Cavarra,et al.  Checking Model Consistency Using Data-Flow Testing , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[152]  Mary Lou Soffa,et al.  Jazz: A Tool for Demand-Driven Structural Testing , 2005, CC.

[153]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[154]  Barbara G. Ryder,et al.  Data-flow-based Testing of Object-Oriented Libraries , 1999 .

[155]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching Time Temporal Logic , 2008, 25 Years of Model Checking.

[156]  共立出版株式会社 コンピュータ・サイエンス : ACM computing surveys , 1978 .

[157]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..