FSCT: A new fuzzy search strategy in concolic testing

Abstract Context Concolic testing is a promising approach to automate structural test data generation. However, combinatorial explosion of the path space, known as path explosion, and also constrained testing budget, makes achieving high code coverage in concolic testing a challenging task. Objective All branches of the previously explored paths make up the search space of concolic testing and search strategy define the mechanism of choosing branches to be flipped to drive the execution toward testing goals. With regard to the large number of candidate branches, choosing the right branch to continue the search is so crucial and has a direct impact on coverage rate and effort. This paper aims to improve the effectiveness of branch testing by considering the characteristics of paths reaching uncovered branches and presenting a novel search strategy for effectively and efficiently exploring the search space. Method We model the branch selection process in concolic testing as a decision making system and introduce a new Fuzzy Search Strategy in Concolic Testing (FSCT). FSCT chooses a branch to be filliped in which the most suitable path with respect to the proposed coverage factors reaches an uncovered branch with the highest priority and this priority is assigned by the designed fuzzy expert system. The proposed coverage factors effectively help to determine the characteristics of paths. Results We implemented FSCT on top of CREST and evaluated it using several popular benchmarks. The experimental results show that FSCT outperforms the state-of-the-art techniques in terms of coverage rate and coverage effort. Conclusion FSCT helps concolic testing to better cope with path explosion problem and shows its capabilities to achieve higher code coverage while at the same time decreases testing efforts in terms of both runtime and number of iterations.

[1]  Chuen-Chien Lee,et al.  Fuzzy logic in control systems: fuzzy logic controller. II , 1990, IEEE Trans. Syst. Man Cybern..

[2]  Carlo Ghezzi,et al.  Enhancing reuse of constraint solutions to improve symbolic execution , 2015, ISSTA.

[3]  Minh Ngoc Ngo,et al.  Detecting large number of infeasible paths through recognizing their patterns , 2007, ESEC-FSE '07.

[4]  Xiaodong Lin,et al.  An empirical investigation into path divergences for concolic execution using CREST , 2015, Secur. Commun. Networks.

[5]  N. Malevris,et al.  Reducing the effects of infeasible paths in branch testing , 1989 .

[6]  Cristian Cadar,et al.  KATCH: high-coverage testing of software patches , 2013, ESEC/FSE 2013.

[7]  Nikolai Tillmann,et al.  Fitness-guided path exploration in dynamic symbolic execution , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[8]  Corina S. Pasareanu,et al.  Symbolic execution with mixed concrete-symbolic solving , 2011, ISSTA '11.

[9]  Lotfi A. Zadeh,et al.  Fuzzy Sets , 1996, Inf. Control..

[10]  Mark Harman,et al.  n empirical investigation into branch coverage for C programs using CUTE and USTIN , 2010 .

[11]  Bernd Becker,et al.  Picoso - A Parallel Interval Constraint Solver , 2009, PDPTA.

[12]  Dawson R. Engler,et al.  RWset: Attacking Path Explosion in Constraint-Based Test Generation , 2008, TACAS.

[13]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[14]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[15]  Vasanth Bala,et al.  Software Profiling for Hot Path Prediction: Less is More , 2000, ASPLOS.

[16]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[17]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[18]  Chao Wang,et al.  Eliminating Path Redundancy via Postconditioned Symbolic Execution , 2018, IEEE Transactions on Software Engineering.

[19]  Joxan Jaffar,et al.  Lazy Symbolic Execution for Enhanced Learning , 2014, RV.

[20]  Akbar Siami Namin,et al.  The influence of size and coverage on test suite effectiveness , 2009, ISSTA.

[21]  Mike Papadakis,et al.  A Symbolic Execution Tool Based on the Elimination of Infeasible Paths , 2010, 2010 Fifth International Conference on Software Engineering Advances.

[22]  Patrice Godefroid,et al.  Proving memory safety of floating-point computations by combining static and dynamic program analysis , 2010, ISSTA '10.

[23]  Arnaud Gotlieb,et al.  Infeasible path generalization in dynamic symbolic execution , 2015, Inf. Softw. Technol..

[24]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[25]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[26]  Mary Lou Soffa,et al.  Marple: a demand-driven path-sensitive buffer overflow detector , 2008, SIGSOFT '08/FSE-16.

[27]  Fei Peng,et al.  X-Force: Force-Executing Binary Programs for Security Applications , 2014, USENIX Security Symposium.

[28]  Ebrahim H. Mamdani,et al.  An Experiment in Linguistic Synthesis with a Fuzzy Logic Controller , 1999, Int. J. Hum. Comput. Stud..

[29]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[30]  L. Zadeh The role of fuzzy logic in the management of uncertainty in expert systems , 1983 .

[31]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[32]  Jun Wang Computational Intelligence In Manufacturing Handbook , 2000 .

[33]  Yang Bai,et al.  Test Generation for Embedded Executables via Concolic Execution in a Real Environment , 2015, IEEE Transactions on Reliability.

[34]  T. Ross Fuzzy Logic with Engineering Applications , 1994 .

[35]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[36]  Giovanni Denaro,et al.  Bidirectional Symbolic Analysis for Effective Branch Testing , 2016, IEEE Transactions on Software Engineering.

[37]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[38]  Matthew B. Dwyer,et al.  Green: reducing, reusing and recycling constraints in program analysis , 2012, SIGSOFT FSE.

[39]  Sunghun Kim,et al.  How we get there: a context-guided search strategy in concolic testing , 2014, SIGSOFT FSE.

[40]  Michael S. Hsiao,et al.  Strategies for scalable symbolic execution-driven test generation for programs , 2011, Science China Information Sciences.

[41]  Dawei Qi,et al.  Path exploration based on symbolic output , 2013, TSEM.

[42]  Yi Zhou,et al.  Achieving High Branch Coverage with Fewer Paths , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops.

[43]  Mark Harman,et al.  FlagRemover: A testability transformation for transforming loop-assigned flags , 2011, TSEM.

[44]  Yang Zhang,et al.  Path reduction of multiple test points in dynamic symbolic execution , 2017, 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS).

[45]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[46]  Cristian Cadar,et al.  Multi-solver Support in Symbolic Execution , 2013, SMT.

[47]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[48]  Taku Shimosawa,et al.  Parallel SMT Solving and Concurrent Symbolic Execution , 2015, 2015 IEEE Trustcom/BigDataSE/ISPA.

[49]  Alvis Cheuk M. Fong,et al.  Dynamic Symbolic Execution Guided by Data Dependency Analysis for High Structural Coverage , 2012, ENASE.

[50]  Gogul Balakrishnan,et al.  Feedback-directed unit test generation for C/C++ using concolic execution , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[51]  Sriram K. Rajamani,et al.  Compositional may-must program analysis: unleashing the power of alternation , 2010, POPL '10.

[52]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[53]  Chen Fu,et al.  CarFast: achieving higher statement coverage faster , 2012, SIGSOFT FSE.

[54]  Roland H. C. Yap,et al.  The CLP( R ) language and system , 1992, TOPL.

[55]  Ting Chen,et al.  State of the art: Dynamic symbolic execution for automated test generation , 2013, Future Gener. Comput. Syst..

[56]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[57]  Moti Schneider,et al.  Fuzzy Expert System Tools , 1996 .

[58]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[59]  Myra B. Cohen,et al.  Directed test suite augmentation: an empirical investigation , 2015, Softw. Test. Verification Reliab..

[60]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[61]  Giovanni Denaro,et al.  Structural coverage of feasible code , 2010, AST '10.

[62]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[63]  Moonzoo Kim,et al.  Industrial application of concolic testing approach: A case study on libexif by using CREST-BV and KLEE , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[64]  Jorge A. Navas,et al.  TRACER: A Symbolic Execution Tool for Verification , 2012, CAV.

[65]  Qinghua Zheng,et al.  Dependence Guided Symbolic Execution , 2017, IEEE Transactions on Software Engineering.

[66]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[67]  Siau-Cheng Khoo,et al.  Goal-oriented dynamic test generation , 2015, Inf. Softw. Technol..

[68]  Mohammad Reza Keyvanpour,et al.  A Classification Framework for Automatic Test Case Generation Techniques for web applications , 2013 .

[69]  Hee Beng Kuan Tan,et al.  Heuristics-based infeasible path detection for dynamic test data generation , 2008, Inf. Softw. Technol..

[70]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[71]  Giovanni Denaro,et al.  Enhancing structural software coverage by incrementally computing branch executability , 2011, Software Quality Journal.

[72]  Jon Edvardsson,et al.  A Survey on Automatic Test Data Generation , 2002 .

[73]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[74]  Marimuthu Palaniswami,et al.  Implementation of fuzzy systems , 1998, Fuzzy logic and expert systems applications.

[75]  Hamidreza Rashidy Kanan,et al.  AVCD-FRA: A novel solution to automatic video cut detection using fuzzy-rule-based approach , 2013, Comput. Vis. Image Underst..

[76]  Cristian Cadar,et al.  make test-zesti: A symbolic execution solution for improving regression testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[77]  Tao Xie,et al.  DSD-Crasher: A hybrid analysis tool for bug finding , 2008 .

[78]  Giovanni Denaro,et al.  Software testing with code-based test generators: data and lessons learned from a case study with an industrial software component , 2013, Software Quality Journal.

[79]  Xiangyu Zhang,et al.  Scaling Up Symbolic Analysis by Removing Z-Equivalent States , 2014, TSEM.

[80]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[81]  Roberto Baldoni,et al.  A Survey of Symbolic Execution Techniques , 2016, ACM Comput. Surv..

[82]  Chuen-Chien Lee FUZZY LOGIC CONTROL SYSTEMS: FUZZY LOGIC CONTROLLER - PART I , 1990 .

[83]  Ajith Abraham,et al.  Rule-Based Expert Systems , 2005 .

[84]  Karl N. Levitt,et al.  SELECT - a formal system for testing and debugging programs by symbolic execution , 1975, Reliable Software.

[85]  Man Gyun Na,et al.  Prediction of major transient scenarios for severe accidents of nuclear power plants , 2004 .

[86]  Hakjoo Oh,et al.  Automatically Generating Search Heuristics for Concolic Testing , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[87]  George Candea,et al.  Efficient state merging in symbolic execution , 2012, Software Engineering.

[88]  Zhendong Su,et al.  Steering symbolic execution to less traveled paths , 2013, OOPSLA.

[89]  Patrice Godefroid,et al.  Automatic partial loop summarization in dynamic test generation , 2011, ISSTA '11.

[90]  Rupak Majumdar,et al.  Testing for buffer overflows with length abstraction , 2008, ISSTA '08.

[91]  Jorge A. Navas,et al.  Boosting concolic testing via interpolation , 2013, ESEC/FSE 2013.

[92]  Boby George,et al.  A structured experiment of test-driven development , 2004, Inf. Softw. Technol..

[93]  Marie-Laure Potet,et al.  Guided Dynamic Symbolic Execution Using Subgraph Control-Flow Information , 2016, SEFM.

[94]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.