OSMOSE: automatic structural testing of executables

Verification is usually performed on a high‐level view of the software, either specification or program source code. However, in certain circumstances verification is more relevant when performed at the machine‐code level. This paper focuses on automatic test data generation from a stand‐alone executable. Low‐level analysis is much more difficult than high‐level analysis since even the control‐flow graph is not available and bit‐level instructions have to be modelled faithfully. The paper shows how ‘path‐based’ structural test data generation can be adapted from structured language to machine code, using both state‐of‐the‐art technologies and innovative techniques. The results have been implemented in a tool named OSMOSE and encouraging experiments have been conducted. Copyright © 2010 John Wiley & Sons, Ltd.

[1]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[2]  Z. Hanna,et al.  A Lazy and Layered SMT ( B V ) Solver for Hard Industrial Verification Problems ⋆ , 2007 .

[3]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[4]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[5]  Krzysztof R. Apt,et al.  Principles of constraint programming , 2003 .

[6]  Xavier Rival,et al.  Certification of compiled assembly code by invariant translation , 2004, International Journal on Software Tools for Technology Transfer.

[7]  Javier Esparza,et al.  jMoped: A Test Environment for Java Programs , 2007, CAV.

[8]  Nikolai Tillmann,et al.  Fitness-guided path exploration in dynamic symbolic execution , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[9]  Toby Walsh,et al.  Handbook of Constraint Programming , 2006, Handbook of Constraint Programming.

[10]  Arnaud Gotlieb,et al.  Symbolic execution of floating‐point computations , 2006, Softw. Test. Verification Reliab..

[11]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[12]  Thomas W. Reps,et al.  CodeSurfer/x86-A Platform for Analyzing x86 Executables , 2005, CC.

[13]  Somesh Jha,et al.  Weighted pushdown systems and their application to interprocedural dataflow analysis , 2003, Sci. Comput. Program..

[14]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[15]  Peter J. Stuckey,et al.  Global difference constraint propagation for finite domain solvers , 2008, PPDP.

[16]  A. Jefferson Offutt,et al.  The dynamic domain reduction procedure for test data generation , 1999, Softw. Pract. Exp..

[17]  Thomas W. Reps,et al.  A System for Generating Static Analyzers for Machine Instructions , 2008, CC.

[18]  Thomas W. Reps,et al.  WYSINWYX: What you see is not what you eXecute , 2005, TOPL.

[19]  Bogdan Korel,et al.  Automated test data generation for programs with procedures , 1996, ISSTA '96.

[20]  Arnaud Gotlieb,et al.  INKA: TEN YEARS AFTER THE FIRST IDEAS , 2006 .

[21]  Thomas W. Reps,et al.  Analyzing Memory Accesses in x86 Executables , 2004, CC.

[22]  Pascale Le Gall,et al.  Generation of All-Paths Unit Test with Function Calls , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[23]  Fabrizio Ferrandi,et al.  Functional verification for SystemC descriptions using constraint solving , 2002, Proceedings 2002 Design, Automation and Test in Europe Conference and Exhibition.

[24]  Arnaud Gotlieb,et al.  Automatic test data generation using constraint solving techniques , 1998, ISSTA '98.

[25]  Andrew S. Tanenbaum,et al.  Structured Computer Organization , 1976 .

[26]  Nikolai Tillmann,et al.  Demand-Driven Compositional Symbolic Execution , 2008, TACAS.

[27]  Philippe Herrmann,et al.  Structural Testing of Executables , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[28]  Alan J. Hu,et al.  Calysto: scalable and precise extended static checking , 2008, ICSE.

[29]  Helmut Veith,et al.  Jakstab: A Static Analysis Platform for Binaries , 2008, CAV.

[30]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[31]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[32]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[33]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[34]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[35]  Daniel Kroening,et al.  Decision Procedures - An Algorithmic Point of View , 2008, Texts in Theoretical Computer Science. An EATCS Series.

[36]  Mary Lou Soffa,et al.  UNA based iterative test data generation and its evaluation , 1999, 14th IEEE International Conference on Automated Software Engineering.

[37]  Armin Biere,et al.  Boolector: An Efficient SMT Solver for Bit-Vectors and Arrays , 2009, TACAS.

[38]  Bruno Marre,et al.  On-the-fly generation of k-path tests for C functions , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[39]  Mary Lou Soffa,et al.  Automated test data generation using an iterative relaxation method , 1998, SIGSOFT '98/FSE-6.

[40]  Helmut Veith,et al.  An Abstract Interpretation-Based Framework for Control Flow Reconstruction from Binaries , 2008, VMCAI.

[41]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.

[42]  Krzysztof R. Apt,et al.  Constraint logic programming using Eclipse , 2007 .

[43]  Zhihong Zeng,et al.  Functional Test Generation using Constraint Logic Programming , 2001, VLSI-SOC.

[44]  Javier Esparza,et al.  A BDD-Based Model Checker for Recursive Programs , 2001, CAV.

[45]  Xavier Leroy,et al.  Formal Verification of a Memory Model for C-Like Imperative Languages , 2005, ICFEM.

[46]  Arnaud Gotlieb,et al.  Constraint-Based Software Testing , 2008 .

[47]  Bruno Marre,et al.  Test sequences generation from LUSTRE descriptions: GATEL , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[48]  Zhenkai Liang,et al.  BitScope: Automatically Dissecting Malicious Binaries , 2007 .

[49]  Philippe Herrmann,et al.  Pruning the Search Space in Path-Based Test Generation , 2009, 2009 International Conference on Software Testing Verification and Validation.