Concolic testing of the multi-sector read operation for flash storage platform software

AbstractIn today’s information society, flash memory has become a virtually indispensable component, particularly for mobile devices. In order for mobile devices to operate successfully, it is essential that flash memory be controlled correctly through flash storage platform software such as the file system, flash translation layer, and low-level device drivers. However, as is typical for embedded software, conventional testing methods often fail to detect hidden flaws in the software due to the difficulty of creating effective test cases. As a different approach, model checking techniques guarantee a complete analysis, but only on a limited scale. In this paper, we describe an empirical study wherein a concolic testing method is applied to the multi-sector read operation for flash storage platform software. This method combines a concrete dynamic execution and a symbolic execution to automatically generate test cases for full path coverage. Through the experiments, we analyze the advantages and weaknesses of the concolic testing approach on the flash storage platform software.

[1]  Moonzoo Kim,et al.  A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study , 2011, IEEE Transactions on Software Engineering.

[2]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[3]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[4]  Moonzoo Kim,et al.  Formal Verification of a Flash Memory Device Driver - An Experience Report , 2008, SPIN.

[5]  Adam Kiezun,et al.  jFuzz: A Concolic Whitebox Fuzzer for Java , 2009, NASA Formal Methods.

[6]  Mark Harman,et al.  Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet? , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[7]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[8]  Moonzoo Kim,et al.  Concolic Testing of the Multi-sector Read Operation for Flash Memory File System , 2009, SBMF.

[9]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[10]  Aaron Stump,et al.  SMT-COMP: Satisfiability Modulo Theories Competition , 2005, CAV.

[11]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[12]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[13]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[14]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[15]  Rupak Majumdar,et al.  Dynamic test input generation for database applications , 2007, ISSTA '07.

[16]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[17]  Anna Philippou,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2018, Lecture Notes in Computer Science.

[18]  Rupak Majumdar,et al.  Directed test generation using symbolic grammars , 2007, ASE.

[19]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[20]  Gerard J. Holzmann,et al.  The SPIN Model Checker , 2003 .

[21]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[22]  Mark Harman,et al.  Handling dynamic data structures in search based testing , 2008, GECCO '08.

[23]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[24]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[25]  David L. Dill,et al.  An Online Proof-Producing Decision Procedure for Mixed-Integer Linear Arithmetic , 2003, TACAS.

[26]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[27]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[28]  Gregg Rothermel,et al.  Infrastructure support for controlled experimentation with software testing and regression testing techniques , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[29]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[30]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[31]  Moonzoo Kim,et al.  Pre-testing Flash Device Driver through Model Checking Techniques , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[32]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[33]  Daniel Jackson,et al.  Formal Modeling and Analysis of a Flash Filesystem in Alloy , 2008, ABZ.

[34]  Bruno Marre,et al.  PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis , 2005, EDCC.

[35]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[36]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[37]  Nikolai Tillmann,et al.  An empirical study of testing file-system-dependent software with mock objects , 2009, 2009 ICSE Workshop on Automation of Software Test.

[38]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[39]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[40]  Moonzoo Kim,et al.  Scalable Distributed Concolic Testing: A Case Study on a Flash Storage Platform , 2010, ICTAC.

[41]  Moonzoo Kim,et al.  SCORE: a scalable concolic testing tool for reliable embedded software , 2011, ESEC/FSE '11.

[42]  Jim Woodcock,et al.  Mechanising a formal model of flash memory , 2009, Sci. Comput. Program..

[43]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[44]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[45]  Nikolai Kosmatov,et al.  Automating structural testing of C programs: Experience with PathCrawler , 2009, 2009 ICSE Workshop on Automation of Software Test.