VDTest: An Automated Framework to Support Testing for Virtual Devices

The use of virtual devices in place of physical hardware is increasing in activities such as design, testing and debugging. Yet virtual devices are simply software applications, and like all software they are prone to faults. A full system simulator (FSS), is a class of virtual machine that includes a large set of virtual devices – enough to run the full target software stack. Defects in an FSS virtual device may have cascading effects as the incorrect behavior can be propagated forward to many different platforms as well as to guest programs. In this work we present VDTest, a novel framework for testing virtual devices within an FSS. VDTest begins by generat- ing a test specification obtained through static analysis. It then employs a two-phase testing approach to test virtual components both individually and in combination. It lever- ages a differential oracle strategy, taking advantage of the existence of a physical or golden device to eliminate the need for manually generating test oracles. In an empirical study using both open source and commercial FSSs, we found 64 faults, 83% more than random testing.

[1]  Kim G. Larsen,et al.  Testing Real-Time Systems Using UPPAAL , 2008, Formal Methods and Testing.

[2]  Li Lei,et al.  Post-silicon conformance checking with virtual prototypes , 2013, 2013 50th ACM/EDAC/IEEE Design Automation Conference (DAC).

[3]  Neeraj Suri,et al.  An empirical study of injected versus actual interface errors , 2014, ISSTA 2014.

[4]  Alex Groce,et al.  Randomized Differential Testing as a Prelude to Formal Verification , 2007, 29th International Conference on Software Engineering (ICSE'07).

[5]  John Regehr,et al.  Random testing of interrupt-driven software , 2005, EMSOFT.

[6]  Myra B. Cohen,et al.  Incremental covering array failure characterization in large configuration spaces , 2009, ISSTA.

[7]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[8]  Fabrice Bellard,et al.  QEMU, a Fast and Portable Dynamic Translator , 2005, USENIX ATC, FREENIX Track.

[9]  Christos A. Papachristou,et al.  A design for testability scheme with applications to data path synthesis , 1991, 28th ACM/IEEE Design Automation Conference.

[10]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[11]  Somayeh Sardashti,et al.  The gem5 simulator , 2011, CARN.

[12]  Tal Garfinkel,et al.  Understanding data lifetime via whole system simulation , 2004 .

[13]  Lorenzo Martignoni,et al.  Testing system virtual machines , 2010, ISSTA '10.

[14]  Daniel Aarno,et al.  Full-System Simulation from Embedded to High-Performance Systems , 2010 .

[15]  David W. Binkley,et al.  Interprocedural slicing using dependence graphs , 1990, TOPL.

[16]  Jens Palsberg,et al.  Avrora: scalable sensor network simulation with precise timing , 2005, IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005..

[17]  Katsuro Inoue,et al.  An effective method to control interrupt handler for data race detection , 2010, AST '10.

[18]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[19]  Asim Kadav,et al.  Understanding modern device drivers , 2012, ASPLOS XVII.

[20]  Li Lei,et al.  Symbolic Execution of Virtual Devices , 2013, 2013 13th International Conference on Quality Software.

[21]  George Candea,et al.  Testing Closed-Source Binary Device Drivers with DDT , 2010, USENIX Annual Technical Conference.

[22]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[23]  Lionel C. Briand,et al.  Environment modeling and simulation for automated testing of soft real-time embedded software , 2013, Software & Systems Modeling.

[24]  George Candea,et al.  S2E: a platform for in-vivo multi-path analysis of software systems , 2011, ASPLOS XVI.

[25]  Myra B. Cohen,et al.  Feedback driven adaptive combinatorial testing , 2011, ISSTA '11.

[26]  Lorenzo Martignoni,et al.  A methodology for testing CPU emulators , 2013, TSEM.

[27]  Hareton K. N. Leung,et al.  Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures , 2015, Inf. Softw. Technol..

[28]  Tavis Ormandy.  An Empirical Study into the Security Exposure to Hosts of Hostile Virtualized Environments Tavis , 2007 .