Automated software testing as a service

This paper makes the case for TaaS--automated software testing as a cloud-based service. We present three kinds of TaaS: a "programmer's sidekick" enabling developers to thoroughly and promptly test their code with minimal upfront resource investment; a "home edition" on-demand testing service for consumers to verify the software they are about to install on their PC or mobile device; and a public "certification service," akin to Underwriters Labs, that independently assesses the reliability, safety, and security of software. TaaS automatically tests software, without human involvement from the service user's or provider's side. This is unlike today's "testing as a service" businesses, which employ humans to write tests. Our goal is to take recently proposed techniques for automated testing--even if usable only on to y programs--and make them practical by modifying them to harness the resources of compute clouds. Preliminary work suggests it is technically feasible to do so, and we find that TaaS is also compelling from a social and business point of view.

[1]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[2]  George Candea,et al.  Execution synthesis: a technique for automated software debugging , 2010, EuroSys '10.

[3]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[4]  Michael D. Ernst,et al.  Reducing wasted development time via continuous testing , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[5]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[6]  Philip J. Guo,et al.  Linux Kernel Developer Responses to Static Analysis Bug Reports , 2009, USENIX Annual Technical Conference.

[7]  Dragan Bosnacki,et al.  Multi-Core Model Checking with SPIN , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[8]  Andrew M. Kuhn,et al.  Code Complete , 2005, Technometrics.

[9]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[10]  George Candea,et al.  Cloud9: a software testing service , 2010, OPSR.

[11]  George Candea,et al.  Testing Closed-Source Binary Device Drivers with DDT , 2010, USENIX Annual Technical Conference.

[12]  Sarfraz Khurshid,et al.  Parallel test generation and execution with Korat , 2007, ESEC-FSE '07.

[13]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[14]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).