Automated test case generation to validate non-functional software requirements

A software system is bounded by a set of requirements. Functional requirements describe what the system must do, in terms of inputs, the behavior, and outputs. Non-functional requirements describe how well these functional requirements are satisfied, in terms of qualities or constraints on the design or implementation of a system. Both requirements are integral parts of software design specification, and should be revisited constantly during all software development phases. In practice, however, technique support for validating these requirements, does not receive equal emphasis. Techniques for validating functional requirements target all levels of software testing phases, and explore both black box and white box approaches. Techniques for validating non-functional requirements, on the other hand, largely operate in a black box manner, and only focus on system testing level. As a result, most software companies put more efforts in validating functional requirements, and only assess non-functional requirements after functional validation is complete. We propose a set of exhaustive white-box testing techniques that enable cost-effective validation of non-functional requirements from two perspectives. For non-functional requirements defined as qualities of a system, we targeted load testing for the purpose of performance validation. We present a load test suite generation approach that uses symbolic execution to exhaustively traverse program execution paths and produce test cases for the ones that expose worst-case resource consumption scenarios. An assessment of the approach on a set of Java applications shows it generates test suites that induce program response times and memory consumption several times worse than the compared alternatives, it scales to large and complex inputs, and it exposes a diversity of resource consuming program behavior. For non-functional requirements defined as constraints on a system, we present an approach for validating contextual constraints that are imposed by external resources with which the software interacts. The approach amplifies existing tests in an exhaustive manner to validate exception handling constructs that are used to handle such constraints. Our assessment of the approach on a set of Android mobile applications indicates that it can be fully automated, is powerful enough to detect 65% of the faults reported in the bug reports of this kind, and is precise enough that 77% of the detected anomalies correspond to faults fixed by the developers. Combined, the two proposed techniques complement the field of automated software testing by providing exhaustive support for non-functional validation. In this proposal we will discuss the completed work as well as the work yet to be done. Once completed, this research will provide benefits for both researchers and practitioners. For researchers, our work paves the way for research on developing precise white box techniques for validating nonfunctional requirements. For practitioners, our approach provides support for generating non-functional test cases at all levels of software testing phases, especially at the unit testing level, where current tool support for non-functional validation is scarce.