A First Step in the Design of a Formally Verified Constraint-Based Testing Tool: FocalTest

Constraint-based test data generators rely on SMT or constraint solvers to automatically generate test data (e.g., Pex, Sage, Gatel, PathCrawler, Euclide). However, for some test data generation requests corresponding to particular test objectives, these tools may fail to deliver the expected test data because they focus on efficiency rather than soundness and completeness. We adopt an opposite view in the development of FocalTest, a test data generation tool for Focalize programs. The goal of the tool is to generate an MC/DC-compliant set of test data over the precondition of user-defined program properties. The development of such a correct-by-construction test data generator requires 1) to provide a formally verified translation of Focalize programs and properties into constraint systems; 2) to introduce a formally verified constraint solver able to solve those constraint systems. This paper is concerned with the first step only where we formally demonstrate with Coq the soundness of the translation of an intermediate functional language into a constraint system. This objective requires to formally define the operational semantics of the source language that features the manipulation of concrete data types via pattern-matching and function calls, constructions that are mirrored in the constraint language. Although such a semantics-oriented formalization is only a first step of a larger goal which is to provide a formally verified constraint-based testing tool, we argue that it is an important contribution to the building of more robust software testing tools.