Type Targeted Testing

We present a new technique called type targeted testing, which translates precise refinement types into comprehensive test-suites. The key insight behind our approach is that through the lens of SMT solvers, refinement types can also be viewed as a high-level, declarative, test generation technique, wherein types are converted to SMT queries whose models can be decoded into concrete program inputs. Our approach enables the systematic and exhaustive testing of implementations from high-level declarative specifications, and furthermore, provides a gradual path from testing to full verification. We have implemented our approach as a Haskell testing tool called TARGET, and present an evaluation that shows how TARGET can be used to test a wide variety of properties and how it compares against state-of-the-art testing approaches.

[1]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[2]  Rajesh Subramanyan,et al.  A survey on model-based testing approaches: a systematic review , 2007, WEASELTech '07.

[3]  Brian Marick How to Misuse Code Coverage , 1999 .

[4]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[5]  Colin Runciman,et al.  Smallcheck and lazy smallcheck: automatic exhaustive testing for small values , 2008, Haskell '08.

[6]  Joshua Dunfield,et al.  Refined typechecking with Stardust , 2007, PLPV.

[7]  Ranjit Jhala,et al.  Abstract Refinement Types , 2013, ESOP.

[8]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[9]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[10]  Ranjit Jhala,et al.  LiquidHaskell: experience with refinement types in the real world , 2014, Haskell.

[11]  Frank Pfenning,et al.  Eliminating array bound checking through dependent types , 1998, PLDI.

[12]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[13]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[14]  Nikolaj Bjørner,et al.  Generalized, efficient array decision procedures , 2009, 2009 Formal Methods in Computer-Aided Design.

[15]  Ranjit Jhala,et al.  Refinement types for Haskell , 2014, ICFP.

[16]  Koen Claessen,et al.  Generating constrained random data with uniform distribution , 2014, Journal of Functional Programming.

[17]  Jens Palsberg,et al.  Constrained types for object-oriented languages , 2008, OOPSLA.

[18]  Juan Chen,et al.  Secure distributed programming with value-dependent types , 2011, Journal of Functional Programming.

[19]  Charles Gregory Nelson,et al.  Techniques for program verification , 1979 .

[20]  Greg Nelson,et al.  Extended static checking for Java , 2002, PLDI '02.

[21]  C. Csallner,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[22]  Daniel Jackson,et al.  Alloy: a lightweight object modelling notation , 2002, TSEM.

[23]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[24]  Matthias Felleisen,et al.  Contract Soundness for object-oriented languages , 2001, OOPSLA '01.

[25]  Johan Jeuring,et al.  A generic deriving mechanism for Haskell , 2010, Haskell '10.

[26]  Koen Claessen,et al.  Generating Constrained Random Data with Uniform Distribution , 2014, FLOPS.

[27]  Colin Runciman,et al.  Haskell program coverage , 2007, Haskell '07.

[28]  Viktor Kuncak,et al.  Constraints as control , 2012, POPL '12.

[29]  Margus Veanes,et al.  Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer , 2008, Formal Methods and Testing.