Constraint programming for random testing of a trading system

Financial markets use complex computer trading systems whose failures can cause serious economic damage, making reliability a major concern. Automated random testing has been shown to be useful in finding defects in these systems, but its inherent test oracle problem (automatic generation of the expected system output) is a drawback that has typically prevented its application on a larger scale. Two main tasks have been carried out in this thesis as a solution to the test oracle problem. First, an independent model of a real trading system based on constraint programming, a method for solving combinatorial problems, has been created. Then, the model has been integrated as a true test oracle in automated random tests. The test oracle maintains the expected state of an order book throughout a sequence of random trade order actions, and provides the expected output of every auction triggered in the order book by generating a corresponding constraint program that is solved with the aid of a constraint programming system. Constraint programming has allowed the development of an inexpensive, yet reliable test oracle. In 500 random test cases, the test oracle has detected two system failures. These failures correspond to defects that had been present for several years without being discovered neither by less complete oracles nor by the application of more systematic testing approaches. The main contributions of this thesis are: (1) empirical evidence of both the suitability of applying constraint programming to solve the test oracle problem and the effectiveness of true test oracles in random testing, and (2) a first attempt, as far as the author is aware, to model a non-theoretical continuous double auction using constraint programming.

[1]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[2]  D Epstein.  On the NP-completeness of cryptarithms , 1987, SIGA.

[3]  Young U. Ryu.  Hierarchical constraint satisfaction of multilateral trade matching in commodity auction markets , 1997, Ann. Oper. Res..

[4]  Dick Hamlet.  When only random testing will do , 2006, RT '06.

[5]  Toby Walsh,et al.  Handbook of Constraint Programming , 2006, Handbook of Constraint Programming.

[6]  Lutz Prechelt,et al.  An Empirical Comparison of Seven Programming Languages , 2000, Computer.

[7]  Avi Ziv,et al.  Using a constraint satisfaction formulation and solution techniques for random test program generation , 2002, IBM Syst. J..

[8]  Toby Walsh,et al.  Handbook of Constraint Programming (Foundations of Artificial Intelligence) , 2006 .

[9]  Rex Black,et al.  Foundations of Software Testing ISTQB Certification , 2006 .

[10]  Douglas Hoffman,et al.  Using Oracles in Test Automation , 2001 .

[11]  Joel Hasbrouck,et al.  Empirical Market Microstructure: The Institutions, Economics, and Econometrics of Securities Trading , 2007 .

[12]  Ho Geun Lee,et al.  Electronic Call Market for Commodity Transactions: Design of Computer-Mediated Order Matching System , 1998, J. Organ. Comput. Electron. Commer..

[13]  Frédéric Peschanski,et al.  A Constraint Logic Programming Approach to Automated Testing , 2008, ICLP.

[14]  John G. Taylor.  Introduction to the Financial Markets , 2002, Neural Networks and the Financial Markets.

[15]  Ho Geun Lee,et al.  A hybrid approach of linear programming and logic modeling for the market core of sealed bid auctions , 1997, Ann. Oper. Res..

[16]  Brian Falkenhainer,et al.  Dynamic Constraint Satisfaction Problems , 1990, AAAI.

[17]  Andrew B. Whinston,et al.  Solving the combinatorial double auction problem , 2005, Eur. J. Oper. Res..

[18]  Andrew B. Whinston,et al.  The Internet and the future of financial markets , 2000, CACM.

[19]  Arnaud Gotlieb,et al.  Automatic test data generation using constraint solving techniques , 1998, ISSTA '98.

[20]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[21]  Christophe Meudec.  ATGen: automatic test data generation using constraint logic programming and symbolic execution † , 2001, Softw. Test. Verification Reliab..

[22]  Guido Tack,et al.  Constraint propagation: models, techniques, implementation , 2009 .

[23]  Pravin Varaiya,et al.  The Combinatorial Seller ’ s Bid Double Auction : An Asymptotically Efficient Market Mechanism * , 2006 .

[24]  Douglas Hoffman.  Mutating Automated Tests STAR East 2000 , 2000 .

[25]  M. Petit,et al.  An ongoing work on statistical structural testing via probabilistic concurrent constraint programming , 2004, Proceedings. 2004 First International Workshop on Model, Design and Validation, 2004..

[26]  Horst Zuse,et al.  A Framework of Software Measurement , 1998 .

[27]  Min Xie,et al.  Software Reliability Modelling , 1991, Series on Quality, Reliability and Engineering Statistics.

[28]  Andreas Zeller,et al.  Why Programs Fail: A Guide to Systematic Debugging , 2005 .

[29]  Douglas Hoffman.  A Taxonomy for Test Oracles , 1998 .

[30]  Mikael Z. Lagerkvist.  Techniques for Efficient Constraint Propagation , 2008 .

[31]  Andreas Lundgren.  M ASTER T HESIS-ABSTRACTION LEVELS OF AUTOMATED TEST SCRIPTS , 2008 .

[32]  Vice President,et al.  An Introduction to Financial Markets , 2010 .

[33]  Ho Soo Lee,et al.  Computational Aspects of Clearing Continuous Call Double Auctions with Assignment Constraints and Indivisible Demand , 2001, Electron. Commer. Res..