Testing with Guarantees and the Failure of Regression Testing in eXtreme Programming

The eXtreme Programming (XP) method eschews all formal design, but compensates for this by rigorous unit testing. Test-sets, which constitute the only enduring specification, are intuitively developed and so may not be complete. This paper presents a method for generating complete unit test-sets for objects, based on simple finite state machines. Using this method, it is possible to prove that saved regression test-sets do not provide the expected guarantees of correctness when applied to modified or extended objects. Such objects, which pass the saved tests, may yet contain introduced faults. This puts the whole practice of regression testing in XP into question. To obtain the same level of guarantee, tests must be regenerated from scratch for the extended object. A notion of guaranteed, repeatable quality after testing is defined.

[1]  Dimitris Dranidis,et al.  Formal modelling of use cases with X-machines , 2003 .

[2]  Laurie Williams,et al.  Extreme Programming and Agile Methods — XP/Agile Universe 2002 , 2002, Lecture Notes in Computer Science.

[3]  Florentin Ipate,et al.  An integration testing method that is proved to find all faults , 1997, Int. J. Comput. Math..

[4]  Saralees Nadarajah,et al.  Letter to the editor , 2007, Int. Trans. Oper. Res..

[5]  Mike Holcombe.  XP after Enron-Can It Survive? , 2003, XP.

[6]  Florentin Ipate,et al.  Correct systems - building a business process solution , 1998, Applied computing.

[7]  Marian Gheorghe,et al.  Functional Test Generation for Extreme Programming , 2001 .

[8]  S. A. Schuman,et al.  Object-oriented subsystem specification , 1987 .

[9]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[10]  Matt Stephens,et al.  Extreme Programming Refactored: The Case Against XP , 2003, Apress.

[11]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[12]  John D. McGregor,et al.  Integrated object-oriented testing and development processes , 1994, CACM.

[13]  Martin Fowler,et al.  Planning Extreme Programming , 2000 .

[14]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[15]  John Daniels,et al.  Designing object systems: object-oriented modelling with Syntropy , 1995 .

[16]  John D. Mcgregor.  Constructing functional test cases using incrementally de~ned state machines , 1994 .

[17]  John D. McGregor,et al.  A note on inheritance and state machines , 1993, SOEN.

[18]  P. David Stotts,et al.  An Informal Formal Method for Systematic JUnit Test Case Generation , 2002, XP/Agile Universe.

[19]  Philippe Kruchten,et al.  The Rational Unified Process: An Introduction , 1998 .

[20]  Jing Yuan,et al.  Where Do Unit Tests Come from? , 2003, XP.

[21]  Michele Marchesi,et al.  Extreme Programming and Agile Processes in Software Engineering , 2003, Lecture Notes in Computer Science.