Test execution checkpointing for web applications

Test isolation is a prerequisite for the correct execution of test suites on web applications. We present Test Execution Checkpointing, a method for efficient test isolation. Our method instruments web applications to support checkpointing and exploits this support to isolate and optimize tests. We have implemented and evaluated this method on five popular PHP web applications. The results show that our method not only provides test isolation essentially for free, it also reduces testing time by 44% on average.

[1]  Jon Howell,et al.  Mugshot: Deterministic Capture and Replay for JavaScript Applications , 2010, NSDI.

[2]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[3]  E. N. Elnozahy,et al.  Checkpointing for peta-scale systems: a look into the future of practical rollback-recovery , 2004, IEEE Transactions on Dependable and Secure Computing.

[4]  Daniel Marques,et al.  Automated application-level checkpointing of MPI programs , 2003, PPoPP '03.

[5]  A. Jefferson Offutt,et al.  Testing Web applications by modeling with FSMs , 2005, Software & Systems Modeling.

[6]  Amin Milani Fard,et al.  An empirical study of bugs in test code , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[7]  Donald Ervin Knuth,et al.  The Art of Computer Programming, 2nd Ed. (Addison-Wesley Series in Computer Science and Information , 1978 .

[8]  Yan Tang,et al.  Efficient checkpointing of java software using context-sensitive capture and replay , 2007, ESEC-FSE '07.

[9]  Michael D. Ernst,et al.  When Tests Collide: Evaluating and Coping with the Impact of Test Dependence , 2015 .

[10]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[11]  Julia L. Lawall,et al.  Efficient incremental checkpointing of Java programs , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[12]  Lori L. Pollock,et al.  Composing a framework to automate testing of operational Web-based software , 2004, 20th IEEE International Conference on Software Maintenance, 2004. Proceedings..

[13]  Gregg Rothermel,et al.  On-demand test suite reduction , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[14]  Angelo Gargantini,et al.  Evolutionary Testing of PHP Web Applications with WETT , 2012, SSBSE.

[15]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.

[16]  Jong-Deok Choi,et al.  Deterministic replay of Java multithreaded applications , 1998, SPDT '98.

[17]  Saurabh Sinha,et al.  Efficient and flexible GUI test execution via test merging , 2013, ISSTA.

[18]  Kivanç Muslu,et al.  Finding bugs by isolating unit tests , 2011, ESEC/FSE '11.

[19]  Tristan Buchs.  Checkpointing-based testing , 2015 .

[20]  Tsong Yueh Chen,et al.  A simulation study on some heuristics for test suite reduction , 1998, Inf. Softw. Technol..

[21]  Emily Hill,et al.  Automated replay and failure detection for web applications , 2005, ASE '05.

[22]  Gordon Fraser,et al.  Redundancy Based Test-Suite Reduction , 2007, FASE.

[23]  Michael D. Ernst,et al.  Empirically revisiting the test independence assumption , 2014, ISSTA 2014.

[24]  Mats Per Erik Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[25]  Yuanyuan Zhou,et al.  Rx: treating bugs as allergies---a safe method to survive software failures , 2005, SOSP '05.

[26]  Donald E. Knuth,et al.  The art of computer programming, volume 3: (2nd ed.) sorting and searching , 1998 .

[27]  Sarfraz Khurshid,et al.  Efficiently Running Test Suites Using Abstract Undo Operations , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[28]  Willy Zwaenepoel,et al.  The performance of consistent checkpointing , 1992, [1992] Proceedings 11th Symposium on Reliable Distributed Systems.

[29]  Gail E. Kaiser,et al.  Unit test virtualization with VMVM , 2014, ICSE.

[30]  Jonathan Bell,et al.  Detecting, isolating, and enforcing dependencies among and within test cases , 2014, SIGSOFT FSE.

[31]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[32]  Hong Mei,et al.  An experimental comparison of four test suite reduction techniques , 2006, ICSE.

[33]  Gail E. Kaiser,et al.  Efficient dependency detection for safe Java test acceleration , 2015, ESEC/SIGSOFT FSE.

[34]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[35]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[36]  Zhongsheng Qian,et al.  Test Case Generation and Optimization for User Session-based Web Application Testing , 2010, J. Comput..

[37]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[38]  Darko Marinov,et al.  Reliable testing: detecting state-polluting tests to prevent test dependency , 2015, ISSTA.

[39]  Neelam Gupta,et al.  A concept analysis inspired greedy algorithm for test suite minimization , 2005, PASTE '05.

[40]  Angelo Gargantini,et al.  Model-Driven Testing for Web Applications Using Abstract State Machines , 2012, ICWE Workshops.

[41]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[42]  Peter K. Szwed,et al.  Application-level checkpointing for shared memory programs , 2004, ASPLOS XI.

[43]  Giuseppe A. Di Lucca,et al.  Testing Web applications , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[44]  Paolo Tonella,et al.  Analysis and testing of Web applications , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.