A systematic mapping study of web application testing

Abstract Context The Web has had a significant impact on all aspects of our society. As our society relies more and more on the Web, the dependability of web applications has become increasingly important. To make these applications more dependable, for the past decade researchers have proposed various techniques for testing web-based software applications. Our literature search for related studies retrieved 147 papers in the area of web application testing, which have appeared between 2000 and 2011. Objective As this research area matures and the number of related papers increases, it is important to systematically identify, analyze, and classify the publications and provide an overview of the trends in this specialized field. Method We review and structure the body of knowledge related to web application testing through a systematic mapping (SM) study. As part of this study, we pose two sets of research questions, define selection and exclusion criteria, and systematically develop and refine a classification schema. In addition, we conduct a bibliometrics analysis of the papers included in our study. Results Our study includes a set of 79 papers (from the 147 retrieved papers) published in the area of web application testing between 2000 and 2011. We present the results of our systematic mapping study. Our mapping data is available through a publicly-accessible repository. We derive the observed trends, for instance, in terms of types of papers, sources of information to derive test cases, and types of evaluations used in papers. We also report the demographics and bibliometrics trends in this domain, including top-cited papers, active countries and researchers, and top venues in this research area. Conclusion We discuss the emerging trends in web application testing, and discuss the implications for researchers and practitioners in this area. The results of our systematic mapping can help researchers to obtain an overview of existing web application testing approaches and indentify areas in the field that require more attention from the research community.

[1]  William C. Chu,et al.  Constructing and Object-Oriented Architecture for Web Application Testing , 2002, J. Inf. Sci. Eng..

[2]  Paolo Tonella,et al.  Construction of the system dependence graph for Web application slicing , 2002, Proceedings. Second IEEE International Workshop on Source Code Analysis and Manipulation.

[3]  Pearl Brereton,et al.  Using Mapping Studies in Software Engineering , 2008, PPIG.

[4]  Lori L. Pollock,et al.  A scalable approach to user-session based testing of Web applications through concept analysis , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[5]  Paolo Tonella,et al.  Using search-based algorithms for Ajax event sequence generation during testing , 2010, Empirical Software Engineering.

[6]  Frank Maurer,et al.  Evaluating Current Testing Processes of Web-Portal Applications , 2005, ICWE.

[7]  Liam Peyton,et al.  Framework testing of web applications using TTCN-3 , 2008, International Journal on Software Tools for Technology Transfer.

[8]  Frank Tip,et al.  A framework for automated testing of javascript web applications , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[9]  Daniel Amyot,et al.  UCM-Driven Testing of Web Applications , 2005, SDL Forum.

[10]  Giuseppe A. Di Lucca,et al.  Testing Web-based applications: The state of the art and future trends , 2006, Inf. Softw. Technol..

[11]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[12]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[13]  Jeff Tian,et al.  Measuring and Modeling Usage and Reliability for Statistical Web Testing , 2001, IEEE Trans. Software Eng..

[14]  Westley Weimer,et al.  Automating regression testing using web-based application similarities , 2010, International Journal on Software Tools for Technology Transfer.

[15]  Emily Hill,et al.  Applying Concept Analysis to User-Session-Based Testing of Web Applications , 2007, IEEE Transactions on Software Engineering.

[16]  Yasuhiko Minamide,et al.  Static approximation of dynamically generated Web pages , 2005, WWW '05.

[17]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[18]  James R. Cordy,et al.  Modelling methods for web application verification and testing: state of the art , 2009, Softw. Test. Verification Reliab..

[19]  Ernesto Cid Brasil de Matos,et al.  From formal requirements to automated web testing and prototyping , 2010, Innovations in Systems and Software Engineering.

[20]  David Chenho Kung,et al.  An agent-based data-flow testing approach for Web applications , 2006, Inf. Softw. Technol..

[21]  Jianhua Hao,et al.  Usage-based statistical testing of web applications , 2006, ICWE '06.

[22]  Michael Benedikt,et al.  VeriWeb: Automatically Testing Dynamic Web Sites , 2002 .

[23]  James R. Cordy,et al.  Modelling methods for web application verification and testing: state of the art , 2009 .

[24]  Lori L. Pollock,et al.  Composing a framework to automate testing of operational Web-based software , 2004, 20th IEEE International Conference on Software Maintenance, 2004. Proceedings..

[25]  Per Runeson,et al.  Testing Software Product Lines , 2011, IEEE Software.

[26]  Mei-Hwa Chen,et al.  Clustering and Tailoring User Session Data for Testing Web Applications , 2009, 2009 International Conference on Software Testing Verification and Validation.

[27]  Alessandro Orso,et al.  Precise interface identification to improve testing and analysis of web applications , 2009, ISSTA.

[28]  A. Jefferson Offutt,et al.  Testing Web applications by modeling with FSMs , 2005, Software & Systems Modeling.

[29]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[30]  Paolo Tonella,et al.  Testing Processes of Web Applications , 2002, Ann. Softw. Eng..

[31]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[32]  Hiroshi Inamura,et al.  Dynamic test input generation for web applications , 2008, ISSTA '08.

[33]  Frank Elberzhager,et al.  A systematic mapping study on the combination of static and dynamic quality assurance techniques , 2012, Inf. Softw. Technol..

[34]  Mark Harman,et al.  Automated Session Data Repair for Web Application Regression Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[35]  Ali Mesbah,et al.  Automated cross-browser compatibility testing , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[36]  Francesco M. Donini,et al.  Design Verification of Web Applications Using Symbolic Model Checking , 2005, ICWE.

[37]  Behzad Bordbar,et al.  MDA and Analysis of Web Applications , 2005, TEAA.

[38]  Richard N. Taylor,et al.  Automatic and versatile publications ranking for research institutions and scholars , 2007, CACM.

[39]  Arie van Deursen,et al.  A Systematic Survey of Program Comprehension through Dynamic Analysis , 2008, IEEE Transactions on Software Engineering.

[40]  Mark Harman,et al.  Improving Web Application Testing using testability measures , 2009, 2009 11th IEEE International Symposium on Web Systems Evolution.

[41]  Paolo Tonella,et al.  A 2-layer model for the white-box testing of Web applications , 2004, Proceedings. Sixth IEEE International Workshop on Web Site Evolution.

[42]  James R. Cordy,et al.  Automating Coverage Metrics for Dynamic Web Applications , 2010, 2010 14th European Conference on Software Maintenance and Reengineering.

[43]  Gregg Rothermel,et al.  Leveraging user-session data to support Web application testing , 2005, IEEE Transactions on Software Engineering.

[44]  Frank Tip,et al.  Finding bugs in dynamic web applications , 2008, ISSTA '08.

[45]  Pearl Brereton,et al.  Using mapping studies as the basis for further research - A participant-observer case study , 2011, Inf. Softw. Technol..

[46]  Xiangyu Zhang,et al.  Statically locating web application bugs caused by asynchronous calls , 2011, WWW.

[47]  Lori L. Pollock,et al.  Automated Oracle Comparators for TestingWeb Applications , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[48]  Robert Feldt,et al.  Alignment of Requirements Specification and Testing: A Systematic Mapping Study , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[49]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[50]  James A. Whittaker,et al.  Exploratory software testing , 2010 .

[51]  James Miller,et al.  Cookies: A deployment study and the testing implications , 2009, TWEB.

[52]  Karthik Pattabiraman,et al.  DoDOM: Leveraging DOM Invariants for Web 2.0 Application Robustness Testing , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[53]  Atif M. Memon,et al.  Advances in Automated Model-Based System Testing of Software Applications with a GUI Front-End , 2010, Adv. Comput..

[54]  Porfirio Tramontana,et al.  Techniques and tools for Rich Internet Applications testing , 2010, 2010 12th IEEE International Symposium on Web Systems Evolution (WSE).

[55]  Frank Tip,et al.  Practical fault localization for dynamic web applications , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[56]  Westley Weimer,et al.  Modeling consumer-perceived web application fault severities for testing , 2010, ISSTA '10.

[57]  Chien-Hung Liu.  Data flow analysis and testing of JSP-based Web applications , 2006, Inf. Softw. Technol..

[58]  Robert L. Glass,et al.  An assessment of systems and software engineering scholars and institutions (1993-1997) , 1997, Journal of Systems and Software.

[59]  Arie van Deursen,et al.  Research Issues in the Automated Testing of Ajax Applications , 2010, SOFSEM.

[60]  A. Jefferson Offutt,et al.  Modeling presentation layers of web applications for testing , 2009, Software & Systems Modeling.

[61]  Alessandro Orso,et al.  Improving test case generation for web applications using automated interface discovery , 2007, ESEC-FSE '07.

[62]  Javier Tuya,et al.  Testing in Service Oriented Architectures with dynamic binding: A mapping study , 2011, Inf. Softw. Technol..

[63]  Frank Maurer,et al.  WIT: A Framework for In-container Testing of Web-Portal Applications , 2005, ICWE.

[64]  Tao Xie,et al.  Perturbation-based user-input-validation testing of web applications , 2010, J. Syst. Softw..

[65]  Akif Günes Koru,et al.  Prioritizing User-Session-Based Test Cases for Web Applications Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[66]  Emily Hill,et al.  Integrating customized test requirements with traditional requirements in web application testing , 2006, TAV-WEB '06.

[67]  Tevfik Bultan,et al.  Eliminating navigation errors in web applications via model checking and runtime enforcement of navigation state machines , 2010, ASE '10.

[68]  S. Sprenkle,et al.  WebVizOr: A Visualization Tool for Applying Automated Oracles and Analyzing Test Results of Web Applications , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[69]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[70]  Paolo Tonella,et al.  Web Application Slicing in Presence of Dynamic Code Generation , 2005, Automated Software Engineering.

[71]  Steve Hanna,et al.  A Symbolic Execution Framework for JavaScript , 2010, 2010 IEEE Symposium on Security and Privacy.

[72]  Arie van Deursen,et al.  Regression Testing Ajax Applications: Coping with Dynamism , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[73]  M. Harman,et al.  Testing Web Services : A Survey , 2011 .

[74]  Paolo Tonella,et al.  A case study-based comparison of web testing techniques applied to AJAX web applications , 2008, International Journal on Software Tools for Technology Transfer.

[75]  Per Runeson,et al.  A systematic review on regression test selection techniques , 2010, Inf. Softw. Technol..

[76]  Porfirio Tramontana,et al.  Rich Internet Application Testing Using Execution Trace Data , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[77]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[78]  Magnus Madsen,et al.  Modeling the HTML DOM and browser API in static analysis of JavaScript web applications , 2011, ESEC/FSE '11.

[79]  Carlo Bellettini,et al.  TestUml: user-metrics driven web applications testing , 2005, SAC '05.

[80]  Giuseppe A. Di Lucca,et al.  Testing Web applications , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[81]  Robert V. Binder,et al.  Testing object‐oriented software: a survey , 1996 .

[82]  Paolo Tonella,et al.  Analysis and testing of Web applications , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[83]  Paolo Tonella,et al.  State-Based Testing of Ajax Web Applications , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[84]  Curtis E. Dyreson,et al.  Scalability issues with using FSMWeb to test web applications , 2010, Inf. Softw. Technol..

[85]  Alessandro Orso,et al.  Automated identification of parameter mismatches in web applications , 2008, SIGSOFT '08/FSE-16.

[86]  Mark Harman,et al.  Crawlability metrics for automated web testing , 2011, International Journal on Software Tools for Technology Transfer.

[87]  Jesse James Garrett.  Ajax: A New Approach to Web Applications , 2007 .

[88]  Shriram Krishnamurthi,et al.  Verifying interactive Web programs , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[89]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[90]  A. Jefferson Offutt,et al.  Bypass testing of Web applications , 2004, 15th International Symposium on Software Reliability Engineering.

[91]  C. Titus Brown,et al.  An introduction to testing web applications with twill and selenium , 2007 .

[92]  Giuseppe A. Di Lucca,et al.  A Technique for Reducing User Session Data Sets in Web Application Testing , 2006, 2006 Eighth IEEE International Symposium on Web Site Evolution (WSE'06).

[93]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[94]  Abdul Azim Abdul Ghani,et al.  Unit Testing Approaches for BPEL: A Systematic Review , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[95]  Robert V. Binder,et al.  Testing Object-Oriented Software , 1997, TOOLS.

[96]  Theodore D. Hellmann,et al.  Agile Interaction Design and Test-Driven Development of User Interfaces - A Literature Review , 2010, Agile Software Development.

[97]  Tiziana Margaria,et al.  Hybrid test of web applications with webtest , 2008, TAV-WEB '08.

[98]  Silvio Romero de Lemos Meira,et al.  A Mapping Study on Software Product Lines Testing Tools , 2012, SEKE.

[99]  Hui Liu,et al.  Testing input validation in Web applications through automated model recovery , 2008, J. Syst. Softw..

[100]  A. Jefferson Offutt,et al.  Applying Mutation Testing to Web Applications , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[101]  Marinus J. Plasmeijer,et al.  Model-Based Testing of Thin-Client Web Applications and Navigation Input , 2008, PADL.

[102]  Paolo Tonella,et al.  Empirical Validation of a Web Fault Taxonomy and its usage for Fault Seeding , 2007, 2007 9th IEEE International Workshop on Web Site Evolution.

[103]  Alessandro Marchetto.  Talking about a Mutation-Based Reverse Engineering for Web Testing: A Preliminary Experiment , 2008, 2008 Sixth International Conference on Software Engineering Research, Management and Applications.

[104]  Pearl Brereton,et al.  The value of mapping studies - A participant-observer case study , 2010, EASE.

[105]  Yevgeny Gerlits.  Testing AJAX functionality with UniTESK , 2010 .

[106]  Robert L. Glass,et al.  An assessment of systems and software engineering scholars and institutions (1998-2002) , 2003, J. Syst. Softw..

[107]  Lori L. Pollock,et al.  A Study of Usage-Based Navigation Models and Generated Abstract Test Cases for Web Applications , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[108]  Jonathan Jacky,et al.  Model-Based Testing of Web Applications Using NModel , 2009, TestCom/FATES.

[109]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[110]  Paolo Tonella,et al.  Statistical testing of Web applications , 2004, J. Softw. Maintenance Res. Pract..

[111]  Natalia Juristo Juzgado,et al.  Reviewing 25 Years of Testing Technique Experiments , 2004, Empirical Software Engineering.

[112]  Emily Hill,et al.  An empirical comparison of test suite reduction techniques for user-session-based testing of Web applications , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[113]  Gerardo Canfora,et al.  Service-Oriented Architectures Testing: A Survey , 2009, ISSSE.

[114]  Li Ma,et al.  Web Testing for Reliability Improvement , 2006, Adv. Comput..

[115]  A Bibliometric Assessment of Canadian Software Engineering Scholars and Institutions (1996-2006) , 2010, Comput. Inf. Sci..

[116]  Wasif Afzal,et al.  A Systematic Mapping Study on Non-Functional Search-based Software Testing , 2008, SEKE.

[117]  Paolo Tonella,et al.  Web application slicing , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[118]  John D. McGregor,et al.  A systematic mapping study of software product lines testing , 2011, Inf. Softw. Technol..

[119]  Aysu Betin Can,et al.  Detecting Execution and HTML Errors in ASP.NET Web Applications , 2011, ICSOFT.

[120]  Beatriz Marín,et al.  Towards testing future Web applications , 2011, 2011 FIFTH INTERNATIONAL CONFERENCE ON RESEARCH CHALLENGES IN INFORMATION SCIENCE.

[121]  McMinnPhil.  Search-based software test data generation: a survey , 2004 .

[122]  Robert L. Glass,et al.  An assessment of systems and software engineering scholars and institutions (1999-2003) , 2005, J. Syst. Softw..

[123]  Emily Hill,et al.  Web Application Testing with Customized Test Requirements - An Experimental Comparison Study , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[124]  Curtis E. Dyreson,et al.  Building test cases and oracles to automate the testing of web database applications , 2009, Inf. Softw. Technol..

[125]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[126]  Arie van Deursen,et al.  Invariant-based automatic testing of AJAX user interfaces , 2009, 2009 IEEE 31st International Conference on Software Engineering.