Choosing the Right Test Automation Tool: a Grey Literature Review of Practitioner Sources

Background: Choosing the right software test automation tool is not trivial, and recent industrial surveys indicate lack of right tools as the main obstacle to test automation. Aim: In this paper, we study how practitioners tackle the problem of choosing the right test automation tool. Method: We synthesize the "voice" of the practitioners with a grey literature review originating from 53 different companies. The industry experts behind the sources had roles such as "Software Test Automation Architect", and "Principal Software Engineer". Results: Common consensus about the important criteria exists but those are not applied systematically. We summarize the scattered steps from individual sources by presenting a comprehensive process for tool evaluation with 12 steps and a total of 14 different criteria for choosing the right tool. Conclusions: The practitioners tend to have general interest in and be influenced by related grey literature as about 78% of our sources had at least 20 backlinks (a reference comparable to a citation) while the variation was between 3 and 759 backlinks. There is a plethora of different software testing tools available, yet the practitioners seem to prefer and adopt the widely known and used tools. The study helps to identify the potential pitfalls of existing processes and opportunities for comprehensive tool evaluation.

[1]  Vahid Garousi,et al.  Citation and Topic Analysis of the ESEM Papers , 2015, 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[2]  Alexander Schill,et al.  Literature Review of Empirical Research Studies within the Domain of Acceptance Testing , 2016, 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[3]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[4]  Pearl Brereton,et al.  Evaluation and assessment in software engineering , 1997, J. Syst. Softw..

[5]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[6]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[7]  Mika Mäntylä,et al.  Using Surveys and Web-Scraping to Select Tools for Software Testing Consultancy , 2016, PROFES.

[8]  Nazim H. Madhavji,et al.  The Impact of Tools on Software Productivity , 1996, IEEE Softw..

[9]  K. A. Hayden,et al.  State-of-the-evidence reviews: advantages and challenges of including grey literature. , 2006, Worldviews on evidence-based nursing.

[10]  John Battelle.  The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture , 2005 .

[11]  Robert Feldt,et al.  Automated System Testing Using Visual GUI Testing Tools: A Comparative Study in Industry , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[12]  Kai Petersen,et al.  Benefits and limitations of automated software testing: Systematic literature review and practitioner survey , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[13]  憲一 中嶋.  If all you have is a hammer, everything looks like a nail , 2017 .

[14]  R. Cialdini.  Influence: Science and Practice , 1984 .

[15]  Suzie Allard,et al.  Design engineers and technical professionals at work: Observing information usage in the workplace , 2009, J. Assoc. Inf. Sci. Technol..

[16]  Robert M. Poston,et al.  Evaluating and selecting testing tools , 1992, IEEE Software.

[17]  Harpreet Kaur,et al.  Comparative Study of Automated Testing Tools: Selenium, Quick Test Professional and Testcomplete , 2013 .

[18]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[19]  Michiel van Genuchten,et al.  Why is Software Late? An Empirical Study of Reasons For Delay in Software Development , 1991, IEEE Trans. Software Eng..

[20]  Ángel Viña,et al.  Semi-Automatic Wrapper Generation for Commercial Web Sources , 2002, Engineering Information Systems in the Internet Context.

[21]  Ossi Taipale,et al.  Software Test Automation in Practice: Empirical Observations , 2010, Adv. Softw. Eng..

[22]  Elliot J. Chikofsky,et al.  Assessing the State of Tools Assessment , 1992 .

[23]  Neil A. M. Maiden,et al.  The Reflective Software Engineer: Reflective Practice , 2014, IEEE Softw..

[24]  Abraham H. Maslow,et al.  The psychology of science: a reconnaissance , 1966 .

[25]  Reza Javidan,et al.  Presenting a Method to Test the Designed Software with a Service-Oriented Approach Using International Software Testing Qualifications Board (ISTQB) , 2016 .

[26]  Vahid Garousi,et al.  When and what to automate in software testing? A multi-vocal literature review , 2016, Inf. Softw. Technol..

[27]  Vahid Garousi,et al.  A systematic literature review of literature reviews in software testing , 2016, Inf. Softw. Technol..

[28]  Bernard J. Jansen.  Review of The search: How google and its rivals rewrote the rules of business and transformed our culture John Battelle, Penguin Group , 2006 .

[29]  A. Huberman,et al.  Qualitative Data Analysis: A Methods Sourcebook , 1994 .

[30]  Shuang Wang,et al.  Comparison of Unit-Level Automated Test Generation Tools , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[31]  Raya Fidel,et al.  The many faces of accessibility: engineers' perception of information sources , 2004, Inf. Process. Manag..

[32]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[33]  Morten Hertzum,et al.  The information-seeking practices of engineers: searching for documents as well as for people , 2000, Inf. Process. Manag..

[34]  Vahid Garousi,et al.  The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature , 2016, EASE.