Trade-off between automated and manual software testing

The study explores the current state of test automation in software testing organizations by focusing on the views and observations of managers, testers and developers in each organization. The case study was conducted in selected organizational units that develop and test technical software for industrial automation or telecommunication domains. The data was collected with 41 theme-based interviews in each unit. The interview data was analyzed qualitatively by using the grounded theory research method. It was found that although test automation was viewed as beneficial, it was not utilized widely in the companies. The main benefits of test automation were quality improvement, the possibility to execute more tests in less time and fluent reuse of testware. The major disadvantages were the costs associated with developing test automation especially in dynamic customized environments. Such issues as properties of tested products, attitudes of employees, resource limitations, and customers influenced the level of test automation in the case organizations.

[1]  Rudolf Ramler,et al.  Economic perspectives in test automation: balancing automated and manual testing with opportunity cost , 2006, AST '06.

[2]  Guy Paré,et al.  Using case study research to build theories of IT implementation , 1997 .

[3]  Ossi Taipale,et al.  Factors affecting software testing time schedule , 2006, Australian Software Engineering Conference (ASWEC'06).

[4]  N. Denzin.  The research act: A theoretical introduction to sociological methods , 1977 .

[5]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[6]  Edward Kit,et al.  Software testing in the real world - improving the process , 1995 .

[7]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[8]  Katja Karhu,et al.  Empirical Observations on Software Testing Automation , 2009, 2009 International Conference on Software Testing Verification and Validation.

[9]  Pamela Jordan.  Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[10]  Giancarlo Succi,et al.  Effort Prediction in Iterative Software Development Processes -- Incremental Versus Global Prediction Models , 2007, ESEM 2007.

[11]  L. C. Briand.  A Critical Analysis of Empirical Research in Software Testing , 2007, ESEM 2007.

[12]  Tsong Yueh Chen,et al.  A preliminary survey on software testing practices in Australia , 2004, 2004 Australian Software Engineering Conference. Proceedings..

[13]  Tathagat Varma,et al.  Automated software testing: introduction, management and performance , 2000, SOEN.

[14]  Ossi Taipale,et al.  Finding and Ranking Research Directions for Software Testing , 2005, EuroSPI.

[15]  S. Berner,et al.  Observations and lessons learned from automated testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[16]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[17]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[18]  Katja Karhu,et al.  Outsourcing and Knowledge Management in Software Testing , 2007, EASE.

[19]  Pedro de Alcântara dos Santos Neto,et al.  Requirements for information systems model-based testing , 2007, SAC '07.

[20]  Katja Karhu,et al.  Observing Software Testing Practice from the Viewpoint of Organizations and Knowledge Management , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[21]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[22]  C. Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[23]  Sandeep Purao,et al.  Software architectures: Blueprint, Literature, Language or Decision? , 2008, Eur. J. Inf. Syst..

[24]  Cem Kaner.  Improving the maintainability of automated test suites , 1997 .

[25]  J. Bach.  Test Automation Snake Oil , 1999 .

[26]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[27]  Craig R. Carter,et al.  Purchasing's Role in Environmental Management: Cross-Functional Development of Grounded Theory , 2001 .

[28]  Barry Boehm,et al.  An Empirical Study on MBASE and LeanMBASE , 2007, ESEM 2007.

[29]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[30]  C. Persson,et al.  Establishment of automated regression testing at ABB: industrial experience report on 'avoiding the pitfalls' , 2004 .

[31]  Béatrix Barafort,et al.  Benefits Resulting from the Combined Use of ISO/IEC 15504 with the Information Technology Infrastructure Library (ITIL) , 2002, PROFES.

[32]  Steve Townend,et al.  Review of: Muhr, T. 1997. ATLAS.ti 5: The Knowledge Workbench. Berlin: Scientific Software Development , 2003 .

[33]  Gregory Tassey,et al.  Prepared for what , 2007 .

[34]  Mark Blackburn,et al.  Why Model-Based Test Automation is Different and What You Should Know to Get Started , 2004 .

[35]  K. Seers.  Qualitative data analysis , 2011, Evidence Based Nursing.

[36]  Richard Torkar,et al.  A survey on testing and reuse , 2003, Proceedings 2003 Symposium on Security and Privacy.

[37]  Emily E. Marsh.  ATLAS.ti, The Knowledge Workbench , 2001 .

[38]  C. Jones,et al.  Economics of software reuse , 1994, Computer.

[39]  Johanna Rothman,et al.  Architectures of Test Automation , 2000 .

[40]  Mary Shaw,et al.  Writing good software engineering research papers , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[41]  Mary Jean Harrold,et al.  Testing: a roadmap , 2000, ICSE '00.

[42]  Christer Persson,et al.  Establishment of automated regression testing at ABB: industrial experience report on 'avoiding the pitfalls' , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[43]  Computer Staff,et al.  International Conference on Software Engineering , 1991 .