Software Test Automation in Practice: Empirical Observations

The objective of this industry study is to shed light on the current situation and improvement needs in software test automation. To this end, 55 industry specialists from 31 organizational units were interviewed. In parallel with the survey, a qualitative study was conducted in 12 selected software development organizations. The results indicated that the software testing processes usually follow systematic methods to a large degree, and have only little immediate or critical requirements for resources. Based on the results, the testing processes have approximately three fourths of the resources they need, and have access to a limited, but usually sufficient, group of testing tools. As for the test automation, the situation is not as straightforward: based on our study, the applicability of test automation is still limited and its adaptation to testing contains practical difficulties in usability. In this study, we analyze and discuss these limitations and difficulties.

[1]  K. Beck,et al.  Extreme Programming Explained , 2002 .

[2]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[3]  J. Knottnerus,et al.  Real world research. , 2010, Journal of clinical epidemiology.

[4]  Reidar Conradi,et al.  An empirical study on software development with open source components in the chinese software industry , 2008, Softw. Process. Improv. Pract..

[5]  Lorraine Carter,et al.  How to Conduct Surveys: A Step-by-Step Guide , 2010 .

[6]  Weider D. Yu,et al.  A Workflow-Based Test Automation Framework for Web Based Systems , 2007, 2007 12th IEEE Symposium on Computers and Communications.

[7]  Zhou Bo,et al.  A test automation solution on GUI functional test , 2008, 2008 6th IEEE International Conference on Industrial Informatics.

[8]  Pamela Jordan.  Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[9]  Yehuda Baruch,et al.  Response Rate in Academic Studies — A Comparative Analysis , 1999 .

[10]  Tsong Yueh Chen,et al.  A preliminary survey on software testing practices in Australia , 2004, 2004 Australian Software Engineering Conference. Proceedings..

[11]  M. Narasimham,et al.  The economics of knowledge , 2003, Int. J. Inf. Technol. Manag..

[12]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[13]  Norman Crolee Dalkey,et al.  An experimental study of group opinion , 1969 .

[14]  Philippe Kruchten,et al.  The Rational Unified Process: An Introduction, Second Edition , 2000 .

[15]  Gregory Tassey,et al.  Prepared for what , 2007 .

[16]  James Bret Michael,et al.  Test automation and safety assessment in rapid systems prototyping , 2005, 16th IEEE International Workshop on Rapid System Prototyping (RSP'05).

[17]  Rafiq Dossani,et al.  The Internet's role in offshored services: A case study of India , 2007, TOIT.

[18]  Christer Persson,et al.  Establishment of automated regression testing at ABB: industrial experience report on 'avoiding the pitfalls' , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[19]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[20]  Tathagat Varma,et al.  Automated software testing: introduction, management and performance , 2000, SOEN.

[21]  Mark Blackburn,et al.  Why Model-Based Test Automation is Different and What You Should Know to Get Started , 2004 .

[22]  DossaniRafiq,et al.  The Internet's role in offshored services , 2007 .

[23]  Béatrix Barafort,et al.  Benefits Resulting from the Combined Use of ISO/IEC 15504 with the Information Technology Infrastructure Library (ITIL) , 2002, PROFES.

[24]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[25]  K. Seers.  Qualitative data analysis , 2011, Evidence Based Nursing.

[26]  Richard Torkar,et al.  A survey on testing and reuse , 2003, Proceedings 2003 Symposium on Security and Privacy.

[27]  Jason F. Cohen,et al.  Agile systems development and stakeholder satisfaction: a South African empirical study , 2008, SAICSIT '08.

[28]  Reidar Conradi,et al.  An empirical study of variations in COTS-based software development processes in the Norwegian IT industry , 2004, 10th International Symposium on Software Metrics, 2004. Proceedings..

[29]  C. J. Harwood.  Test Process Improvement: A Practical Step‐by‐Step Guide to Structured Testing , 2000 .

[30]  Dieter Kreuer,et al.  Applying test automation to type acceptance testing of telecom networks: a case study with customer participation , 1999, 14th IEEE International Conference on Automated Software Engineering.

[31]  Tore Dybå,et al.  An Instrument for Measuring the Key Factors of Success in Software Process Improvement , 2000, Empirical Software Engineering.

[32]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[33]  J. Bach.  Test Automation Snake Oil , 1999 .

[34]  Cem Kaner.  Improving the maintainability of automated test suites , 1997 .

[35]  Daniel J. Mosley,et al.  Just Enough Software Test Automation , 2002 .

[36]  Leon J. Osterweil,et al.  Software Processes Are Software Too, Revisited: An Invited Talk on the Most Influential Paper of ICSE 9 , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[37]  C. Brodsky.  The Discovery of Grounded Theory: Strategies for Qualitative Research , 1968 .

[38]  Rajesh Subramanyan,et al.  Automation of GUI testing using a model-driven approach , 2006, AST '06.

[39]  N. Denzin.  The research act: A theoretical introduction to sociological methods , 1977 .

[40]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[41]  Guy Paré,et al.  Using case study research to build theories of IT implementation , 1997 .

[42]  Edward Kit,et al.  Software testing in the real world - improving the process , 1995 .

[43]  Andraž Cej,et al.  Agile software development with Scrum , 2010 .

[44]  J. Crisp,et al.  The Delphi method? , 1997, Nursing research.

[45]  A. Strauss.  Basics Of Qualitative Research , 1992 .

[46]  Katja Karhu,et al.  Empirical Observations on Software Testing Automation , 2009, 2009 International Conference on Software Testing Verification and Validation.

[47]  S. Berner,et al.  Observations and lessons learned from automated testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[48]  Kuan Yew Wong.  An exploratory study on knowledge management adoption in the Malaysian industry , 2008, Int. J. Bus. Inf. Syst..

[49]  P. Krutchen,et al.  The Rational Unified Process: An Introduction , 2000 .

[50]  James A. Whittaker,et al.  What is software testing? And why is it so hard? , 2000 .

[51]  Ossi Taipale,et al.  Factors affecting software testing time schedule , 2006, Australian Software Engineering Conference (ASWEC'06).

[52]  Ossi Taipale,et al.  Improving software testing by observing practice , 2006, ISESE '06.

[53]  Pedro de Alcântara dos Santos Neto,et al.  Requirements for information systems model-based testing , 2007, SAC '07.