A critical examination of recent industrial surveys on agile method usage

Abstract Context Practitioners and researchers often claim that agile methods have moved into the mainstream for the last few years. To support this claim they refer to recent industrial surveys which tend to report high rates of agile method usage. However many of these industrial surveys are conducted by agile consultants, tool vendors, professional societies and independent technology and market research organizations. This raises some important concerns about the possible conflict of interest and the overall trustworthiness of these studies. Objective In response to the above concerns, a secondary study was carried out. Its objective was to examine industrial surveys published in 2011 and 2012, determine the extent to which we could trust their reported high rates of agile method usage and provide recommendations on how quality of research could be improved in the future. Method Following a rigorous search procedure, nine industrial surveys on agile method usage published in 2011 and 2012 were extracted from both academia and industry. Their thoroughness in reporting and trustworthiness were evaluated using a newly proposed assessment framework based on Guba's four attributes of trustworthiness (truth value, applicability, consistency and neutrality) and a number of methods for assessing survey research in related fields as information, communication and management studies. Results The careful examination of the reviewed surveys shows that most of the studies have insufficient thoroughness in reporting and (subsequently) low trustworthiness. Only one (out of nine) study is considered as a scientific contribution in determining the current 2011/2012 rate of agile method usage. Conclusions The obtained results support our initial considerations about the trustworthiness of recent industrial surveys on agile method usage and suggest a number of recommendations for increasing the quality and value of future survey research in this regard.

[1]  Tore Dybå,et al.  A Preliminary Roadmap for Empirical Research on Agile Software Development , 2008, Agile 2008 Conference.

[2]  James Sheffield,et al.  Factors associated with the software development agility of successful projects , 2013 .

[3]  Jay Brand,et al.  File Drawer Problem , 2022, The SAGE Encyclopedia of Research Design.

[4]  James D. Arthur,et al.  A disciplined approach to adopting agile practices: the agile adoption framework , 2007, Innovations in Systems and Software Engineering.

[5]  Nomi Baruah,et al.  A Survey of the Use of Agile Methodologies in Different Indian Small and Medium Scale Enterprises (SMEs) , 2012 .

[6]  Richard W. Selby.  Empirical Software Engineering Research Roadmap Discussion and Summary , 2006, Empirical Software Engineering Issues.

[7]  Stefan Biffl,et al.  Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering , 2003, ESERNET.

[8]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[9]  Kai Petersen,et al.  Agile Software Development Practice Adoption Survey , 2012, XP.

[10]  Andrew K. Shenton.  Strategies for ensuring trustworthiness in qualitative research projects , 2004, Educ. Inf..

[11]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[12]  H. Fawcett.  Manual of Political Economy , 1995 .

[13]  Charles P. Auger,et al.  Information sources in grey literature , 1989 .

[14]  Mikko Rönkkö,et al.  Examining the Effects of Agile Methods and Process Maturity on Software Product Development Performance , 2011, ICSOB.

[15]  Elaine J. Weyuker,et al.  Empirical Software Engineering Research - The Good, The Bad, The Ugly , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[16]  Roel Wieringa.  Towards a unified checklist for empirical research in software engineering: first proposal , 2012, EASE.

[17]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[18]  Alain Pinsonneault,et al.  Survey Research Methodology in Management Information Systems: An Assessment , 1993, J. Manag. Inf. Syst..

[19]  E. Guba.  Criteria for assessing the trustworthiness of naturalistic inquiries , 1981 .

[20]  Alberto Sillitti.  Agile Processes in Software Engineering and Extreme Programming, 11th International Conference, XP 2010, Trondheim, Norway, June 1-4, 2010. Proceedings , 2010, XP.

[21]  F. Song,et al.  Dissemination and publication of research findings: an updated review of related biases. , 2010, Health technology assessment.

[22]  Kuda Nageswara Rao,et al.  A Study of the Agile Software Development Methods, Applicability and Implications in Industry , 2011 .

[23]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[24]  Daniel E. Turk,et al.  Drivers of agile software development use: Dialectic interplay between benefits and hindrances , 2012, Inf. Softw. Technol..

[25]  Daniela Cruzes,et al.  Synthesizing evidence in software engineering research , 2010, ESEM '10.

[26]  Muhammad Asim Ali,et al.  International Journal of Information and Communication Technology Research Survey on the State of Agile Practices Implementation in Pakistan , 2022 .

[27]  Pekka Abrahamsson,et al.  Agile methods rapidly replacing traditional methods at Nokia: A survey of opinions on agile transformation , 2011, Inf. Softw. Technol..

[28]  Mira Kajko-Mattsson,et al.  Survey of Agile Tool Usage and Needs , 2011, 2011 AGILE Conference.

[29]  David Lorge Parnas.  Empirical Research in Software Engineering: A Critical View , 2009 .

[30]  Jouni Markkula,et al.  Survey on agile and lean usage in finnish software industry , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[31]  Muhammad Ali Babar,et al.  Conducting empirical studies in industry: Balancing rigor and relevance , 2013, 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI).

[32]  V. Grover,et al.  An assessment of survey research in POM: from constructs to theory , 1998 .

[33]  Laurie A. Williams,et al.  What agile teams think of agile principles , 2012, Commun. ACM.

[34]  Fergal McCaffery,et al.  Barriers to Adopting Agile Practices When Developing Medical Device Software , 2012, SPICE.

[35]  Brian Henderson-Sellers,et al.  A framework to support the evaluation, adoption and improvement of agile methods in practice , 2008, J. Syst. Softw..

[36]  G. B. Wills,et al.  Factor Analysis: Investigating Important Aspects for Agile Adoption in Malaysia , 2012, 2012 Agile India.

[37]  Torgeir Dingsøyr,et al.  A decade of agile methodologies: Towards explaining agile software development , 2012, J. Syst. Softw..