Qualitative research on software development: a longitudinal case study methodology

This paper reports the use of a qualitative methodology for conducting longitudinal case study research on software development. We provide a detailed description and explanation of appropriate methods of qualitative data collection and analysis that can be utilized by other researchers in the software engineering field. Our aim is to illustrate the utility of longitudinal case study research, as a complement to existing methodologies for studying software development, so as to enable the community to develop a fuller and richer understanding of this complex, multi-dimensional phenomenon. We discuss the insights gained and lessons learned from applying a longitudinal qualitative approach to an empirical case study of a software development project in a large multi-national organization. We evaluate the methodology used to emphasize its strengths and to address the criticisms traditionally made of qualitative research.

[1]  Barry Boehm,et al.  A view of 20th and 21st century software engineering , 2006, ICSE.

[2]  B. Pentland.  Building Process Theory with Narrative: from Description to Explanation , 1999 .

[3]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[4]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[5]  Graeme G. Shanks,et al.  Successfully completing case study research: combining rigour, relevance and pragmatism , 1998, Inf. Syst. J..

[6]  Michael I. Barrett,et al.  Negotiating ICT development and use: The case of a telemedicine system in the healthcare region of Crete , 2006, Inf. Organ..

[7]  Michael D. Myers,et al.  The qualitative interview in IS research: Examining the craft , 2007, Inf. Organ..

[8]  Magne Jørgensen,et al.  Can you Trust a Single Data Source Exploratory Software Engineering Case Study? , 2002, Empirical Software Engineering.

[9]  D MyersMichael,et al.  A set of principles for conducting and evaluating interpretive field studies in information systems , 1999 .

[10]  Laurie Carina McLeod,et al.  Understanding IS development and acquisition: a process approach , 2008 .

[11]  Janice Singer,et al.  Ethical Issues in Empirical Studies of Software Engineering , 2002, IEEE Trans. Software Eng..

[12]  Susan Leigh Star,et al.  The Structure of Ill-Structured Solutions: Boundary Objects and Heterogeneous Distributed Problem Solving , 1989, Distributed Artificial Intelligence.

[13]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[14]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[15]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[16]  Angèle L. M. Cavaye,et al.  Case study research: a multi‐faceted research approach for IS , 1996, Inf. Syst. J..

[17]  Joan E. Sieber,et al.  Protecting Research Subjects, Employees and Researchers: Implications for Software Engineering , 2001, Empirical Software Engineering.

[18]  A. Whittle.  Preaching and practising ‘flexibility’: Implications for theories of subjectivity at work , 2005 .

[19]  Venkataraman Ramesh,et al.  RESRES: The story behind the paper "Research in software engineering: An analysis of the literature" , 2009, Inf. Softw. Technol..

[20]  Joe Nandhakumar,et al.  Too close for comfort? Distance and engagement in interpretive information systems research , 1997, Inf. Syst. J..

[21]  C. Ciborra,et al.  Formative contexts and information technology: Understanding the dynamics of innovation in organizations , 1994 .

[22]  Venkataraman Ramesh,et al.  Research in software engineering: an analysis of the literature , 2002, Inf. Softw. Technol..

[23]  A. Kellerman,et al.  The Constitution of Society : Outline of the Theory of Structuration , 2015 .

[24]  B. Latour.  Science in action : how to follow scientists and engineers through society , 1989 .

[25]  A. Pettigrew.  WHAT IS A PROCESSUAL ANALYSIS. , 1997 .

[26]  M. Lynne Markus,et al.  Special Issue on Intensive Research in Information Systems: Using Qualitative, Interpretive, and Case Methods to Study Information Technology-Third Installment; Foreword , 2000, MIS Q..

[27]  Capers Jones,et al.  Why software fails , 1996 .

[28]  M. Lynne Markus,et al.  Special issue on intensive research in information systems: using qualitative, interpretive, and case methods to study information technology—foreword , 1999 .

[29]  Janice Singer,et al.  Studying Software Engineers: Data Collection Techniques for Software Field Studies , 2005, Empirical Software Engineering.

[30]  Geoff Walsham,et al.  Interpretive case studies in IS research: nature and method , 1995 .

[31]  Izak Benbasat,et al.  The Case Research Strategy in Studies of Information Systems , 1987, MIS Q..

[32]  Mark Keil,et al.  Software project risks and their effect on outcomes , 2004, CACM.

[33]  Susan Elliott Sim,et al.  Case studies for software engineers , 2005, Proceedings. 26th International Conference on Software Engineering.

[34]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[35]  Judy L. Wynekoop,et al.  Studying system development methodologies: an examination of research methods , 1997, Inf. Syst. J..

[36]  Janice Singer,et al.  For the Special issue on Qualitative Software Engineering Research , 2007, Inf. Softw. Technol..

[37]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[38]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[39]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[40]  A. Langley.  Strategies for Theorizing from Process Data , 1999 .

[41]  Per Runeson,et al.  Tutorial: Case Studies in Software Engineering , 2009, PROFES.

[42]  Shoshana Zuboff,et al.  In the Age of the Smart Machine: The Future of Work and Power , 1989 .

[43]  Jacques Wainer,et al.  Empirical research in CSCW — a review of the ACM/CSCW conferences from 1998 to 2004 , 2007, Journal of the Brazilian Computer Society.

[44]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[45]  A. Langley,et al.  A PROCESS STUDY OF NEW TECHNOLOGY ADOPTION IN SMALLER MANUFACTURING FIRMS , 1994 .

[46]  Jacky Swan,et al.  There’s More to Information Systems Development than Structured Approaches: Information Requirements Analysis as a Socially Mediated Process , 2000, Requirements Engineering.

[47]  E. Guba,et al.  The Paradigm dialog , 1990 .

[48]  Dorothy Leonard-Barton,et al.  A Dual Methodology for Case Studies: Synergistic Use of a Longitudinal Single Site with Replicated Multiple Sites , 1990 .

[49]  Per Runeson,et al.  Checklists for Software Engineering Case Study Research , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[50]  Wiebe E. Bijker,et al.  Science in action : how to follow scientists and engineers through society , 1989 .

[51]  B. Flyvbjerg.  Five Misunderstandings About Case-Study Research , 2006, 1304.1186.

[52]  Martin Tolich,et al.  Starting fieldwork : an introduction to qualitative research in New Zealand , 1999 .

[53]  A. Pettigrew.  Longitudinal Field Research on Change: Theory and Practice , 1990 .

[54]  E. Guba.  The alternative paradigm dialog. , 1990 .

[55]  The Challenges of Complex IT Projects , 2004 .

[56]  Richard T. Vidgen,et al.  A framework for understanding how a unique and local IS development method emerges in practice , 2006, Eur. J. Inf. Syst..

[57]  Janice Singer,et al.  Beg, borrow, or steal: using multidisciplinary approaches in empirical software engineering research , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[58]  M. Markus,et al.  Information technology and organizational change: causal structure in theory and research , 1988 .

[59]  Bill Doolin,et al.  Alternative Views of Case Research in Information Systems , 1996, Australas. J. Inf. Syst..

[60]  Stefan Biffl,et al.  Directions and Methodologies for Empirical Software Engineering Research , 2004, Empirical Software Engineering.

[61]  Susan Gasson,et al.  A social action model of situated information systems design , 1999, DATB.

[62]  Geoff Walsham,et al.  Interpreting Information Systems in Organizations , 1993 .

[63]  C. Geertz.  Thick Description: Towards an Interpretive Theory of Culture , 1973 .

[64]  Geoff Walsham,et al.  GIS for District-Level Administration in India: Problems and Opportunities , 1999, MIS Q..

[65]  David E. Avison,et al.  The fiction of methodological development: a field study of information systems development , 1999, Inf. Technol. People.

[66]  Christopher Westrup,et al.  Information systems methodologies in use , 1993, J. Inf. Technol..

[67]  Geoff Walsham,et al.  Doing interpretive research , 2006, Eur. J. Inf. Syst..

[68]  Rory V. O'Connor,et al.  Using grounded theory to understand software process improvement: A study of Irish software product companies , 2007, Inf. Softw. Technol..

[69]  Stephen G. MacDonell,et al.  Factors that affect software systems development project outcomes: A survey of research , 2011, CSUR.

[70]  Khaled El Emam,et al.  A Replicated Survey of IT Software Project Failures , 2008, IEEE Software.

[71]  Alan H. Bond,et al.  Distributed Artificial Intelligence , 1988 .

[72]  James Miller,et al.  Triangulation as a basis for knowledge discovery in software engineering , 2008, Empirical Software Engineering.

[73]  R.N. Charette,et al.  Why software fails [software failure] , 2005, IEEE Spectrum.

[74]  Giovan Francesco Lanzara,et al.  Between transient constructs and persistent structures: designing systems in action , 1999, J. Strateg. Inf. Syst..

[75]  John Mingers,et al.  Combining IS Research Methods: Towards a Pluralist Methodology , 2001, Inf. Syst. Res..

[76]  Helen Sharp,et al.  The type of evidence produced by empirical software engineers , 2005, ACM SIGSOFT Softw. Eng. Notes.

[77]  Wanda J. Orlikowski,et al.  A Practice Perspective on Technology-Mediated Network Relations: The Use of Internet-Based Self-Serve Technologies , 2004, Inf. Syst. Res..

[78]  Ian K. Allison,et al.  Software process improvement as emergent change : a structurational analysis , 2007, Inf. Softw. Technol..

[79]  Christopher A. Voss,et al.  Case research in operations management , 2002 .

[80]  Eswaran Subrahmanian,et al.  Boundary Objects and Prototypes at the Interfaces of Engineering Design , 2003, Computer Supported Cooperative Work (CSCW).

[81]  C. Geertz.  The interpretation of cultures: Selected essays , 1975 .

[82]  Donald R. Cooper,et al.  Business Research Methods , 1980 .

[83]  Shoshana Zuboff.  In the Age of the Smart Machine , 1988 .

[84]  Yvonne Dittrich,et al.  'Bad practice' or 'Bad methods' are software engineering and ethnographic discourses incompatible? , 2002, Proceedings International Symposium on Empirical Software Engineering.

[85]  Karen Locke,et al.  Appealing Work: An Investigation of How Ethnographic Texts Convince , 1993 .

[86]  Yvonne Dittrich,et al.  Cooperative method development , 2008, Empirical Software Engineering.

[87]  Helen Sharp,et al.  Ethnographically-informed empirical studies of software practice , 2007, Inf. Softw. Technol..