Towards a decision-making structure for selecting a research design in empirical software engineering

Several factors make empirical research in software engineering particularly challenging as it requires studying not only technology but its stakeholders’ activities while drawing concepts and theories from social science. Researchers, in general, agree that selecting a research design in empirical software engineering research is challenging, because the implications of using individual research methods are not well recorded. The main objective of this article is to make researchers aware and support them in their research design, by providing a foundation of knowledge about empirical software engineering research decisions, in order to ensure that researchers make well-founded and informed decisions about their research designs. This article provides a decision-making structure containing a number of decision points, each one of them representing a specific aspect on empirical software engineering research. The article provides an introduction to each decision point and its constituents, as well as to the relationships between the different parts in the decision-making structure. The intention is the structure should act as a starting point for the research design before going into the details of the research design chosen. The article provides an in-depth discussion of decision points in relation to the research design when conducting empirical research.

[1]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[2]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[3]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[4]  G. Susman,et al.  An Assessment of the Scientific Merits of Action Research. , 1978 .

[5]  C. Achilles,et al.  Evaluation: A Systematic Approach , 1980 .

[6]  Peter Checkland,et al.  Systems Thinking, Systems Practice , 1981 .

[7]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[8]  Sam Kash Kachigan.  Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods , 1986 .

[9]  Izak Benbasat,et al.  The Case Research Strategy in Studies of Information Systems , 1987, MIS Q..

[10]  S. Shye.  Inductive and deductive reasoning: A structural reanalysis of ability tests. , 1988 .

[11]  L. A. Marascuilo,et al.  Statistical Methods for the Social and Behavioral Sciences. , 1989 .

[12]  Allen S. Lee.  A Scientific Methodology for MIS Case Studies , 1989, MIS Q..

[13]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[14]  Leonard A. Marascuilo,et al.  Statistical methods for the social and behavioral sciences , 1990 .

[15]  Wanda J. Orlikowski,et al.  Studying Information Technology in Organizations: Research Approaches and Assumptions , 1991, Inf. Syst. Res..

[16]  Jay F. Nunamaker,et al.  Systems Development in Information Systems Research , 1990, J. Manag. Inf. Syst..

[17]  J. McKernan.  Curriculum Action Research: A Handbook of Methods and Resources for the Reflective Practitioner , 1991 .

[18]  B. Glaser.  Basics of Grounded Theory Analysis: Emergence Vs. Forcing , 1992 .

[19]  Victor R. Basili,et al.  The Experimental Paradigm in Software Engineering , 1992, Experimental Software Engineering Issues.

[20]  Alain Pinsonneault,et al.  Survey Research Methodology in Management Information Systems: An Assessment , 1993, J. Manag. Inf. Syst..

[21]  B. Kitchenham,et al.  Case Studies for Method and Tool Evaluation , 1995, IEEE Softw..

[22]  Paul Lukowicz,et al.  Experimental evaluation in computer science: A quantitative study , 1995, J. Syst. Softw..

[23]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[24]  M. D. Myers,et al.  Dialectical hermeneutics: a theoretical framework for the implementation of information systems , 1995, Inf. Syst. J..

[25]  G. Nigel Gilbert,et al.  Social Science Microsimulation [Dagstuhl Seminar, May, 1995] , 1996, Social Science Microsimulation.

[26]  C. F. Johnson,et al.  Deductive versus inductive reasoning: A closer look at economics , 1996 .

[27]  Marvin V. Zelkowitz,et al.  Experimental validation in software engineering , 1997, Inf. Softw. Technol..

[28]  Judy L. Wynekoop,et al.  Studying system development methodologies: an examination of research methods , 1997, Inf. Syst. J..

[29]  O.W. Bertelsen.  Toward A Unified Field Of SE Research And Practice , 1997, IEEE Software.

[30]  Michael D. Myers,et al.  Qualitative Research in Information Systems , 1997, MIS Q..

[31]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[32]  Tom Butler,et al.  Towards a hermeneutic method for interpretive research in information systems , 1998, J. Inf. Technol..

[33]  C. Perry.  Processes of a case study methodology for postgraduate research in marketing , 1998 .

[34]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[35]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[36]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[37]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[38]  Ajay S. Vinze,et al.  Understanding the Philosophical Underpinnings of Software Engineering Research in Information Systems , 2001, Inf. Syst. Frontiers.

[39]  John Mingers,et al.  Combining IS Research Methods: Towards a Pluralist Methodology , 2001, Inf. Syst. Res..

[40]  J. Grix.  Introducing Students to the Generic Terminology of Social Research , 2002 .

[41]  Cathy Urquhart,et al.  Regrounding grounded theory or reinforcing old prejudices? A brief reply to Bryant , 2002 .

[42]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[43]  Mary Shaw,et al.  What makes good research in software engineering? , 2002, International Journal on Software Tools for Technology Transfer.

[44]  CAROLE BROOKE,et al.  What does it mean to be ‘critical’ in IS research? , 2002, J. Inf. Technol..

[45]  Shari Lawrence Pfleeger,et al.  Principles of survey research part 2: designing a survey , 2002, SOEN.

[46]  Mary Shaw,et al.  Writing good software engineering research papers , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[47]  Mary Shaw,et al.  Writing good software engineering research papers: minitutorial , 2003, ICSE 2003.

[48]  Graeme G. Shanks,et al.  Positivist single case study research in information systems: a critical analysis , 2003, ECIS.

[49]  Regula Kyburz-graber.  Does case‐study methodology lack rigour? The need for quality criteria for sound case‐study research, as illustrated by a recent case in secondary and higher education , 2004 .

[50]  Rudy Hirschheim,et al.  A paradigmatic and methodological examination of information systems research from 1991 to 2001 , 2004, Inf. Syst. J..

[51]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[52]  Ian K. Allison,et al.  Software process improvement as emergent change : a structurational analysis , 2007, Inf. Softw. Technol..

[53]  Laura Lehtola,et al.  Using the focus group method in software engineering: obtaining practitioner and user experiences , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[54]  A. Onwuegbuzie,et al.  Mixed Methods Research: A Research Paradigm Whose Time Has Come , 2004 .

[55]  Robert M. Davison,et al.  Principles of canonical action research , 2004, Inf. Syst. J..

[56]  Jean Hartley,et al.  Case study research , 2004 .

[57]  R. Schutt,et al.  The Practice of Research in Social Work , 2005 .

[58]  Janice Singer,et al.  Studying Software Engineers: Data Collection Techniques for Software Field Studies , 2005, Empirical Software Engineering.

[59]  N. Mack,et al.  Qualitative research methods: a data collectors field guide. , 2005 .

[60]  Liisa von Hellens,et al.  Qualitative Research in Information Systems , 2007, Australas. J. Inf. Syst..

[61]  Miroslaw Staron,et al.  Empirical assessment of using stereotypes to improve comprehension of UML models: A set of experiments , 2006, J. Syst. Softw..

[62]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[63]  F. Myrick,et al.  Grounded Theory: An Exploration of Process and Procedure , 2006, Qualitative health research.

[64]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[65]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[66]  Rory V. O'Connor,et al.  Using grounded theory to understand software process improvement: A study of Irish software product companies , 2007, Inf. Softw. Technol..

[67]  Kathleen M. Carley,et al.  Simulation modeling in organizational and management research , 2007 .

[68]  M. Apple,et al.  What Does It Mean to be Critical , 2008 .

[69]  Edmund Chattoe.  Dagstuhl Seminar on Social Science Microsimulation : A Challenge to Computer Science , 2008 .

[70]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[71]  Rory O'Connor,et al.  Investigating software process in practice: A grounded theory perspective , 2008, J. Syst. Softw..

[72]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[73]  Richard Baskerville,et al.  What design science is not , 2008, Eur. J. Inf. Syst..

[74]  Tony Gorschek,et al.  Empirical evidence in global software engineering: a systematic review , 2010, Empirical Software Engineering.

[75]  Howard Lyons.  Case study research methodology for publishing developments in ICT‐facilitated learning in higher education – a prescriptive approach , 2009 .

[76]  Sean W. Hansen,et al.  Getting on the same page: Collective hermeneutics in a systems development team , 2010, Inf. Organ..

[77]  Sebastian K. Boell,et al.  Literature Reviews and the Hermeneutic Circle , 2010 .

[78]  Stephen G. MacDonell,et al.  Qualitative research on software development: a longitudinal case study methodology , 2011, Empirical Software Engineering.

[79]  Philippe Kruchten,et al.  Using grounded theory to study the experience of software development , 2011, Empirical Software Engineering.

[80]  David R. Hannah,et al.  Counting in Qualitative Research: Why to Conduct it, When to Avoid it, and When to Closet it , 2011 .

[81]  Ciarán Dunne.  The place of the literature review in grounded theory research , 2011 .

[82]  Dubravka Cecez-Kecmanovic,et al.  Doing critical information systems research – arguments for a critical research methodology , 2011, Eur. J. Inf. Syst..

[83]  Tore Dybå,et al.  Qualitative research in software engineering , 2011, Empirical Software Engineering.

[84]  J. D'Ambra,et al.  Factors Affecting the Success of Businesses in Effective Utilization of Search Engine Advertising Research-in-Progress , 2011 .

[85]  Markus Helfert,et al.  Commonality in Various Design Science Methodologies , 2011, FedCSIS.

[86]  Michael D. Myers,et al.  A Set of Principles for Conducting Critical Research in Information Systems , 2011, MIS Q..

[87]  Austen Rainer,et al.  Case Study Research in Software Engineering - Guidelines and Examples , 2012 .

[88]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[89]  Tore Dybå,et al.  Challenges of shared decision-making: A multiple case study of agile software development , 2012, Inf. Softw. Technol..

[90]  Anol Bhattacherjee.  Social Science Research: Principles, Methods, and Practices , 2012 .

[91]  Marcia Mkansi,et al.  Research philosophy debates and classifications: students' dilemma , 2012 .

[92]  Pekka Abrahamsson,et al.  Making the leap to a software platform strategy: Issues and challenges , 2012, Inf. Softw. Technol..

[93]  Syed Nasirin,et al.  Grounded theory method in information systems research: its nature, diversity and opportunities , 2013, Eur. J. Inf. Syst..

[94]  John D'Ambra,et al.  Determinant of Intention to Use Search Engine Advertising: A Conceptual Model , 2013, Int. J. Enterp. Inf. Syst..

[95]  Richard T. Vidgen,et al.  An exploration of technical debt , 2013, J. Syst. Softw..

[96]  Claes Wohlin,et al.  Offshore insourcing in software development: Structuring the decision-making process , 2013, J. Syst. Softw..

[97]  Lefteris Angelis,et al.  Software quality across borders: Three case studies on company internal alignment , 2014, Inf. Softw. Technol..