Towards Multi-Method Research Approach in Empirical Software Engineering

This paper presents results of a literature analysis on Empirical Research Approaches in Software Engineering (SE). The analysis explores reasons why traditional methods, such as statistical hypothesis testing and experiment replication are weakly utilized in the field of SE. It appears that basic assumptions and preconditions of the traditional methods are contradicting the actual situation in the SE. Furthermore, we have identified main issues that should be considered by the researcher when selecting the research approach. In virtue of reasons for weak utilization of traditional methods we propose stronger use of Multi-Method approach with Pragmatism as the philosophical standpoint.

[1]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[2]  John W. Daly,et al.  Statistical power and its subcomponents - missing and misunderstood concepts in empirical software engineering research , 1997, Inf. Softw. Technol..

[3]  Steve Sawyer,et al.  Studying Organizational Computing Infrastructures: Multi-method Approaches , 2000, Organizational and Social Perspectives on IT.

[4]  Magne Jørgensen,et al.  Can you Trust a Single Data Source Exploratory Software Engineering Case Study? , 2002, Empirical Software Engineering.

[5]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[6]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[7]  Markku Oivo.  New Opportunities for Empirical Research , 2006, Empirical Software Engineering Issues.

[8]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[9]  Victor R. Basili.  Measurement and Model Building Introduction , 2006, Empirical Software Engineering Issues.

[10]  Jay F. Nunamaker,et al.  Systems Development in Information Systems Research , 1990, J. Manag. Inf. Syst..

[11]  Stuart Anderson,et al.  Requirements Evolution from Process to Product Oriented Management , 2001, PROFES.

[12]  Dietmar Pfahl,et al.  Reporting Experiments in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[13]  Christopher M. Lott,et al.  Repeatable software engineering experiments for comparing defect-detection techniques , 2004, Empirical Software Engineering.

[14]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[15]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[16]  Marvin V. Zelkowitz,et al.  Experimental validation in software engineering , 1997, Inf. Softw. Technol..

[17]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[18]  Victor R. Basili,et al.  The role of experimentation in software engineering: past, current, and future , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[19]  Albert Endres,et al.  A handbook of software and systems engineering - empirical observations, laws and theories , 2003, The Fraunhofer IESE series on software engineering.

[20]  B. Curtis,et al.  Measurement and experimentation in software engineering , 1980, Proceedings of the IEEE.

[21]  John Mingers,et al.  Combining IS Research Methods: Towards a Pluralist Methodology , 2001, Inf. Syst. Res..

[22]  James Miller,et al.  Replicating software engineering experiments: a poisoned chalice or the Holy Grail , 2005, Inf. Softw. Technol..

[23]  Mary Shaw,et al.  What makes good research in software engineering? , 2002, International Journal on Software Tools for Technology Transfer.

[24]  Victor R. Basili,et al.  Paradigms for experimentation and empirical studies in software engineering , 1991 .

[25]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[26]  John W. Daly,et al.  Multi-method research: An empirical investigation of object-oriented technology , 1999, J. Syst. Softw..

[27]  Khaled El Emam,et al.  Elements of Software Process Assessment & Improvement , 1999 .

[28]  A. Brooks,et al.  Replication's Role in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[29]  Robert D. Galliers,et al.  Research issues in information systems , 1993, J. Inf. Technol..

[30]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[31]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[32]  James Miller.  Statistical significance testing--a panacea for software technology experiments? , 2004, J. Syst. Softw..

[33]  Stacie Petter,et al.  Toward a framework for classifying and guiding mixed method research in information systems , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[34]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[35]  Jouni Similä,et al.  Software Engineering Research Strategy: Combining Experimental and Explorative Research (EER) , 2004, PROFES.

[36]  Venkataraman Ramesh,et al.  Research in computer science: an empirical study , 2004, J. Syst. Softw..

[37]  Sira Vegas,et al.  Measurement and Model Building Discussion and Summary , 2006, Empirical Software Engineering Issues.

[38]  K. Howe.  Against the Quantitative-Qualitative Incompatibility Thesis or Dogmas Die Hard , 1988 .

[39]  Victor R. Basili,et al.  Experimentation in software engineering , 1986, IEEE Transactions on Software Engineering.

[40]  Robert L. Glass,et al.  Science and substance: a challenge to software engineers , 1994, IEEE Software.