A Taxonomy of Evaluation Approaches in Software Engineering

As in any academic discipline, the evaluation of proposed methodologies and techniques is of vital importance for assessing the validity of novel ideas or findings in Software Engineering. Over the years, a large number of evaluation approaches have been employed, some of them drawn from other domains and other particularly developed for the needs of software engineering related research. In this paper we present the results of a survey of evaluation techniques that have been utilized in research papers that appeared in three leading software engineering journal and propose a taxonomy of evaluation approaches which might be helpful towards the organization of knowledge regarding the different strategies for the validation of research outcomes. The applicability of the proposed taxonomy has been evaluated by classifying the articles retrieved from ICSE'2012.

[1]  N. Savin.  A TEST OF THE MONTE CARLO HYPOTHESIS: COMMENT* , 1977 .

[2]  Keng Siau,et al.  Evaluation techniques for systems analysis and design modelling methods – a review and comparative analysis , 2011, Inf. Syst. J..

[3]  Roger W. Hockney.  The science of computer benchmarking , 1995, Software, environments, tools.

[4]  David L. Olson,et al.  Advanced Data Mining Techniques , 2008 .

[5]  Jan Muntermann,et al.  Taxonomy development in information systems: Developing a taxonomy of mobile applications , 2009, ECIS.

[6]  Venkataraman Ramesh,et al.  Research in software engineering: an analysis of the literature , 2002, Inf. Softw. Technol..

[7]  R R Sokal,et al.  Classification: Purposes, Principles, Progress, Prospects , 1974, Science.

[8]  Manuel Trenz,et al.  Building Taxonomies in IS and Management - A Systematic Approach Based on Content Analysis , 2013, Wirtschaftsinformatik.

[9]  Venkataraman Ramesh,et al.  A unified classification system for research in the computing disciplines , 2005, Inf. Softw. Technol..

[10]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[11]  Alicia Perez,et al.  Evaluation of Taxonomic Knowledge in Ontologies and Knowledge Bases , 1999 .

[12]  Sandro Morasca,et al.  Property-Based Software Engineering Measurement , 1996, IEEE Trans. Software Eng..

[13]  Peter J. Weinberger,et al.  Research in software , 1992, AT&T Technical Journal.

[14]  Mary Shaw,et al.  What makes good research in software engineering? , 2002, International Journal on Software Tools for Technology Transfer.

[15]  Rebecca Green,et al.  Typologies and taxonomies: An introduction to classification techniques , 1996 .

[16]  David A. Patterson,et al.  Computer Architecture: A Quantitative Approach , 1969 .

[17]  P. Rich.  The Organizational Taxonomy: Definition and Design , 1992 .

[18]  N. Breslow.  A generalized Kruskal-Wallis test for comparing K samples subject to unequal patterns of censorship , 1970 .

[19]  T. Hales.  Formal Proof , 2008 .