Applying Systematic Reviews to Diverse Study Types: An Experience Report

Systematic reviews are one of the key building blocks of evidence-based software engineering. Current guidelines for such reviews are, for a large part, based on standard meta-analytic techniques. However, such quantitative techniques have only limited applicability to software engineering research. In this paper, therefore, we describe our experience with an approach to combine diverse study types in a systematic review of empirical research of agile software development.

[1]  Paula Gomes Mian,et al.  Systematic Review in Software Engineering , 2005 .

[2]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[3]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[4]  Deborah J. Cook,et al.  Systematic Reviews: Synthesis of Best Evidence for Health Care Decisions , 1998, Annals of Internal Medicine.

[5]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[6]  Chaomei Chen,et al.  Representing the Semantics of Virtual Spaces , 1999, IEEE Multim..

[7]  G. Noblit,et al.  Meta-Ethnography: Synthesizing Qualitative Studies , 1988 .

[8]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[9]  David R. Jones,et al.  Synthesising qualitative and quantitative evidence: A review of possible methods , 2005 .

[10]  Lucy Dillon,et al.  Quality in qualitative evaluation: A framework for assessing research evidence. A quality framework. , 2003 .

[11]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[12]  Rosane Minghim,et al.  Text Map Explorer: a Tool to Create and Explore Document Maps , 2006, Tenth International Conference on Information Visualisation (IV'06).

[13]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[14]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[15]  Mary Dixon-Woods,et al.  Synthesising qualitative and quantitative evidence: a review of possible methods. , 2005, Journal of health services research & policy.

[16]  Pak Chung Wong,et al.  Visualizing association rules for text mining , 1999, Proceedings 1999 IEEE Symposium on Information Visualization (InfoVis'99).

[17]  Tore Dybå,et al.  A Systematic Review of Theory Use in Software Engineering Experiments , 2007, IEEE Transactions on Software Engineering.

[18]  Haim Levkowitz,et al.  From Visual Data Exploration to Visual Data Mining: A Survey , 2003, IEEE Trans. Vis. Comput. Graph..

[19]  A. Harden,et al.  Methodological Issues in Combining Diverse Study Types in Systematic Reviews , 2005 .

[20]  Rosane Minghim,et al.  Visual text mining using association rules , 2007, Comput. Graph..

[21]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[22]  James Miller,et al.  Applying meta-analytical procedures to software engineering experiments , 2000, J. Syst. Softw..

[23]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007 .

[24]  Susan Wiedenbeck,et al.  Empirical studies of software engineering , 2004, Int. J. Hum. Comput. Stud..

[25]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[26]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[27]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[28]  Rosane Minghim,et al.  On Improved Projection Techniques to Support Visual Exploration of Multi-Dimensional Data Sets , 2003, Inf. Vis..

[29]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[30]  S. Hawker,et al.  Appraising the Evidence: Reviewing Disparate Data Systematically , 2002, Qualitative health research.

[31]  Douglas G. Altman,et al.  Systematic Reviews in Health Care: Meta-Analysis in Context: Second Edition , 2008 .

[32]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[33]  C. Pope,et al.  Using meta ethnography to synthesise qualitative research: a worked example , 2002, Journal of health services research & policy.

[34]  Christos Faloutsos,et al.  FastMap: a fast algorithm for indexing, data-mining and visualization of traditional and multimedia datasets , 1995, SIGMOD '95.

[35]  Mark Staples,et al.  Experiences using systematic review guidelines , 2006, J. Syst. Softw..

[36]  Trisha Greenhalgh,et al.  How to Read a Paper , 1999 .

[37]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[38]  Douglas G. Altman,et al.  Systematic Reviews in Health Care , 2001 .