Towards understanding the relation between citations and research quality in software engineering studies

The importance of achieving high quality in research practice has been highlighted in different disciplines. At the same time, citations are utilized to measure the impact of academic researchers and institutions. One open question is whether the quality in the reporting of research is related to scientific impact, which would be desired. In this exploratory study we aim to: (1) Investigate how consistently a scoring rubric for rigor and relevance has been used to assess research quality of software engineering studies; (2) Explore the relationship between rigor, relevance and citation count. Through backward snowball sampling we identified 718 primary studies assessed through the scoring rubric. We utilized cluster analysis and conditional inference tree to explore the relationship between quality in the reporting of research (represented by rigor and relevance) and scientiometrics (represented by normalized citations). The results show that only rigor is related to studies’ normalized citations. Besides that, confounding factors are likely to influence the number of citations. The results also suggest that the scoring rubric is not applied the same way by all studies, and one of the likely reasons is because it was found to be too abstract and in need to be further refined. Our findings could be used as a basis to further understand the relation between the quality in the reporting of research and scientific impact, and foster new discussions on how to fairly acknowledge studies for performing well with respect to the emphasized research quality. Furthermore, we highlighted the need to further improve the scoring rubric.

[1]  Anthony F. J. van Raan,et al.  Scientometrics: State-of-the-art , 2006, Scientometrics.

[2]  Tony Gorschek,et al.  The lean gap: A review of lean approaches to large-scale software systems development , 2013, J. Syst. Softw..

[3]  Frank Elberzhager,et al.  Reducing test effort: A systematic mapping study on existing approaches , 2012, Inf. Softw. Technol..

[4]  Sairam Vakkalanka,et al.  Investigating Research on Teaching Modeling in Software Engineering -A Systematic Mapping Study , 2013 .

[5]  Claes Wohlin,et al.  Context in industrial software engineering research , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[6]  Helmut Strasser,et al.  On the Asymptotic Theory of Permutation Statistics , 1999 .

[7]  Kai Petersen,et al.  Considering rigor and relevance when evaluating test driven development: A systematic review , 2014, Inf. Softw. Technol..

[8]  Danny Weyns,et al.  Variability in Software Systems—A Systematic Literature Review , 2014, IEEE Transactions on Software Engineering.

[9]  PetersenKai,et al.  Software quality trade-offs , 2012 .

[10]  Paris Avgeriou,et al.  Variability in quality attributes of service-based software systems: A systematic literature review , 2013, Inf. Softw. Technol..

[11]  Ajmal Iqbal,et al.  Communication Risks and Best practices in Global Software Development , 2012 .

[12]  Per Runeson,et al.  Checklists for Software Engineering Case Study Research , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[13]  Claes Wohlin,et al.  An analysis of the most cited articles in software engineering journals - 2002 , 2009, Inf. Softw. Technol..

[14]  Phani Shashank Saripalli,et al.  Finding common denominators for agile software development : a systematic literature review , 2011 .

[15]  Tsong Yueh Chen,et al.  An assessment of systems and software engineering scholars and institutions (2002-2006) , 2009, J. Syst. Softw..

[16]  Robert L. Glass,et al.  An assessment of systems and software engineering scholars and institutions (1993-1997) , 1997, Journal of Systems and Software.

[17]  D. Aksnes CHARACTERISTICS OF HIGHLY CITED PAPERS , 2003 .

[18]  Anders Mårtensson,et al.  Extending Rigor and Relevance: Towards Credible, Contributory and Communicable Research , 2007, ECIS.

[19]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[20]  Brahim Hamadicharef,et al.  Scientometric Study of the IEEE Transactions on Software Engineering 1980-2010 , 2012 .

[21]  Pearl Brereton,et al.  Evidence relating to Object-Oriented software design: A survey , 2007, ESEM 2007.

[22]  Muhammad Ayaz,et al.  Investigation of Customer-Driven Innovation , 2013 .

[23]  Claes Wohlin,et al.  A systematic literature review on the industrial use of software process simulation , 2014, J. Syst. Softw..

[24]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[25]  K. Hornik,et al.  Unbiased Recursive Partitioning: A Conditional Inference Framework , 2006 .

[26]  Vahid Garousi,et al.  Quantity versus impact of software engineering papers: a quantitative study , 2017, Scientometrics.

[27]  Marvin V. Zelkowitz,et al.  An update to experimental models for validating computer technology , 2009, J. Syst. Softw..

[28]  Claes Wohlin,et al.  A new index for the citation curve of researchers , 2009, Scientometrics.

[29]  Dag W. Aksnes,et al.  Citation rates and perceptions of scientific contribution , 2006, J. Assoc. Inf. Sci. Technol..

[30]  Kelvin J. Ross,et al.  Industry Academia Collaboration: An Experience Report at a Small University , 2009, 2009 22nd Conference on Software Engineering Education and Training.

[31]  Aybüke Aurum,et al.  Software quality trade-offs: A systematic map , 2012, Inf. Softw. Technol..

[32]  João M. Fernandes,et al.  Authorship trends in software engineering , 2014, Scientometrics.

[33]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[34]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[35]  Vahid Garousi,et al.  Highly-cited papers in software engineering: The top-100 , 2016, Inf. Softw. Technol..

[36]  Austen Rainer,et al.  Case Study Research in Software Engineering - Guidelines and Examples , 2012 .

[37]  Vahid Garousi A bibliometric analysis of the Turkish software engineering research community , 2015, Scientometrics.

[38]  Tore Dybå,et al.  Strength of evidence in systematic reviews in software engineering , 2008, ESEM '08.

[39]  Raya Fidel,et al.  The many faces of accessibility: engineers' perception of information sources , 2004, Inf. Process. Manag..

[40]  Mary Shaw,et al.  Writing good software engineering research papers , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[41]  Daren Yu,et al.  Mining typical features for highly cited papers , 2011, Scientometrics.

[42]  M. Amin,et al.  Impact factors: use and abuse. , 2003, Medicina.

[43]  Marcelo Fantinato,et al.  The use of software product lines for business process management: A systematic literature review , 2013, Inf. Softw. Technol..

[44]  Vahid Garousi,et al.  Web application testing: A systematic literature review , 2014, J. Syst. Softw..

[45]  Gabriel Alberto,et al.  A Process Framework for Managing Interactions between Sotware Quality Charateristics , 2014 .

[46]  Vahid Garousi,et al.  Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review , 2016, Inf. Softw. Technol..

[47]  Tore Dybå,et al.  Conducting realistic experiments in software engineering , 2002, Proceedings International Symposium on Empirical Software Engineering.

[48]  Tony Gorschek,et al.  A method for evaluating rigor and industrial relevance of technology evaluations , 2011, Empirical Software Engineering.

[49]  Jilles van Gurp Variability in Software Systems , 2000 .

[50]  Carlo Ghezzi,et al.  Determining the Impact of Software Engineering Research on Practice , 2008, Computer.

[51]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[52]  Vahid Garousi,et al.  Using Citation Behavior to Rethink Academic Impact in Software Engineering , 2015, 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[53]  Moshe Yitzhaki,et al.  Accessibility and use of information sources among computer scientists and software engineers in Israel: Academy versus industry , 2004, J. Assoc. Inf. Sci. Technol..

[54]  Joaquim P. Marques de Sá,et al.  Applied statistics : using SPSS, STATISTICA, and MATLAB , 2003 .

[55]  Tony Gorschek,et al.  Software development in startup companies: A systematic mapping study , 2014, Inf. Softw. Technol..

[56]  Austen Rainer,et al.  Software engineering practice versus evidence- based software engineering research , 2005 .

[57]  Per Runeson,et al.  Checklists for Software Engineering Case Study Research , 2007, ESEM 2007.

[58]  Claes Wohlin,et al.  An analysis of the most cited articles in software engineering journals - 1999 , 2005, Inf. Softw. Technol..

[59]  Gabriel Alberto García-Mireles,et al.  The Influence of Process Quality on Product Usability: A Systematic Review , 2013, CLEI Electron. J..

[60]  Pearl Brereton,et al.  Evidence-Based Software Engineering and Systematic Reviews , 2015 .

[61]  John Mingers,et al.  Beyond rigour and relevance: A critical realist approach to business education , 2010 .

[62]  Pearl Brereton,et al.  A systematic review of systematic review process research in software engineering , 2013, Inf. Softw. Technol..

[63]  David Lorge Parnas,et al.  Stop the numbers game , 2007, CACM.

[64]  WohlinClaes An analysis of the most cited articles in software engineering journals - 2002 , 2009 .

[65]  Izak Benbasat,et al.  Empirical Research in Information Systems: The Practice of Relevance , 1999, MIS Q..

[66]  Mary Shaw,et al.  Writing good software engineering research papers: minitutorial , 2003, ICSE 2003.

[67]  Uno Fors,et al.  Evaluating research: A multidisciplinary approach to assessing research practice and quality , 2016 .