Highly-cited papers in software engineering: The top-100

Abstract Context According to the search reported in this paper, as of this writing (May 2015), a very large number of papers (more than 70,000) have been published in the area of Software Engineering (SE) since its inception in 1968. Citations are crucial in any research area to position the work and to build on the work of others. Identification and characterization of highly-cited papers are common and are regularly reported in various disciplines. Objective The objective of this study is to identify the papers in the area of SE that have influenced others the most as measured by citation count. Studying highly-cited SE papers helps researchers to see the type of approaches and research methods presented and applied in such papers, so as to be able to learn from them to write higher quality papers which will likely receive high citations. Method To achieve the above objective, we conducted a study, comprised of five research questions, to identify and classify the top-100 highly-cited SE papers in terms of two metrics: total number of citations and average annual number of citations. Results By total number of citations, the top paper is " A metrics suite for object-oriented design ", cited 1817 times and published in 1994. By average annual number of citations, the top paper is " QoS-aware middleware for Web services composition ", cited 154.2 times on average annually and published in 2004. Conclusion It is concluded that it is important to identify the highly-cited SE papers and also to characterize the overall citation landscape in the SE field. We hope that this paper will encourage further discussions in the SE community towards further analysis and formal characterization of the highly-cited SE papers.

[1]  João M. Fernandes,et al.  Authorship trends in software engineering , 2014, Scientometrics.

[2]  Elizabeth Smith Aversa,et al.  Citation patterns of highly cited papers and their relationship to literature aging: A study of the working literature , 1985, Scientometrics.

[3]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[4]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[5]  Claes Wohlin,et al.  An analysis of the most cited articles in software engineering journals - 2001 , 2008, Inf. Softw. Technol..

[6]  Claes Wohlin,et al.  An analysis of the most cited articles in software engineering journals - 2002 , 2009, Inf. Softw. Technol..

[7]  Claes Wohlin,et al.  An analysis of the most cited articles in software engineering journals - 1999 , 2005, Inf. Softw. Technol..

[8]  Henry G. Small,et al.  Why authors think their papers are highly cited , 2004, Scientometrics.

[9]  Rajkumar Buyya,et al.  CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms , 2011, Softw. Pract. Exp..

[10]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[11]  A. Lozano,et al.  Highly cited works in neurosurgery. Part I: the 100 top-cited papers in neurosurgical journals. , 2010, Journal of neurosurgery.

[12]  Tsong Yueh Chen,et al.  An assessment of systems and software engineering scholars and institutions (2002-2006) , 2009, J. Syst. Softw..

[13]  Vahid Garousi,et al.  Development of Scientific Software: a Systematic Mapping, a bibliometrics Study, and a Paper Repository , 2013, Int. J. Softw. Eng. Knowl. Eng..

[14]  T.C. Lethbridge,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK) and the Software Engineering Education Knowledge (SEEK) - a preliminary mapping , 2001, 10th International Workshop on Software Technology and Engineering Practice.

[15]  Zoë Corbyn To be the best, cite the best , 2010 .

[16]  Robert L. Glass,et al.  An assessment of systems and software engineering scholars and institutions (1993-1997) , 1997, Journal of Systems and Software.

[17]  Tindaro Cicero,et al.  Are the authors of highly cited articles also the most productive ones? , 2018, J. Informetrics.

[18]  Robert L. Glass An assessment of systems and software engineering scholars and institutions (1994-1998) , 1999, J. Syst. Softw..

[19]  Richard Van Noorden,et al.  The top 100 papers , 2014, Nature.

[20]  Claes Wohlin,et al.  An analysis of the most cited articles in software engineering journals - 2000 , 2007, Inf. Softw. Technol..

[21]  E GARFIELD,et al.  Citation indexes for science; a new dimension in documentation through association of ideas. , 2006, Science.

[22]  Raymond J. A. Buhr,et al.  Use Case Maps as Architectural Entities for Complex Systems , 1998, IEEE Trans. Software Eng..

[23]  Jerffeson Teixeira de Souza,et al.  Ten Years of Search Based Software Engineering: A Bibliometric Analysis , 2011, SSBSE.

[24]  Olle Persson,et al.  Are highly cited papers more international? , 2010, Scientometrics.

[25]  Robert L. Glass,et al.  An assessment of systems and software engineering scholars and institutions (2001-2005) , 2008, J. Syst. Softw..

[26]  Thed N. van Leeuwen,et al.  Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? , 2002, Scientometrics.

[27]  Lars Michael Kristensen,et al.  Coloured Petri Nets and CPN Tools for modelling and validation of concurrent systems , 2007, International Journal on Software Tools for Technology Transfer.

[28]  Tsong Yueh Chen,et al.  An assessment of systems and software engineering scholars and institutions (1993-1997) , 1997, J. Syst. Softw..

[29]  Audris Mockus,et al.  An Empirical Study of Speed and Communication in Globally Distributed Software Development , 2003, IEEE Trans. Software Eng..

[30]  D. Richardson,et al.  Who cites who in the invasion zoo: insights from an analysis of the most highly cited papers in invasion ecology. , 2006 .

[31]  John P. A. Ioannidis,et al.  Bibliometrics: Is your most cited work your best? , 2014, Nature.

[32]  V. Basili Software modeling and measurement: the Goal/Question/Metric paradigm , 1992 .

[33]  A Straw,et al.  Guide to the Software Engineering Body of Knowledge , 1998 .

[34]  Vahid Garousi,et al.  A bibliometric/Geographic Assessment of 40 Years of Software Engineering Research (1969-2009) , 2013, Int. J. Softw. Eng. Knowl. Eng..

[35]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..

[36]  Wang Yi,et al.  Uppaal in a nutshell , 1997, International Journal on Software Tools for Technology Transfer.

[37]  E. Garfield Citation indexes for science. A new dimension in documentation through association of ideas. 1955. , 1955, International journal of epidemiology.

[38]  A Bibliometric Assessment of Canadian Software Engineering Scholars and Institutions (1996-2006) , 2010, Comput. Inf. Sci..

[39]  David Lorge Parnas,et al.  Stop the numbers game , 2007, CACM.

[40]  Kai-Yuan Cai,et al.  An analysis of research topics in software engineering - 2006 , 2008, J. Syst. Softw..

[41]  Nobuko Miyairi,et al.  Bibliometric characteristics of highly cited papers from Taiwan, 2000–2009 , 2012, Scientometrics.

[42]  Gerald G. Brown,et al.  Assessing What Distinguishes Highly Cited from Less-Cited Papers Published in Interfaces , 2010, Interfaces.

[43]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[44]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[45]  Vahid Garousi A bibliometric analysis of the Turkish software engineering research community , 2015, Scientometrics.

[46]  D. Aksnes CHARACTERISTICS OF HIGHLY CITED PAPERS , 2003 .

[47]  David J. Eaton,et al.  Highly cited papers in Medical Physics , 2014 .

[48]  Chester A. Schriesheim,et al.  What makes articles highly cited , 2014 .

[49]  Edward M. Reingold,et al.  Graph drawing by force‐directed placement , 1991, Softw. Pract. Exp..

[50]  Daren Yu,et al.  Mining typical features for highly cited papers , 2011, Scientometrics.

[51]  Giovanni Vigna,et al.  Understanding Code Mobility , 1998, IEEE Trans. Software Eng..

[52]  Barbara Kitchenham,et al.  What's up with software metrics? - A preliminary mapping study , 2010, J. Syst. Softw..

[53]  Lutz Bornmann,et al.  How are excellent (highly cited) papers defined in bibliometrics? A quantitative analysis of the literature , 2014, ArXiv.

[54]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[55]  Abdullah Abrizah,et al.  LIS journals scientific impact and subject categorization: a comparison between Web of Science and Scopus , 2012, Scientometrics.

[56]  Rickard Danell,et al.  Can the quality of scientific work be predicted using information on the author's track record? , 2011, J. Assoc. Inf. Sci. Technol..

[57]  Éric Archambault,et al.  Comparing bibliometric statistics obtained from the Web of Science and Scopus , 2009 .

[58]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[59]  Mark E. J. Newman,et al.  Prediction of highly cited papers , 2013, ArXiv.

[60]  Amer Diwan,et al.  The DaCapo benchmarks: java benchmarking development and analysis , 2006, OOPSLA '06.

[61]  Masood Fooladi,et al.  A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases , 2013, ArXiv.

[62]  Loet Leydesdorff,et al.  Do Scientific Advancements Lean on the Shoulders of Giants? A Bibliometric Investigation of the Ortega Hypothesis , 2010, PloS one.