Factors Affecting the Scientific Impact of Literature Reviews: A Scientometric Study

Standalone literature reviews are fundamental in every scientific discipline. Their value is reflected by a profound scientific impact in terms of citations. Although previous empirical research has shown that this impact has a large variance, it is largely unknown which specific factors influence the impact of literature reviews. Against this background, the purpose of our study is to shed light on the driving factors that make a difference in the scientific impact of literature reviews. Our analysis of an exhaustive set of 214 IS literature reviews reveals that factors on the author level (e.g., expertise, collaboration, and conceptual feedback) and on the article level (e.g., methodological rigor) are significant and robust predictors of scientific impact over and above journal level factors. These insights enhance our understanding of what distinguishes highly cited literature reviews. In so doing, our study informs future guidelines on literature reviews and provides insights for prospective authors.

[1]  R. Merton.  Priorities in scientific discovery: A chapter in the sociology of science. , 1957 .

[2]  E Garfield,et al.  "Science Citation Index"--A New Dimension in Indexing. , 1964, Science.

[3]  D J PRICE,et al.  NETWORKS OF SCIENTIFIC PAPERS. , 1965, Science.

[4]  J. R. Cole,et al.  Scientific output and recognition: a study in the operation of the reward system in science. , 1967, American sociological review.

[5]  R. Merton.  The Matthew Effect in Science , 1968, Science.

[6]  Stephen Cole,et al.  MEASURING THE QUALITY OF SOCIOLOGICAL RESEARCH : PROBLEMS IN THE USE OF THE SCIENCE CITATION ZNDEX , 1971 .

[7]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[8]  M. Moravcsik,et al.  Some Results on the Function and Quality of Citations , 1975 .

[9]  H. Inhaber,et al.  Quality of Research and the Nobel Prizes , 1976 .

[10]  Diana Crane,et al.  Invisible colleges. Diffusion of knowledge in scientific communities , 1972, Medical History.

[11]  G. Gilbert.  Referencing as Persuasion , 1977 .

[12]  Renata Tagliacozzo,et al.  SELF‐CITATIONS IN SCIENTIFIC LITERATURE , 1977 .

[13]  Stephen D. Gottfredson,et al.  Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. , 1978 .

[14]  B. Latour,et al.  Laboratory Life: The Construction of Scientific Facts , 1979 .

[15]  Scott Hamilton,et al.  The journal communication system for MIS research , 1982, DATB.

[16]  Scott Hamilton,et al.  Knowledge Utilization Among MIS Researchers , 1982, MIS Q..

[17]  C. D. Hurt.  Conceptual citation differences in science, technology, and social sciences literature , 1987, Inf. Process. Manag..

[18]  H. Cooper.  Organizing knowledge syntheses: A taxonomy of literature reviews , 1988 .

[19]  P. Seglen,et al.  From bad to worse: evaluation by Journal Impact. , 1989, Trends in biochemical sciences.

[20]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[21]  A. Cameron,et al.  Regression-based tests for overdispersion in the Poisson model☆ , 1990 .

[22]  M. Holbrook,et al.  The Intellectual Structure of Consumer Research: A Bibliometric Study of Author Cocitations in the First 15 Years of the Journal of Consumer Research , 1993 .

[23]  P. David,et al.  Toward a new economics of science , 1994 .

[24]  Clyde W. Holsapple,et al.  Business Computing Research Journals: A Normalized Citation Analysis , 1994, J. Manag. Inf. Syst..

[25]  A. F. J. Van Raan,et al.  On determinants of citation scores: a case study in chemical engineering , 1994 .

[26]  J. M. Beyer,et al.  The Review Process and the Fates of Manuscripts Submitted to AMJ , 1995 .

[27]  Jeretta Horn Nord,et al.  MIS research: Journal status assessment and analysis , 1995, Inf. Manag..

[28]  Mark I. H. Hwang,et al.  The use of meta-analysis in MIS research: promises and problems , 1996, DATB.

[29]  Bill C. Hardgrave,et al.  Forums for MIS scholars , 1997, CACM.

[30]  R. Baumeister,et al.  Writing Narrative Literature Reviews , 1997, Review of General Psychology.

[31]  S. Ghoshal,et al.  Social Capital, Intellectual Capital, and the Organizational Advantage , 1998 .

[32]  C. Hart.  Doing a literature review: releasing the social science research imagination. , 1998 .

[33]  A. Fink.  Conducting research literature reviews , 1998 .

[34]  S. Baldi.  Normative versus social constructivist processes in the allocation of citations : A network-analytic model , 1998 .

[35]  H. Cooper.  Synthesizing Research: A Guide for Literature Reviews , 1998 .

[36]  Ian Ayres,et al.  Determinants of Citations to Articles in Elite Law Reviews , 1999 .

[37]  K. Hyland,et al.  Academic Attribution: Citation and the Construction of Disciplinary Knowledge. , 1999 .

[38]  Susan Athey,et al.  An Evaluation of Research Productivity in Academic IT , 2000, Commun. Assoc. Inf. Syst..

[39]  Reyes Gonzalez,et al.  An analysis of research in information systems (1981-1997) , 2000, Inf. Manag..

[40]  Clare Atkins,et al.  Reclaiming Knowledge: A Case for Evidence-Based Information Systems , 2000, ECIS.

[41]  Richard T. Watson,et al.  Introducing MISQ Review - A New Department in MIS Quarterly , 2001, MIS Q..

[42]  Nikolaos A. Mylonopoulos,et al.  On site: global perceptions of IS journals , 2001, CACM.

[43]  Dorothy E. Leidner,et al.  Review: Knowledge Management and Knowledge Management Systems: Conceptual Foundations and Research Issues , 2001, MIS Q..

[44]  David N. Laband.  Contribution, attribution and the allocation of intellectual property rights: economics versus agricultural economics , 2002 .

[45]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[46]  Detmar W. Straub,et al.  Measuring Researcher-Production in Information Systems , 2002, J. Assoc. Inf. Syst..

[47]  R Brian Haynes,et al.  BMC Medicine BioMed Central , 2003 .

[48]  Vincent R. Brown,et al.  Enhancing ideational creativity in groups: Lessons from research on brainstorming. , 2003 .

[49]  John Ingham,et al.  Why do people use information technology? A critical review of the technology acceptance model , 2003, Inf. Manag..

[50]  Soongoo Hong,et al.  Quality and Knowledge Contributions of MISQ: A Citation Analysis , 2003, Commun. Assoc. Inf. Syst..

[51]  Gerardine DeSanctis,et al.  The Social Life of Information Systems Research: A Response to Benbasat and Zmud's Call for Returning to the IT Artifact , 2003, J. Assoc. Inf. Syst..

[52]  K. Peffers,et al.  Identifying and Evaluating the Universe of Outlets for Information Systems Research: Ranking the Journals , 2003 .

[53]  Soongoo Hong,et al.  Objective quality ranking of computing journals , 2003, CACM.

[54]  Ephraim R. McLean,et al.  The DeLone and McLean Model of Information Systems Success: A Ten-Year Update , 2003, J. Manag. Inf. Syst..

[55]  Kenneth L. Kraemer,et al.  Review: Information Technology and Organizational Performance: An Integrative Model of IT Business Value , 2004, MIS Q..

[56]  Paul Benjamin Lowry,et al.  Global Journal Prestige and Supporting Disciplines: A Scientometric Study of Information Systems Journals , 2004, J. Assoc. Inf. Syst..

[57]  M. Wade,et al.  Review: the resource-based view and information systems research: review, extension, and suggestions for future research , 2004 .

[58]  Hendrik P. van Dalen,et al.  What makes a scientific article influential? The case of demographers , 2001, Scientometrics.

[59]  Concepción S. Wilson,et al.  The Literature of Bibliometrics, Scientometrics, and Informetrics , 2001, Scientometrics.

[60]  Loet Leydesdorff,et al.  Towards a theory of citation? , 1987, Scientometrics.

[61]  Mark D. Miller,et al.  Examining differences across journal rankings , 2005, CACM.

[62]  S.-M. Huang,et al.  Intelligent Cache Management for Mobile Data Warehouse Systems , 2005, J. Database Manag..

[63]  Susan E. Cozzens,et al.  What do citations count? the rhetoric-first model , 1989, Scientometrics.

[64]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[65]  William R. King,et al.  Understanding the Role and Methods of Meta-Analysis in IS Research , 2005, Commun. Assoc. Inf. Syst..

[66]  Hariolf Grupp,et al.  The scientometric weight of 50 nations in 27 science areas, 1989–1993. Part I. All fields combined, mathematics, engineering, chemistry and physics , 2005, Scientometrics.

[67]  Benjamin F. Jones.  The Burden of Knowledge and the &Apos;Death of the Renaissance Man&Apos;: Is Innovation Getting Harder? , 2005 .

[68]  J. Koricheva,et al.  What determines the citation frequency of ecological papers? , 2005, Trends in ecology & evolution.

[69]  William H. Starbuck,et al.  How Much Better are the Most Prestigious Journals? The Statistics of Academic Publication , 2005, Organ. Sci..

[70]  J. Ioannidis,et al.  Relative Citation Impact of Various Study Designs in the Health Sciences , 2005, JAMA.

[71]  Arie Rip,et al.  Qualitative conditions of scientometrics: The new challenges , 2006, Scientometrics.

[72]  Donald D. Bergh,et al.  Some predictors of SMJ article impact , 2006 .

[73]  Jennifer Jie Xu,et al.  The Social Identity of IS: Analyzing the Collaboration Network of the ICIS Conferences (1980-2005) , 2006, ICIS.

[74]  Julian Birkinshaw,et al.  Scientific Collaboration Results in Higher Citation Rates of Published Articles , 2006, Pharmacotherapy.

[75]  P. Verhoef,et al.  The Quest for Citations: Drivers of Article Impact , 2006 .

[76]  Varun Grover,et al.  A Citation Analysis of the Evolution and State of Information Systems within a Constellation of Reference Disciplines , 2006, J. Assoc. Inf. Syst..

[77]  Michael Wade,et al.  The Value of Scientometric Studies: An Introduction to a Debate on IS as a Reference Discipline , 2006, J. Assoc. Inf. Syst..

[78]  Nittaya Campbell,et al.  Improving The Writing Of Literature Reviews Through A Literature Integration Exercise , 2006 .

[79]  Kalle Lyytinen,et al.  The Legacy of "Power and Politics" in Disciplinary Discourse: A Citation Analysis , 2006, ICIS.

[80]  Anna Sidorova,et al.  Uncovering the Intellectual Core of the Information Systems Discipline , 2008, MIS Q..

[81]  Yair Levy,et al.  A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research , 2006, Informing Sci. Int. J. an Emerg. Transdiscipl..

[82]  E. Garfield.  The history and meaning of the journal impact factor. , 2006, JAMA.

[83]  Jay M. Pasachoff,et al.  Credit where credit is due , 2006 .

[84]  B. Green,et al.  Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. , 2006, Journal of chiropractic medicine.

[85]  Wynne W. Chin,et al.  Understanding frameworks and reviews: a commentary to assist us in moving our field forward by analyzing our past , 2007, DATB.

[86]  Jeppe Nicolaisen,et al.  Citation analysis , 2007, Annu. Rev. Inf. Sci. Technol..

[87]  L. Fleming,et al.  Collaborative Brokerage, Generative Creativity, and Creative Success , 2007 .

[88]  Markus,et al.  Editor's Comments: Looking for a Few Good Concepts...and Theories...for the Information Systems Field , 2007 .

[89]  Claudia Löbbecke,et al.  Research Importance in the Information Systems Field: A Citations Analysis , 2007, ICIS.

[90]  Raquel Benbunan-Fich,et al.  Analyzing IS research productivity: an inclusive approach to global IS scholarship , 2007, Eur. J. Inf. Syst..

[91]  T. Judge,et al.  What Causes a Management Article to be Cited—Article, Author, or Journal? , 2007 .

[92]  John Mingers,et al.  Exploring the dynamics of journal citations: Modelling with s-curves , 2008, J. Oper. Res. Soc..

[93]  Clyde W. Holsapple,et al.  A publication power approach for identifying premier information systems journals , 2008, J. Assoc. Inf. Sci. Technol..

[94]  Leslie P. Willcocks,et al.  The ranking of top IS journals: a perspective from the London School of Economics , 2008, Eur. J. Inf. Syst..

[95]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[96]  Duane P. Truex,et al.  Assessing Scholarly Influence: Using the Hirsch Indices to Reframe the Discourse , 2009, J. Assoc. Inf. Syst..

[97]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[98]  Maria J Grant,et al.  A typology of reviews: an analysis of 14 review types and associated methodologies. , 2009, Health information and libraries journal.

[99]  Chad Anderson,et al.  Journal Quality and Citations: Common Metrics and Considerations about Their Use , 2010 .

[100]  Fang Xu,et al.  The drivers of citations in management science journals , 2010, Eur. J. Oper. Res..

[101]  Raymond Hubbard,et al.  Marketing's “Oscars”: a citation analysis of award‐winning articles , 2010 .

[102]  Claudia Löbbecke,et al.  Engaging scientometrics in Information Systems: Combining the Quantitative and Qualitative , 2010, ICIS.

[103]  Jasjit Singh,et al.  Lone Inventors as Source of Breakthroughs: Myth or Reality? , 2009, Manag. Sci..

[104]  Tadhg Nagle,et al.  Assessing the Theoretical Strength within the Literature Review Process: A Tool for Doctoral Researchers , 2010, DSS.

[105]  Varun Grover,et al.  The Effect of an IS Article's Structure on Its Impact , 2010, Commun. Assoc. Inf. Syst..

[106]  Jörgen Sandberg,et al.  Generating Research Questions Through Problematization , 2011 .

[107]  Pearl Brereton,et al.  Using mapping studies as the basis for further research - A participant-observer case study , 2011, Inf. Softw. Technol..

[108]  M. Alvesson,et al.  Ways of constructing research questions: gap-spotting or problematization? , 2011 .

[109]  Noel A. Card.  Applied Meta-Analysis for Social Science Research , 2011 .

[110]  Ana Marušić,et al.  A Systematic Review of Research on the Meaning, Ethics and Practices of Authorship across Scholarly Disciplines , 2011, PloS one.

[111]  Briony J. Oates,et al.  Evidence-based information systems: a decade later , 2011, ECIS.

[112]  Joon-Oh Park,et al.  The Increasing Dominance of Teams in Production of Knowledge , 2011 .

[113]  Rasoul Zavaraqi,et al.  Scientometrics or science of science: quantitative, qualitative or mixed one , 2012 .

[114]  Heekyung Hellen Kim,et al.  The Effect of Free Access on the Diffusion of Scholarly Ideas , 2012, ICIS.

[115]  Helen M. Edwards,et al.  A Model-Driven Method for the Systematic Literature Review of Qualitative Empirical Research , 2012, ICIS.

[116]  Frantz Rowe,et al.  Toward a richer diversity of genres in information systems research: new categorization and guidelines , 2012, Eur. J. Inf. Syst..

[117]  Alexander Oettl,et al.  Reconceptualizing Stars: Scientist Helpfulness and Peer Performance , 2012, Manag. Sci..

[118]  Carol Stoak Saunders,et al.  Publication Trade-Offs for Junior Scholars in IS: Conjoint Analysis of Preferences for Quality, First Authorship, Collaboration, and Time , 2013, ICIS.

[119]  Erik Brynjolfsson,et al.  Network Analysis for Predicting Academic Impact , 2013, ICIS.

[120]  J. Gans,et al.  Exploring Tradeoffs in the Organization of Scientific Work: Collaboration and Scientific Reward , 2013 .

[121]  Giuseppe Lippi,et al.  Article downloads and citations: is there any relationship? , 2013, Clinica chimica acta; international journal of clinical chemistry.

[122]  Dennis F. Galletta,et al.  Evaluating Journal Quality and the Association for Information Systems Senior Scholars' Journal Basket Via Bibliometric Measures: Do Expert Journal Assessments Add Value? , 2013, MIS Q..

[123]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[124]  E. Burton Swanson,et al.  A Simple Research Impacts Model Applied to the Information Systems Field , 2014, Commun. Assoc. Inf. Syst..

[125]  Suzanne Rivard,et al.  Editor's comments: the ions of theory construction , 2014 .

[126]  Gustaf Nelhans.  Qualitative Scientometrics , 2014 .

[127]  Frantz Rowe,et al.  What literature review is not: diversity, boundaries and recommendations , 2014, Eur. J. Inf. Syst..

[128]  Franz Rothlauf,et al.  Model-Based Decision Support in Manufacturing and Service Networks , 2014, Bus. Inf. Syst. Eng..

[129]  Sebastian K. Boell,et al.  A Hermeneutic Approach for Conducting Literature Reviews and Literature Searches , 2014, Commun. Assoc. Inf. Syst..

[130]  Varun Grover,et al.  What Affects Citation Counts in MIS Research Articles? An Empirical Investigation , 2014, Commun. Assoc. Inf. Syst..

[131]  Carol Tenopir,et al.  The role of trust and authority in the citation behaviour of researchers , 2015, Inf. Res..

[132]  Suzie Allard,et al.  Peer review: still king in the digital age , 2015, Learn. Publ..

[133]  Elena Gorbacheva,et al.  Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support , 2015, Commun. Assoc. Inf. Syst..

[134]  Guy Paré,et al.  A Framework for Guiding and Evaluating Literature Reviews , 2015, Commun. Assoc. Inf. Syst..

[135]  Mike Chiasson,et al.  Avoiding methodological overdose: a declaration for independent ends , 2015, J. Inf. Technol..

[136]  Briony J. Oates,et al.  On systematic reviews for evidence-based practice , 2015, J. Inf. Technol..

[137]  Björn Niehaves,et al.  Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information Systems Research , 2015, Commun. Assoc. Inf. Syst..

[138]  Sebastian K. Boell,et al.  Debating systematic literature reviews (SLR) and their ramifications for IS: a rejoinder to Mike Chiasson, Briony Oates, Ulrike Schultze, and Richard Watson , 2015, J. Inf. Technol..

[139]  Alexander Benlian,et al.  Theory of Knowledge for Literature Reviews: An Epistemological Model, Taxonomy and Empirical Analysis of IS Literature , 2015, ICIS.

[140]  Sebastian K. Boell,et al.  On being ‘systematic’ in literature reviews in IS , 2015, J. Inf. Technol..

[141]  Johann Kranz,et al.  A Framework for Rigorously Identifying Research Gaps in Qualitative Literature Reviews , 2015, ICIS.

[142]  Mary Tate,et al.  Introduction to the Special Issue: The Literature Review in Information Systems , 2015, Commun. Assoc. Inf. Syst..

[143]  Chitu Okoli,et al.  A Guide to Conducting a Standalone Systematic Literature Review , 2015, Commun. Assoc. Inf. Syst..

[144]  Ulrike Schultze,et al.  Skirting SLR’s language trap: reframing the ‘systematic’ vs ‘traditional’ literature review opposition as a continuum , 2015, J. Inf. Technol..

[145]  Guido Schryen,et al.  Writing Qualitative IS Literature Reviews - Guidelines for Synthesis, Interpretation, and Guidance of Research , 2015, Commun. Assoc. Inf. Syst..

[146]  Bernd Carsten Stahl,et al.  Critical Discourse Analysis as a Review Methodology: An Empirical Example , 2015, Commun. Assoc. Inf. Syst..

[147]  Richard T. Watson,et al.  Beyond being systematic in literature reviews in IS , 2015, J. Inf. Technol..

[148]  Guy Paré,et al.  Synthesizing information systems knowledge: A typology of literature reviews , 2015, Inf. Manag..

[149]  Charles J. Fornaciari,et al.  "What Causes a Business and Management Education Article to Be Cited: Article, Author, or Journal?" , 2016 .