Sample Selection in Systematic Literature Reviews of Management Research

Systematic review techniques are about to become the “new normal” in reviews of management research. However, there is not yet much advice on how to organize the sample selection process as part of such reviews. This article addresses this void and analyzes this vital part of systematic reviews in more detail. In particular, it offers a critical review of systematic literature reviews published in the Academy of Management Annals and the International Journal of Management Reviews between 2004 and 2018. Based on this methodological literature review, the article presents issues to consider in the most critical choices during the sample selection process. Furthermore, this review identifies several descriptive features such as the mean number of research items included in systematic reviews, the mean number of databases used, and the mean coverage period of such reviews. These numbers may be used as benchmark figures in future reviews.

[1]  J. O’Mahoney.  Archetypes of Translation: Recommendations for Dialogue , 2016 .

[2]  Ivan Zupic,et al.  Bibliometric Methods in Management and Organization , 2014 .

[3]  Joseph J. Martocchio,et al.  Microcomputer playfulness: development of a measure with workplace implications , 1992 .

[4]  Neal R Haddaway,et al.  Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources , 2020, Research synthesis methods.

[5]  E. Todeva,et al.  Accounting for Research Quality: Research Audits and the Journal Rankings Debate , 2015 .

[6]  Ruth Sealy,et al.  Individual‐Level Foci of Identification at Work: A Systematic Review of the Literature , 2017 .

[7]  Marcus M. Keupp,et al.  The Strategic Management of Innovation: A Systematic Review and Paths for Future Research , 2011, International Journal of Management Reviews.

[8]  Andy Neely,et al.  Adoption of Promising Practices: A Systematic Review of the Evidence , 2004 .

[9]  Isabell M. Welpe,et al.  Employer Branding: A Brand Equity‐Based Literature Review and Research Agenda , 2018 .

[10]  T. Reay,et al.  What's the Evidence on Evidence-Based Management? , 2009 .

[11]  D. Knights,et al.  Pushing the Boundaries of Amnesia and Myopia: A Critical Review of the Literature on Identity in Management and Organization Studies , 2017 .

[12]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[13]  Norbert Steigenberger,et al.  The Challenge of Integration: A Review of the M&A Integration Literature , 2017 .

[14]  O. Boiral,et al.  Adoption and Outcomes of ISO 14001: A Systematic Review , 2018 .

[15]  Catherine L. Wang,et al.  Entrepreneurial Learning: Past Research and Future Challenges , 2014 .

[16]  M. Sampson,et al.  Checking reference lists to find additional studies for systematic reviews. , 2011, The Cochrane database of systematic reviews.

[17]  Sara Saggese,et al.  Evolution of the Debate on Control Enhancing Mechanisms: A Systematic Review and Bibliometric Analysis , 2016 .

[18]  Matthew S. Wood,et al.  Opportunity Evaluation as Future Focused Cognition: Identifying Conceptual Themes and Empirical Trends , 2015 .

[19]  Gerardo A. Okhuysen,et al.  Conceptualizing Emergent States: A Strategy to Advance the Study of Group Dynamics , 2016 .

[20]  G. Kalyanaram,et al.  The Negative Effects of Social Capital in Organizations: A Review and Extension , 2017 .

[21]  Dermot Breslin,et al.  Theorizing Through Literature Reviews: The Miner-Prospector Continuum , 2020, Organizational Research Methods.

[22]  Andreas G. F. Hoepner,et al.  Explicit and Implicit Subject Bias in the ABS Journal Quality Guide , 2012 .

[23]  Matt Vassar,et al.  Database selection in systematic reviews: an insight through clinical neurology , 2017, Health information and libraries journal.

[24]  Davide Ravasi,et al.  How Do I Know Who You Think You are? A Review of Research Methods on Organizational Identity , 2013 .

[25]  M. D. Rond,et al.  A Matter of Life or Death: How Extreme Context Research Matters for Management and Organization Studies , 2017 .

[26]  Scott L. Newbert,et al.  Strategies and Tools for Entrepreneurial Resource Access: A Cross‐Disciplinary Review and Typology , 2016 .

[27]  Phil Johnson,et al.  Evaluative Practices in Qualitative Management Research: A Critical Review , 2018 .

[28]  Anne-Wil Harzing,et al.  Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison , 2015, Scientometrics.

[29]  A. Ordanini,et al.  The Many Moods of Inter-Organizational Imitation: A Critical Review , 2008 .

[30]  Justin Paul,et al.  The art of writing literature review: What do we know and what do we need to know? , 2020 .

[31]  M. Lombard,et al.  Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability , 2002 .

[32]  Julio Pindado,et al.  Family Business Performance from a Governance Perspective: A Review of Empirical Research , 2015 .

[33]  Constance E. Helfat,et al.  Quo Vadis, Dynamic Capabilities? A Content-Analytic Review of the Current State of Knowledge and Recommendations for Future Research , 2018 .

[34]  K. Lukka,et al.  Validation in interpretive management accounting research , 2010 .

[35]  S. Lewis,et al.  Maternity Management in SMEs: A Transdisciplinary Review and Research Agenda , 2018 .

[36]  Katri Kauppi,et al.  Sourcing from Africa: A Systematic Review and a Research Agenda , 2018 .

[37]  David Denyer,et al.  Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool , 2012 .

[38]  D. Seidl,et al.  The New CEO and the Post‐Succession Process: An Integration of Past Research and Future Directions , 2015 .

[39]  P. Bansal,et al.  Partnering Up: Including Managers as Research Partners in Systematic Reviews , 2020, Organizational Research Methods.

[40]  Kimberly D. Elsbach,et al.  The Academy of Management Annals: Looking Back, Looking Forward , 2018 .

[41]  H. Rothstein,et al.  Revealed or Concealed? Transparency of Procedures, Decisions, and Judgment Calls in Meta-Analyses , 2012 .

[42]  O. Jones,et al.  Editorial: The Future of Writing and Reviewing for IJMR , 2014 .

[43]  D. Tranfield,et al.  Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review , 2003 .

[44]  Elena Novelli,et al.  Redirecting research efforts on the diversification-performance linkage: The search for synergy , 2017 .

[45]  R. Whittington,et al.  Strategy-as-Practice: Taking Social Practices Seriously , 2012 .

[46]  S. Maguire,et al.  Green Human Resource Management: A Review and Research Agenda , 2013 .

[47]  B. Green,et al.  Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. , 2006, Journal of chiropractic medicine.

[48]  On the experience of conducting a Systematic Review in Industrial , Work , and Organizational Psychology : Yes , it is worthwhile , 2011 .

[49]  S. Endres,et al.  Towards a Three‐Component Model of Relational Social Constructionist Leadership: A Systematic Review and Critical Interpretive Synthesis , 2017 .

[50]  Saeed Akbar,et al.  Effectiveness of Performance Appraisal: An Integrated Framework , 2015 .

[51]  Wolfgang H. Güttel,et al.  The Dynamic Capability View in Strategic Management: A Bibliometric Review , 2012 .

[52]  Lawrence M. Rudner.  Scoring and Classifying Examinees Using Measurement Decision Theory , 2009 .

[53]  Ajai S. Gaur,et al.  A systematic approach to conducting review studies: An assessment of content analysis in 25years of IB research , 2017 .

[54]  R. Bakker.  Taking Stock of Temporary Organizational Forms : A Systematic Review and Research , 2010 .

[55]  Laura B. Cardinal,et al.  An Aspirational View of Organizational Control Research: Re-invigorating Empirical Work to Better Meet the Challenges of 21st Century Organizations , 2017 .

[56]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[57]  R. Torraco.  Writing Integrative Literature Reviews: Guidelines and Examples , 2005 .

[58]  Mark B. Houston,et al.  Review articles: purpose, process, and structure , 2018 .

[59]  S. Xiao,et al.  A Multidisciplinary Cognitive Behavioural Framework of Impulse Buying: A Systematic Review of the Literature , 2013 .

[60]  Deana M. Raffo,et al.  Re-examining systematic literature review in management research: Additional benefits and execution protocols , 2020 .

[61]  M. Linnenluecke,et al.  Resilience in Business and Management Research: A Review of Influential Publications and a Research Agenda , 2017 .

[62]  R. Adams,et al.  Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies , 2017 .

[63]  R. Thorpe,et al.  Using Knowledge within Small and Medium-Sized Firms: A Systematic Review of the Evidence , 2005 .

[64]  Davide Nicolini,et al.  In Search of ‘Managerial Work’: Past, Present and Future of an Analytical Category , 2017 .

[65]  R. Bartholomew,et al.  Science for sale: the rise of predatory journals , 2014, Journal of the Royal Society of Medicine.

[66]  Jackie MacDonald,et al.  Systematic Approaches to a Successful Literature Review , 2014 .

[67]  V. Albino,et al.  Search and Recombination Process to Innovate: A Review of the Empirical Evidence and a Research Agenda , 2017 .

[68]  Jeremy C. Short,et al.  The Art of Writing a Review Article , 2009 .

[69]  R. Piekkari,et al.  Legitimizing Diverse Uses for Qualitative Research: A Rhetorical Analysis of Two Management Journals , 2013 .

[70]  M. Cronin,et al.  The Why and How of the Integrative Review , 2020, Organizational Research Methods.

[71]  J. Söderlund,et al.  Liminality in Management and Organization Studies: Process, Position and Place , 2018 .

[72]  Frederik Anseel,et al.  A Self-Motives Perspective on Feedback-Seeking Behavior: Linking Organizational Behavior and Social Psychology Research , 2007 .

[73]  Juliette Wilson,et al.  Expanding the Domain of Festival Research: A Review and Research Agenda , 2017 .

[74]  Alan C. McKinnon,et al.  Starry-eyed II: the logistics journal ranking debate revisited , 2017 .

[75]  Enzo Peruffo,et al.  Knowledge Spillovers from FDI: A Critical Review from the International Business Perspective , 2016 .

[76]  John Dumay,et al.  On the shoulders of giants: undertaking a structured literature review in accounting , 2016 .

[77]  Jeffrey Beall,et al.  Predatory publishers are corrupting open access , 2012, Nature.

[78]  J. Cope,et al.  Entrepreneurship Education , 2007 .

[79]  G. Hopkinson,et al.  Power‐Base Research in Marketing Channels: A Narrative Review , 2014 .

[80]  I. Olkin,et al.  Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement , 1999, The Lancet.

[81]  Jean‐Pascal Gond,et al.  The Performativity of Literature Reviewing: Constituting the Corporate Social Responsibility Literature Through Re-presentation and Intervention , 2020, Organizational Research Methods.

[82]  William C. Barley,et al.  Valuing Multiple Trajectories of Knowledge: A Critical Review and Agenda for Knowledge Management Research , 2018 .

[83]  C. Shepherd,et al.  Revisiting Paradigm(s) in Management Research: A Rhetorical Analysis of the Paradigm Wars , 2013 .

[84]  Heather Skipworth,et al.  Towards a theory of supply chain alignment enablers: a systematic literature review , 2012 .

[85]  Mauricio Marrone,et al.  Conducting systematic literature reviews and bibliometric analyses , 2019, Australian Journal of Management.

[86]  Jill K. Jesson,et al.  Doing Your Literature Review: Traditional and Systematic Techniques , 2011 .

[87]  H. Volberda,et al.  Strategic Renewal: Past Research, Theoretical Tensions and Future Challenges , 2018 .

[88]  Dennis Tourish,et al.  In defiance of folly: Journal rankings, mindless measures and the ABS guide , 2015 .

[89]  T. Greenhalgh,et al.  Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources , 2005, BMJ : British Medical Journal.

[90]  D. Rousseau,et al.  Evidence in Management and Organizational Science: Assembling the Field's Full Weight of Scientific Knowledge through Syntheses , 2008 .

[91]  J. Kleijnen,et al.  How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3) , 2016, Expert review of pharmacoeconomics & outcomes research.

[92]  Herman Aguinis,et al.  Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews , 2020, Organizational Research Methods.

[93]  M. Hitt,et al.  All Things Great and Small: Organizational Size, Boundaries of the Firm, and a Changing Environment , 2015 .

[94]  M. Peltoniemi.  Reviewing Industry Life‐Cycle Theory: Avenues for Future Research , 2011 .

[95]  W. Dullaert,et al.  Consumer Behaviour and Order Fulfilment in Online Retailing: A Systematic Review , 2018 .

[96]  J. Bartunek,et al.  Time in Strategic Change Research , 2017 .

[97]  Justus J. Randolph.  A Guide to Writing the Dissertation Literature Review. , 2009 .

[98]  Mark Gahegan,et al.  Biodiversity data should be published, cited, and peer reviewed. , 2013, Trends in ecology & evolution.

[99]  Eva Niesten,et al.  The Impact of Alliance Management Capabilities on Alliance Attributes and Performance: A Literature Review , 2015 .

[100]  P. Cronin,et al.  Undertaking a literature review: a step-by-step approach. , 2008, British journal of nursing.

[101]  Pamela S. Schindler,et al.  Business research methods 12th ed. , 2014 .

[102]  John Houghton,et al.  Where is the evidence: realising the value of grey literature for public policy and practice , 2014 .

[103]  Douglas G. Altman,et al.  University of Southern Denmark Systematic review adherence to methodological or reporting quality , 2017 .

[104]  Gordon Müller-Seitz.  Leadership in Interorganizational Networks: A Literature Review and Suggestions for Future Research , 2011 .

[105]  Anne Delarue,et al.  Teamworking and Organizational Performance: A Review of Survey-Based Research , 2008 .

[106]  D. Tranfield,et al.  Producing a systematic review. , 2009 .

[107]  Judit Bar-Ilan,et al.  Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar , 2010, Scientometrics.

[108]  Hillary Anger Elfenbein,et al.  More than a Phase: Form and Features of a General Theory of Negotiation , 2018 .

[109]  J. Hulland,et al.  Why systematic review papers and meta-analyses matter: an introduction to the special issue on generalizations in marketing , 2020, Journal of the Academy of Marketing Science.

[110]  G. Radaelli,et al.  Middle Managers and the Translation of New Ideas in Organizations: A Review of Micro‐Practices and Contingencies , 2016 .

[111]  P. Jarzabkowski,et al.  Strategy-as-Practice: A Review and Future Directions for the Field , 2009 .

[112]  J. Froyd,et al.  Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields , 2014 .

[113]  Jan-Folke Siebels,et al.  A Review of Theory in Family Business Research: The Implications for Corporate Governance , 2012 .

[114]  Laura B. Cardinal,et al.  Feature Topic at Organizational Research Methods , 2018 .

[115]  Michael Gusenbauer,et al.  Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases , 2018, Scientometrics.

[116]  R. Delbridge,et al.  Does Japan Still Matter? Past Tendencies and Future Opportunities in the Study of Japanese Firms , 2015 .

[117]  Beth A. Bechky,et al.  Meaning at the Source: The Dynamics of Field Formation in Institutional Research , 2017 .

[118]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[119]  Maren Duvendack,et al.  The benefits and challenges of using systematic reviews in international development research , 2012 .

[120]  O. Jones,et al.  Editorial: Strategies for the Development of International Journal of Management Reviews , 2010 .

[121]  Paul H. J. Hendriks,et al.  Connecting Knowledge to Management: The Case of Academic Research , 2008 .

[122]  Hugh Willmott,et al.  Journal list fetishism and the perversion of scholarship: reactivity and the ABS list , 2011 .

[123]  Herman Aguinis,et al.  What You See is What You Get? Enhancing Methodological Transparency in Management Research , 2017 .

[124]  Torsten M. Pieper,et al.  Family Firm Goals and Their Effects on Strategy, Family and Organization Behavior: A Review and Research Agenda , 2018 .

[125]  Emmanuel B. Raufflet,et al.  Pluralism in Organizations: Learning from Unconventional Forms of Organizations , 2018 .

[126]  A. Neely,et al.  Networking and Innovation: A Systematic Review of the Evidence , 2004 .

[127]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[128]  Allison W. Pearson,et al.  Oh, the Places We’ll Go! Reviewing Past, Present, and Future Possibilities in Family Business Research , 2016 .