Best Practices in Data Collection and Preparation: Recommendations for Reviewers, Editors, and Authors

We offer best-practice recommendations for journal reviewers, editors, and authors regarding data collection and preparation. Our recommendations are applicable to research adopting different epistemological and ontological perspectives—including both quantitative and qualitative approaches—as well as research addressing micro (i.e., individuals, teams) and macro (i.e., organizations, industries) levels of analysis. Our recommendations regarding data collection address (a) type of research design, (b) control variables, (c) sampling procedures, and (d) missing data management. Our recommendations regarding data preparation address (e) outlier management, (f) use of corrections for statistical and methodological artifacts, and (g) data transformations. Our recommendations address best practices as well as transparency issues. The formal implementation of our recommendations in the manuscript review process will likely motivate authors to increase transparency because failure to disclose necessary information may lead to a manuscript rejection decision. Also, reviewers can use our recommendations for developmental purposes to highlight which particular issues should be improved in a revised version of a manuscript and in future research. Taken together, the implementation of our recommendations in the form of checklists can help address current challenges regarding results and inferential reproducibility as well as enhance the credibility, trustworthiness, and usefulness of the scholarly knowledge that is produced.

[1]  Evan Mayo-Wilson,et al.  Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. , 2018, The American psychologist.

[2]  A. Tsui.  The Spirit of Science and Socially Responsible Scholarship , 2013, Management and Organization Review.

[3]  James A Grand,et al.  From Outcome to Process Focus: Fostering a More Robust Psychological Science Through Registered Reports and Results-Blind Reviewing , 2018, Perspectives on psychological science : a journal of the Association for Psychological Science.

[4]  Peter A. Bamberger,et al.  The Effects of Team Reflexivity on Psychological Well-Being in Manufacturing Teams , 2017, The Journal of applied psychology.

[5]  Chad H. Van Iddekinge,et al.  A Meta-Analysis of the Interactive, Additive, and Relative Effects of Cognitive Ability and Motivation on Performance , 2018 .

[6]  Herman Aguinis,et al.  What You See is What You Get? Enhancing Methodological Transparency in Management Research , 2017 .

[7]  P. Bliese.  Editorial , 2018 .

[8]  R. Dodhia.  A Review of Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.) , 2005 .

[9]  Brayden G. King,et al.  Keeping up Appearances , 2013 .

[10]  E. Byington,et al.  Solutions to the Credibility Crisis in Management Science , 2017 .

[11]  Christine A. Henle,et al.  Measurement in Work and Organizational Psychology , 2001 .

[12]  W. Schulze,et al.  Are Family Firms Good Employers , 2017 .

[13]  Joel A. C. Baum,et al.  Reflections on Scientific Misconduct in Management: Unfortunate Incidents or a Normative Crisis? , 2017, Academy of Management Perspectives.

[14]  Charles Teddlie,et al.  Mixed Methods Sampling A Typology With Examples , 2016 .

[15]  Thomas E. Becker,et al.  Nonlinear Transformations in Organizational Research: Possible Problems and Potential Solutions , 2018, Organizational Research Methods.

[16]  Herman Aguinis,et al.  Not All Non-normal Distributions Are Created Equal: Improved Theoretical and Measurement Precision , 2017 .

[17]  R. Mantegna,et al.  Zipf plots and the size distribution of firms , 1995 .

[18]  Herman Aguinis,et al.  Best-Practice Recommendations for Defining, Identifying, and Handling Outliers , 2013 .

[19]  Herman Aguinis,et al.  Not All Nonnormal Distributions Are Created Equal: Improved Theoretical and Measurement Precision , 2017, The Journal of applied psychology.

[20]  N. Butler,et al.  The Gray Zone : Questionable Research Practices in the Business School , 2017 .

[21]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[22]  Herman Aguinis,et al.  Star Performers in Twenty‐First Century Organizations , 2014 .

[23]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[24]  Paul E. Spector,et al.  Methodological Urban Legends: The Misuse of Statistical Control Variables , 2011 .

[25]  David J. Ketchen,et al.  The Use of Archival Proxies in Strategic Management Studies , 2013 .

[26]  Daniel A. Newman,et al.  Missing Data , 2014 .

[27]  Jason L. Huang,et al.  Personality and adaptive performance at work: a meta-analytic investigation. , 2014, The Journal of applied psychology.

[28]  Bill McKelvey,et al.  Power law distributions in entrepreneurship: : Implications for theory and research , 2015 .

[29]  Herman Aguinis,et al.  Transparency and replicability in qualitative research: The case of interviews with elite informants , 2019, Strategic Management Journal.

[30]  Jeffrey R. Edwards,et al.  Statistical control in correlational studies: 10 essential recommendations for organizational researchers. , 2016 .

[31]  Jan-Benedict E. M. Steenkamp,et al.  The Effect of CRM Outsourcing on Shareholder Value: A Contingency Perspective , 2012, Manag. Sci..

[32]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[33]  Cheryl L. Adkins,et al.  Questions About Questionable Research Practices in the Field of Management , 2016 .

[34]  Constance E. Helfat,et al.  Creating repeatable cumulative knowledge in strategic management , 2016 .

[35]  C. Teddlie,et al.  Mixed Methods Sampling , 2007 .

[36]  Jeremy B. Bernerth,et al.  A Critical Review and Best‐Practice Recommendations for Control Variable Usage , 2016 .

[37]  Bennett E. Postlethwaite,et al.  A matter of context: A meta‐analytic investigation of the relative validity of contextualized and noncontextualized personality measures. , 2012 .

[38]  Herman Aguinis,et al.  The First 20 Years of Organizational Research Methods: Trajectory, Impact, and Predictions for the Future , 2018, Organizational Research Methods.

[39]  Richard G. Gardner,et al.  My Family Made Me Do It: A Cross-Domain, Self-Regulatory Perspective on Antecedents to Abusive Supervision , 2016 .

[40]  E. F. Stone-Romero.  Research strategies in industrial and organizational psychology: Nonexperimental, quasi-experimental, and randomized experimental research in special purpose and nonspecial purpose settings. , 2011 .

[41]  J. Reuer,et al.  Geographic Co-location of Partners and Rivals: Implications for the Design of R&D Alliances , 2017, Academy of Management Journal.

[42]  F. Schmidt.  Meta-Analysis , 2008 .

[43]  R. Vandenberg,et al.  Debunking Myths and Urban Legends about How to Identify Infl uential Outliers , 2014 .

[44]  Christine A. Henle,et al.  Green by comparison: Deviant and normative transmutations of job search envy in a temporal context , 2017 .

[45]  J. Creswell,et al.  Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board task force report. , 2018, The American psychologist.

[46]  Katherine C. Kellogg,et al.  The Radical Flank Effect and Cross-occupational Collaboration for Technology Development during a Power Shift , 2016, Administrative science quarterly.

[47]  Herman Aguinis,et al.  First Decade of Organizational Research Methods , 2009 .

[48]  Harbir Singh,et al.  Who Does Private Equity Buy? Evidence on the Role of Private Equity From Buyouts of Divested Businesses , 2017 .

[49]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[50]  Prasad Balkundi,et al.  When managers become leaders: The role of manager network centralities, social power, and followers' perception of leadership , 2017 .

[51]  Herman Aguinis,et al.  Gender Productivity Gap Among Star Performers in STEM and Other Scientific Fields , 2018, The Journal of applied psychology.

[52]  J. Edwards,et al.  Methodological Wishes for the Next Decade and How to Make Wishes Come True , 2014 .

[53]  Herman Aguinis,et al.  Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings , 2017 .

[54]  Paul Montgomery,et al.  Experimental and Quasi-Experimental Designs , 2011 .

[55]  D. Simonton,et al.  Can Super Smart Leaders Suffer From too Much of a Good Thing? The Curvilinear Effect of Intelligence on Perceived Leadership Behavior , 2017, The Journal of applied psychology.

[56]  Robert J. Vandenberg,et al.  An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection , 2014 .

[57]  A. Kristof-brown,et al.  Resolution, Relief, and Resignation: A Qualitative Study of Responses to Misfit at Work , 2017 .

[58]  J. Schafer,et al.  Missing data: our view of the state of the art. , 2002, Psychological methods.

[59]  Dan J. Putka,et al.  The Multifaceted Nature of Measurement Artifacts and Its Implications for Estimating Construct-Level Relationships , 2009 .

[60]  Herman Aguinis,et al.  Conducting Field Experiments Using eLancing’s Natural Environment , 2012, Academy of Management Proceedings.

[61]  Peter Rausch,et al.  Performance Management , 2011, Informatik-Spektrum.