An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection

We rely on classic as well as recently published sources to offer a review of theory, research design, and measurement issues that should be considered prior to conducting any empirical study. First, we examine theory-related issues that should be addressed before research design and measurement considerations. Specifically, we discuss how to make meaningful theoretical progress including the use of inductive and deductive approaches, address an important issue, and conduct research with a practical end in mind. Second, we offer recommendations regarding research design, including how to address the low statistical power challenge, design studies that strengthen inferences about causal relationships, and use control variables appropriately. Finally, we address measurement issues. Specifically, we discuss how to improve the link between underlying constructs and their observable indicators. Our review offers a checklist for use by researchers to improve research quality prior to data collection and by jour...

[1]  Joseph Little,et al.  Understanding Statistical Significance: A Conceptual History , 2001 .

[2]  J. Pfeffer.  Barriers to the Advance of Organizational Science: Paradigm Development as a Dependent Variable , 1993 .

[3]  Paul E. Meehl,et al.  High school yearbooks: A reply to Schwarz. , 1971 .

[4]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[5]  Herman Aguinis,et al.  Revival of test bias research in preemployment testing. , 2010, The Journal of applied psychology.

[6]  Paul E. Spector,et al.  Why negative affectivity should not be controlled in job stress research: don't throw out the baby with the bath water , 2000, Journal of Organizational Behavior.

[7]  D. Lykken.  Statistical significance in psychological research. , 1968, Psychological bulletin.

[8]  R. Rosenthal.  Science and Ethics in Conducting, Analyzing, and Reporting Psychological Research , 1994, Psychological science.

[9]  Herman Aguinis,et al.  First Decade of Organizational Research Methods , 2009 .

[10]  Richard P. Bagozzi,et al.  Assessing Construct Validity in Organizational Research , 1991 .

[11]  Herman Aguinis,et al.  Customer-Centric Science: Reporting Significant Research Results With Rigor, Relevance, and Practical Impact in Mind , 2010 .

[12]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[13]  Wayne F. Cascio,et al.  Research in industrial and organizational psychology from 1963 to 2007: changes, choices, and trends. , 2008, The Journal of applied psychology.

[14]  Dan J. Putka,et al.  An Examination of G-Theory Methods for Modeling Multitrait–Multimethod Data , 2012 .

[15]  Dan J. Putka,et al.  The Multifaceted Nature of Measurement Artifacts and Its Implications for Estimating Construct-Level Relationships , 2009 .

[16]  K. Murphy,et al.  Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests, Second Ediction , 1998 .

[17]  Imre Lakatos,et al.  The methodology of scientific research programmes: Falsification and the methodology of scientific research programmes , 1978 .

[18]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[19]  John K. Kruschke,et al.  The Time Has Come : Bayesian Methods for Data Analysis in the Organizational Sciences , 2012 .

[20]  J. Edwards.  Construct validation in organizational behavior research. , 2003 .

[21]  Scott Tonidandel,et al.  The 2011 SIOP Graduate Program Benchmarking Survey Part 3: Curriculum and Competencies: (559272013-007) , 2014 .

[22]  Frank L. Schmidt,et al.  What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. , 1992 .

[23]  James P. Walsh,et al.  Presidential Address: Embracing the Sacred in Our Secular Scholarly World , 2011 .

[24]  D. Hambrick.  THE FIELD OF MANAGEMENT'S DEVOTION TO THEORY: TOO MUCH OF A GOOD THING? , 2007 .

[25]  Mark A. Mone,et al.  THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH , 1996 .

[26]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[27]  Donald D. Bergh,et al.  The Developmental Editor: Assessing and Directing Manuscript Contribution , 2008 .

[28]  David J. Ketchen,et al.  Constructs in Strategic Management , 2013 .

[29]  Murray S. Davis,et al.  That's Interesting! , 1971 .

[30]  Wayne F. Cascio,et al.  OPEN A NEW WINDOW IN RATIONAL RESEARCH PLANNING: ADJUST ALPHA TO MAXIMIZE STATISTICAL POWER , 1983 .

[31]  D. Shepherd,et al.  Inductive Top-Down Theorizing: A Source of New Theories of Organization , 2011 .

[32]  Gerald V. Barrett,et al.  Validity of Personnel Decisions: A Conceptual Analysis of the Inferential and Evidential Bases , 1989 .

[33]  Kevin D. Carlson,et al.  The Illusion of Statistical Control , 2012 .

[34]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[35]  Marvin D. Dunnette,et al.  Blending the science and practice of industrial and organizational psychology: Where are we and where are we going? , 1990 .

[36]  Herman Aguinis,et al.  Best-Practice Recommendations for Estimating Cross-Level Interaction Effects Using Multilevel Modeling , 2013 .

[37]  Irene Pollach,et al.  Taming Textual Data: The Contribution of Corpus Linguistics to Computer-Aided Text Analysis , 2012 .

[38]  William H. Starbuck.  Vita Contemplativa , 2004, Duft der Zeit.

[39]  James C. Beaty,et al.  Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. , 2005, The Journal of applied psychology.

[40]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[41]  James P. Walsh,et al.  EMBRACING THE SACRED IN OUR SECULAR SCHOLARLY WORLD. , 2011 .

[42]  D. R. Dalton,et al.  Research Methodology in Organizational Studies , 1987 .

[43]  Denise M. Rousseau,et al.  A STICKY, LEVERAGING, AND SCALABLE STRATEGY FOR HIGH-QUALITY CONNECTIONS BETWEEN ORGANIZATIONAL PRACTICE AND SCIENCE , 2007 .

[44]  J. Mathieu,et al.  Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. , 2012, The Journal of applied psychology.

[45]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[46]  J. Hollenbeck.  The Role of Editing in Knowledge Development: Consensus Shifting and Consensus Creation , 2008 .

[47]  K. Weick.  Theory Construction as Disciplined Imagination , 1989 .

[48]  Stephen G West,et al.  Doctoral training in statistics, measurement, and methodology in psychology: replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in North America. , 2008, The American psychologist.

[49]  Christopher Marquis,et al.  Prospects for Organizational Theory in the Early 21st Century: Institutional Fields and Mechanisms , 2005 .

[50]  Peter H. Gray,et al.  Pursuing Failure , 2010 .

[51]  M. Tushman,et al.  Research and Relevance: Implications of Pasteur'S Quadrant for Doctoral Programs and Faculty Development , 2007 .

[52]  Herman Aguinis,et al.  Using Market Basket Analysis in Management Research , 2012 .

[53]  Timothy R. Hinkin.  A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires , 1998 .

[54]  Eugene F. Srone-Romero.  Construct validity issues in organizational behavior research. , 2013 .

[55]  Bradley L. Kirkman,et al.  Perceived Causes and Solutions of the Translation Problem in Management Research , 2007 .

[56]  Neal M. Ashkanasy,et al.  Publishing today is more difficult than ever , 2009 .

[57]  P. Meehl.  Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. , 1978 .

[58]  Neil Anderson,et al.  The practitioner‐researcher divide revisited: Strategic‐level bridges and the roles of IWO psychologists , 2007 .

[59]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[60]  K. Popper,et al.  Conjectures and refutations;: The growth of scientific knowledge , 1972 .

[61]  Donald C. Hambrick,et al.  1993 presidential address: What if the academy actually mattered? , 1994 .

[62]  Jeffrey R. Edwards,et al.  Reconsidering Theoretical Progress in Organizational and Management Research , 2010 .

[63]  Herman Aguinis,et al.  Measurement Malaise in Strategic Management Studies , 2013 .

[64]  K. Popper,et al.  Conjectures and refutations;: The growth of scientific knowledge , 1972 .

[65]  A. Georges L. Romme,et al.  A Design Science Approach to Evidence-Based Management , 2012 .

[66]  Marcia J. Simmering,et al.  Control Variable Use and Reporting in Macro and Micro Management Research , 2012 .

[67]  R. Bagozzi,et al.  A General Approach for Representing Constructs in Organizational Research , 1998 .

[68]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[69]  Claudia C. Cogliser,et al.  Construct Validation Using Computer-Aided Text Analysis (CATA) , 2010 .

[70]  Kathleen Bentein,et al.  Commitment profiles and employee turnover , 2013 .

[71]  Jeffrey Pfeffer,et al.  A Modest Proposal: How We might change the Process and Product of Managerial Research , 2007 .

[72]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[73]  R. Hamming,et al.  You and Your Research , 1986 .

[74]  John P. Meyer,et al.  A person-centered approach to the study of commitment , 2013 .

[75]  K. Michele Kacmar,et al.  An Additional Rating Method for Journal Articles in the Field of Management , 2000 .

[76]  Jeff Peterson,et al.  Theory Pruning: Strategies to Reduce Our Dense Theoretical Landscape , 2010 .

[77]  Jeffrey R. Edwards,et al.  To prosper, organizational psychology should … overcome methodological barriers to progress , 2008 .

[78]  G. Tyge Payne,et al.  Using Computer-Aided Text Analysis to Elevate Constructs , 2013 .

[79]  Charles E. Lance,et al.  More Statistical and Methodological Myths and Urban Legends , 2011 .

[80]  Alan E. Kazdin,et al.  Graduate Training in Statistics, Methodology, and Measurement in Psychology: A Survey of PhD Programs in North America , 1990 .

[81]  R. Ruback,et al.  The relevance and irrelevance of psychological research. The example of prison crowding. , 1988, The American psychologist.

[82]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[83]  Kathleen Bentein,et al.  Commitment Profiles and Employee Turnover , 2013 .

[84]  D. Hambrick.  What if the Academy Actually Mattered , 1994 .

[85]  Toby D. Wall,et al.  The Neglected Science and Art of Quasi-Experimentation , 2009 .

[86]  James A. Breaugh,et al.  Rethinking the Control of Nuisance Variables in Theory Testing , 2006 .

[87]  Jeffrey Pfeffer,et al.  Special Issue: Frontiers of Organization Science, Part 1 of 2: Building Organization Theory from First Principles: The Self-Enhancement Motive and Understanding Power and Influence , 2005, Organ. Sci..

[88]  Michael L. Barnett,et al.  (Un)Learning and (Mis)Education Through the Eyes of Bill Starbuck: An Interview with Pandora's Playmate , 2005 .

[89]  E. A. Locke.  The Case for Inductive Theory Building† , 2007 .

[90]  Gerald F. Davis,et al.  Do Theories of Organizations Progress? , 2010 .

[91]  Scott B. MacKenzie,et al.  Are we really measuring what we say we're measuring? Using video techniques to supplement traditional construct validation procedures. , 2013, The Journal of applied psychology.

[92]  Paul E. Spector,et al.  Methodological Urban Legends: The Misuse of Statistical Control Variables , 2011 .

[93]  Herman Aguinis,et al.  The Too-Much-of-a-Good-Thing Effect in Management , 2013 .

[94]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[95]  R. Vandenberg,et al.  The role of change in the relationship between commitment and turnover: a latent growth modeling approach. , 2005, The Journal of applied psychology.

[96]  Herman Aguinis,et al.  Social-Role versus Structural Models of Gender and Influence Use in Organizations , 1998 .

[97]  J. Hunter.  Needed: A Ban on the Significance Test , 1997 .

[98]  Thomas E. Becker.  Potential Problems in the Statistical Control of Variables in Organizational Research: A Qualitative Analysis With Recommendations , 2005 .

[99]  David G. Sirmon,et al.  Competition and Scholarly Productivity in Management: Investigating Changes in Scholarship from 1988 to 2008 , 2010 .

[100]  C. Schriesheim.  Causal Analysis: Assumptions, Models, and Data , 1982 .

[101]  Robert J. Vandenberg,et al.  12 Structural Equation Modeling in Management Research: A Guide for Improved Analysis , 2009 .

[102]  Herman Aguinis,et al.  Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact , 2011 .

[103]  J. Edwards,et al.  The Presence of Something or the Absence of Nothing: Increasing Theoretical Precision in Management Research , 2010 .

[104]  J. Colquitt,et al.  TRENDS IN THEORY BUILDING AND THEORY TESTING: A FIVE-DECADE STUDY OF THE ACADEMY OF MANAGEMENT JOURNAL , 2007 .

[105]  Scott B. MacKenzie,et al.  The dangers of poor construct conceptualization , 2003 .

[106]  Norma Harrison,et al.  The Questions we ask , 2008 .

[107]  Marc Orlitzky,et al.  How Can Significance Tests Be Deinstitutionalized? , 2012 .

[108]  S. Rynes,et al.  What Makes Management Research Interesting, And Why Does It Matter? , 2006 .

[109]  Kenneth G. Brown,et al.  HR professionals' beliefs about effective human resource practices: Correspondence between research and practice. , 2002 .

[110]  Melenie J. Lankau,et al.  Improving Construct Measurement In Management Research: Comments and a Quantitative Approach for Assessing the Theoretical Content Adequacy of Paper-and-Pencil Survey-Type Instruments , 1993 .

[111]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[112]  Gerd Gigerenzer.  We need statistical thinking, not statistical rituals , 1998, Behavioral and Brain Sciences.

[113]  Christopher Marquis,et al.  Special Issue: Frontiers of Organization Science, Part 1 of 2: Prospects for Organization Theory in the Early Twenty-First Century: Institutional Fields and Mechanisms , 2005, Organ. Sci..

[114]  Lillian T. Eby,et al.  Qualitative research: The redheaded stepchild in organizational and social science research? , 2008 .