To prosper, organizational psychology should … overcome methodological barriers to progress

Progress in organizational psychology (OP) research depends on the rigor and quality of the methods we use. This paper identifies ten methodological barriers to progress and offers suggestions for overcoming the barriers, in part or whole. The barriers address how we derive hypotheses from theories, the nature and scope of the questions we pursue in our studies, the ways we address causality, the manner in which we draw samples and measure constructs, and how we conduct statistical tests and draw inferences from our research. The paper concludes with recommendations for integrating research methods into our ongoing development goals as scholars and framing methods as tools that help us achieve shared objectives in our field.

[1]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[2]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[3]  N. Schmitt Method bias: The importance of theory and measurement , 1994 .

[4]  F. Lord Large-Sample Covariance Analysis when the Control Variable is Fallible , 1960 .

[5]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[6]  Eugene Komaroff,et al.  Effect of Simultaneous Violations of Essential /g=t/-Equivalence and Uncorrelated Error on Coefficient /g=a/ , 1997 .

[7]  J. Edwards The Study of Congruence in Organizational Behavior Research: Critique and a Proposed Alternative , 1994 .

[8]  R. Frick Accepting the null hypothesis , 1995, Memory & cognition.

[9]  P. M. Podsakoff,et al.  Self-Reports in Organizational Research: Problems and Prospects , 1986 .

[10]  C. Schriesheim Causal Analysis: Assumptions, Models, and Data , 1982 .

[11]  Joseph A. Cote,et al.  Lack of method variance in self-reported affect and perceptions at work: Reality or artifact? , 1989 .

[12]  J. Scott Armstrong,et al.  Publication Bias against Null Results , 1997 .

[13]  Avraham N. Kluger,et al.  The error of accepting the "theoretical" null hypothesis: the rise, fall, and resurrection of commonsense hypotheses in psychology. , 2001, Psychological bulletin.

[14]  John E. Delery,et al.  Alternative Conceptualizations of the Relationship Between Voluntary Turnover and Organizational Performance , 2005 .

[15]  Jacob Cohen The earth is round (p < .05) , 1994 .

[16]  Paul E. Meehl,et al.  High school yearbooks: A reply to Schwarz. , 1971 .

[17]  D. Rubin,et al.  Contrasts and Effect Sizes in Behavioral Research , 1999 .

[18]  David A. Specht,et al.  On the evaluation of causal models , 1975 .

[19]  Robert C. MacCallum,et al.  Alternative Strategies for Cross-Validation of Covariance Structure Models. , 1994, Multivariate behavioral research.

[20]  R. Rosenthal The file drawer problem and tolerance for null results , 1979 .

[21]  D. Rubin,et al.  Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach , 1999 .

[22]  P. Holland Statistics and Causal Inference , 1985 .

[23]  Donald E. Conlon,et al.  Justice at the millennium: a meta-analytic review of 25 years of organizational justice research. , 2001, The Journal of applied psychology.

[24]  Jason Millman,et al.  Research on human behavior;: A systematic guide to method , 1972 .

[25]  E. R. Harcum,et al.  The highly inappropriate calibrations of statistical significance. , 1989 .

[26]  Raymond Hubbard,et al.  The Spread of Statistical Significance Testing in Psychology , 1997 .

[27]  A. Stone,et al.  The science of self-report. Implications for research and practice , 1999 .

[28]  Ronald D. Snee,et al.  Validation of Regression Models: Methods and Examples , 1977 .

[29]  C. I. Mosier I. Problems and Designs of Cross-Validation 1 , 1951 .

[30]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[31]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[32]  G. Cumming,et al.  A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions , 2001 .

[33]  Bruno D. Zumbo,et al.  Coefficient Alpha as an Estimate of Test Reliability Under Violation of Two Assumptions , 1993 .

[34]  L. Harlow,et al.  What if there were no significance tests , 1997 .

[35]  J. Hull,et al.  Moderator Variables in Personality Research: The Problem of Controlling for Plausible Alternatives , 1992 .

[36]  S. Greenland,et al.  Risk factors, confounding, and the illusion of statistical control. , 2004, Psychosomatic medicine.

[37]  T Raykov,et al.  Scale Reliability, Cronbach's Coefficient Alpha, and Violations of Essential Tau-Equivalence with Fixed Congeneric Components. , 1997, Multivariate behavioral research.

[38]  J. Conway,et al.  Estimation and Uses of the Proportion of Method Variance for Multitrait-Multimethod Data , 1998 .

[39]  Kevin J. Williams,et al.  USING SIGNAL‐CONTINGENT EXPERIENCE SAMPLING METHODOLOGY TO STUDY WORK IN THE FIELD: A DISCUSSION AND ILLUSTRATION EXAMINING TASK PERCEPTIONS AND MOOD , 1993 .

[40]  M. R. Novick,et al.  Coefficient alpha and the reliability of composite measurements. , 1967, Psychometrika.

[41]  Elizabeth L. Haines,et al.  Implicit Measurement of Attitudes, Stereotypes, and Self-Concepts in Organizations , 2006 .

[42]  Lawrence R. James,et al.  The unmeasured variables problem in path analysis. , 1980 .

[43]  Seymour Sudman,et al.  Thinking about Answers: The Application of Cognitive Process to Survey Methodology , 1996 .

[44]  J. Platt Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. , 1964, Science.

[45]  Structural Properties and Psychometric Qualities of Organizational Self-Reports: Field Tests of Connections Predicted by Cognitive Theory , 1996 .

[46]  Richard P. DeShon,et al.  A Cautionary Note on Measurement Error Corrections in Structural Equation Models , 1998 .

[47]  W. Tryon Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: an integrated alternative method of conducting null hypothesis statistical tests. , 2001, Psychological methods.

[48]  J. Hage,et al.  Theories of organizations , 1980 .

[49]  R. Gordon Issues in Multiple Regression , 1968, American Journal of Sociology.

[50]  R. Tourangeau,et al.  Cognitive Processes Underlying Context Effects in Attitude Measurement , 1988 .

[51]  Richard J. Harris Significance Tests Have Their Place , 1997 .

[52]  Hubert M. Blalock,et al.  Theory Construction: From Verbal to Mathematical Formulations. , 1970 .

[53]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[54]  Yoav Ganzach,et al.  Misleading Interaction and Curvilinear Terms , 1997 .

[55]  Thomas D. Taber,et al.  TRIANGULATING JOB ATTITUDES WITH INTERPRETIVE AND POSITIVIST MEASUREMENT METHODS , 2006 .

[56]  Robert P. Abelson,et al.  On the Surprising Longevity of Flogged Horses: Why There Is a Case for the Significance Test , 1997 .

[57]  Lawrence R. James,et al.  Measurement of Personality via Conditional Reasoning , 1998 .

[58]  Christopher Winship,et al.  Sampling Weights and Regression Analysis , 1994 .

[59]  Robert C. MacCallum,et al.  Distinguishing between moderator and quadratic effects in multiple regression. , 1995 .

[60]  D. Bonett,et al.  Comments and recommendations regarding the hypothesis testing controversy , 2007 .

[61]  R. L. Hagen In praise of the null hypothesis statistical test. , 1997 .

[62]  M. Mahoney,et al.  Open exchange and epistemic progress. , 1985 .

[63]  D. Day,et al.  Difference Scores: Rationale, Formulation, and Interpretation , 1994 .

[64]  Edward G. Carmines,et al.  Assessing the Reliability of Linear Composites , 1980 .

[65]  Raymond Hubbard,et al.  Statistical Significance with Comments by Editors of Marketing Journals , 2000 .

[66]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[67]  George A. Marcoulides,et al.  Estimating Variance Components in Generalizability Theory: The Covariance Structure Analysis Approach. Teacher's Corner. , 1996 .

[68]  R. Nickerson,et al.  Null hypothesis significance testing: a review of an old and continuing controversy. , 2000, Psychological methods.

[69]  N. Schwarz,et al.  Thinking About Answers: The Application of Cognitive Processes to Survey Methodology , 1995, Quality of Life Research.

[70]  William P. Dunlap,et al.  Partialling Factor Scores Does Not Control Method Variance: A Reply to Podsakoff and Todor , 1986 .

[71]  Ned Hall,et al.  Causation and counterfactuals , 2004 .

[72]  James A. Breaugh,et al.  Rethinking the Control of Nuisance Variables in Theory Testing , 2006 .

[73]  R. Folger,et al.  When is it Acceptable to Accept a Null Hypothesis: No Way, Jose? , 1998 .

[74]  T. Mitchell,et al.  Building Better Theory: Time and The Specification of When Things Happen , 2001 .

[75]  A. Greenwald,et al.  Measuring individual differences in implicit cognition: the implicit association test. , 1998, Journal of personality and social psychology.

[76]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[77]  J. Pfeffer Barriers to the Advance of Organizational Science: Paradigm Development as a Dependent Variable , 1993 .

[78]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[79]  P. Meehl Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[80]  O. D. Duncan,et al.  Introduction to Structural Equation Models. , 1977 .

[81]  Norbert Schwarz,et al.  Cognitive aspects of survey methodology , 2007 .

[82]  Tenko Raykov,et al.  Bias of Coefficient afor Fixed Congeneric Measures with Correlated Errors , 2001 .

[83]  William H. Glick,et al.  Common Methods Bias: Does Common Methods Variance Really Bias Results? , 1998 .

[84]  M. Csíkszentmihályi,et al.  Experience Sampling Method: Measuring the Quality of Everyday Life , 2006 .

[85]  S. Shiffman,et al.  Ecological momentary assessment. , 2008, Annual review of clinical psychology.

[86]  S. Maxwell,et al.  The proof of the pudding: an illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. , 2000, Psychological methods.

[87]  J. R. Larson,et al.  Research strategies and tactics in industrial and organizational psychology. , 1990 .

[88]  J. Conway,et al.  Method Variance and Method Bias in Industrial and Organizational Psychology , 2008 .

[89]  M. D. Dunnette Handbook of Industrial and Organizational Psychology , 2005 .

[90]  T. Nayak Statistical Significance: Rationale, Validity and Utility , 1997 .

[91]  S. West,et al.  Multiple Regression: Testing and Interpreting Interactions. , 1994 .

[92]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[93]  J. Pearl Causality: Models, Reasoning and Inference , 2000 .

[94]  James M. LeBreton,et al.  Measurement issues associated with conditional reasoning tests: indirect measurement and test faking. , 2007, The Journal of applied psychology.

[95]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[96]  J. Krueger,et al.  Null hypothesis significance testing. On the survival of a flawed method. , 2001, The American psychologist.

[97]  F. Schmidt Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[98]  Terry E. Duncan,et al.  Approaches to Testing Interaction Effects Using Structural Equation Modeling Methodology. , 1998, Multivariate behavioral research.

[99]  Leigh Thompson,et al.  Learning Negotiation Skills: Four Models of Knowledge Creation and Transfer , 2003, Manag. Sci..

[100]  L. J. Williams,et al.  An Alternative Approach to Method Effects by Using Latent-Variable Models: Applications in Organizational Behavior Research , 1994 .

[101]  David A. Harrison,et al.  Cognitive processes in self-report responses: tests of item context effects in work attitude measures. , 1993, The Journal of applied psychology.

[102]  L. Cronbach Coefficient alpha and the internal structure of tests , 1951 .

[103]  Herman Aguinis,et al.  Social-Role versus Structural Models of Gender and Influence Use in Organizations , 1998 .

[104]  H. Harman Modern factor analysis , 1961 .

[105]  Raymond Hubbard,et al.  The Historical Growth of Statistical Significance Testing in Psychology--and Its Future Prospects. , 2000 .

[106]  J. Wagner,et al.  Percept-Percept Inflation in Microorganizational Research: An Investigation of Prevalence and Effect , 1994 .

[107]  D. A. Kenny,et al.  Analysis of the multitrait^multimethod matrix by confirmatory factor analysis , 1992 .

[108]  Gina J. Medsker,et al.  A Review of Current Practices for Evaluating Causal Models in Organizational Behavior and Human Resources Management Research , 1994 .

[109]  N. Schwarz Self-reports: How the questions shape the answers. , 1999 .

[110]  H. Marsh,et al.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models , 1991 .

[111]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[112]  K. Widaman Hierarchically Nested Covariance Structure Models for Multitrait-Multimethod Data , 1985 .

[113]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[114]  K. Popper,et al.  The Logic of Scientific Discovery , 1960 .

[115]  R. Hauser,et al.  The Decomposition of Effects in Path Analysis , 1975 .

[116]  Theresa J. DeMaio,et al.  Cognitive interviewing techniques: In the lab and in the field. , 1996 .

[117]  Edwin A. Locke,et al.  Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez–Latham dispute regarding participation in goal setting. , 1988 .

[118]  R. P. Carver The Case Against Statistical Significance Testing , 1978 .

[119]  Larry J. Williams,et al.  Measurement and nonmeasurement processes with negative affectivity and employee attitudes , 1996 .

[120]  Karl G. Jöreskog,et al.  Interaction and nonlinear modeling: Issues and approaches. , 1998 .

[121]  J. Edwards Construct validation in organizational behavior research. , 2003 .

[122]  Neal Schmitt,et al.  Methodology Review: Analysis of Multitrait-Multimethod Matrices , 1986 .

[123]  George A. Marcoulides,et al.  Interaction and Nonlinear Effects in Structural Equation Modeling , 1998 .

[124]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[125]  Paul E. Spector Method Variance in Organizational Research , 2006 .

[126]  William D. Todor,et al.  Relationships Between Leader Reward and Punishment Behavior and Group Processes and Productivity , 1985 .

[127]  A. Gelman,et al.  The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant , 2006 .

[128]  R. Sitgreaves Psychometric theory (2nd ed.). , 1979 .

[129]  Paul E. Spector Using self‐report questionnaires in OB research: A comment on the use of a controversial method , 1994 .

[130]  Jeffrey R. Edwards,et al.  An Examination Of Competing Versions Of The Person-Environment Fit Approach To Stress , 1996 .

[131]  D. Bakan,et al.  The test of significance in psychological research. , 1966, Psychological bulletin.

[132]  D. Kahneman,et al.  Well-being : the foundations of hedonic psychology , 1999 .

[133]  Kent W. Smith On Estimating the Reliability of Composite Indexes Through Factor Analysis , 1974 .

[134]  K. Jöreskog Statistical analysis of sets of congeneric tests , 1971 .

[135]  Lawrence R. James,et al.  A Tale of Two Methods , 2006 .

[136]  W. Dunlap,et al.  Testing Interaction Effects in LISREL: Examination and Illustration of Available Procedures , 2001 .

[137]  H. Marsh Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[138]  P. Meehl Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. , 1978 .

[139]  R. Kline Principles and practice of structural equation modeling, 2nd ed. , 2005 .

[140]  S. Hershberger,et al.  A Simple Rule for Generating Equivalent Models in Covariance Structure Modeling. , 1990, Multivariate behavioral research.

[141]  W. W. Rozeboom The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[142]  N. Schmitt Uses and abuses of coefficient alpha. , 1996 .

[143]  Steven G. Rogelberg,et al.  Handbook of research methods in industrial and organizational psychology , 2004 .

[144]  R. Millsap Sufficient conditions for rotational uniqueness in the additive MTMM model , 1992 .

[145]  Daniel R. Ilgen,et al.  Laboratory Research: A Question of When, Not If. , 1985 .

[146]  Daniel Kahneman,et al.  Experiences of collaborative research. , 2003, The American psychologist.

[147]  D. Kahneman,et al.  Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration , 2001, Psychological science.

[148]  Thomas E. Becker Potential Problems in the Statistical Control of Variables in Organizational Research: A Qualitative Analysis With Recommendations , 2005 .

[149]  Jack M. Feldman,et al.  Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. , 1988 .

[150]  C Loehlin John,et al.  Latent variable models: an introduction to factor, path, and structural analysis , 1986 .

[151]  Wesley C. Salmon,et al.  Causality and Explanation , 1998 .

[152]  S. Maxwell,et al.  Bias in cross-sectional analyses of longitudinal mediation. , 2007, Psychological methods.

[153]  D. Harrison,et al.  TIES, LEADERS, AND TIME IN TEAMS: STRONG INFERENCE ABOUT NETWORK STRUCTURE'S EFFECTS ON TEAM VIABILITY AND PERFORMANCE , 2006 .

[154]  Peter A. White,et al.  Limitations on verbal reports of internal events: A refutation of Nisbett and Wilson and of Bem. , 1980 .

[155]  R. Frick,et al.  The appropriate use of null hypothesis testing. , 1996 .

[156]  T. Mitchell An Evaluation of the Validity of Correlational Research Conducted in Organizations , 1985 .

[157]  Robert J. House,et al.  Judgment calls in research , 1982 .

[158]  Joel M. Hektner,et al.  Experience sampling method , 2007 .

[159]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[160]  I. Stelzl Changing a Causal Hypothesis without Changing the Fit: some Rules for Generating Equivalent Path Models. , 1986, Multivariate behavioral research.

[161]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[162]  F. Schmidt,et al.  Measurement Error in Psychological Research: Lessons From 26 Research Scenarios , 1996 .

[163]  Larry J. Williams,et al.  Method Variance in Organizational Behavior and Human Resources Research: Effects on Correlations, Path Coefficients, and Hypothesis Testing , 1994 .

[164]  J. Elashoff,et al.  Multiple Regression in Behavioral Research. , 1975 .

[165]  K. Murphy,et al.  PROGRESS IN PSYCHOMETRICS: CAN INDUSTRIAL AND ORGANIZATIONAL PSYCHOLOGY CATCH UP? , 2000 .

[166]  Duane T. Wegener,et al.  The problem of equivalent models in applications of covariance structure analysis. , 1993, Psychological bulletin.

[167]  R. Folger Significance Tests and the Duplicity of Binary Decisions , 1989 .

[168]  S. Presser,et al.  Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context , 1996 .

[169]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[170]  R. Mauro Understanding L.O.V.E. (left out variables error): A method for estimating the effects of omitted variables. , 1990 .

[171]  Howard Wainer,et al.  Drawing inferences from self-selected samples , 1986 .

[172]  N. Bolger,et al.  Diary methods: capturing life as it is lived. , 2003, Annual review of psychology.

[173]  Terry M. Coalter,et al.  Context, Cognition, and Common Method Variance: Psychometric and Verbal Protocol Evidence , 1996 .

[174]  James A. Breaugh,et al.  Effect Size Estimation: Factors to Consider and Mistakes to Avoid , 2003 .

[175]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[176]  P. Meehl Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It , 1990 .

[177]  M. Browne,et al.  Cross-Validation Of Covariance Structures. , 1983, Multivariate behavioral research.

[178]  D. Lykken Statistical significance in psychological research. , 1968, Psychological bulletin.

[179]  David R. Heise,et al.  VALIDITY, INVALIDITY, AND RELIABILITY , 1970 .