Methodological Artifacts in Moderated Multiple Regression and Their Effects on Statistical Power

Monte Carlo simulations were conducted to examine the degree to which the statistical power of moderated multiple regression (MMR) to detect the effects of a dichotomous moderator variable was affected by the main and interactive effects of (a) predictor variable range restriction, (b) total sample size, (c) sample sizes for 2 moderator variablebased subgroups, (d) predictor variable intercorrelation, and (e) magnitude of the moderating effect. Results showed that the main and interactive influences of these variables may have profound effects on power. Thus, future attempts to detect moderating effects with MMR should consider the power implications of both the main and interactive effects of the variables assessed in the present study. Otherwise, even moderating effects of substantial magnitude may go undetected.

[1]  James P. Walsh,et al.  Corporate Raiders and their Disciplinary Role in the Market for Corporate Control , 1993 .

[2]  Wayne F. Cascio,et al.  Statistical Implications of Six Methods of Test Score Use in Personnel Selection , 1991 .

[3]  C. Judd,et al.  Statistical difficulties of detecting interactions and moderator effects. , 1993, Psychological bulletin.

[4]  T. Cleary.  TEST BIAS: PREDICTION OF GRADES OF NEGRO AND WHITE STUDENTS IN INTEGRATED COLLEGES , 1968 .

[5]  R. Sitgreaves.  Psychometric theory (2nd ed.). , 1979 .

[6]  R. Alexander,et al.  Effect of Error Variance Heterogeneity on the Power of Tests for Regression Slope Differences , 1994 .

[7]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[8]  Philip Bobko,et al.  TESTING FOR FAIRNESS WITH A MODERATED MULTIPLE REGRESSION STRATEGY: AN ALTERNATIVE TO DIFFERENTIAL ANALYSIS , 1978 .

[9]  Herman Aguinis.  A Quickbasic Program for Generating Correlated Multivariate Random Normal Scores , 1994 .

[10]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[11]  Edward R. Kemery,et al.  Effects of predictor intercorrelations and reliabilities on moderated multiple regression , 1988 .

[12]  Michael G. Rumsey,et al.  Personnel Selection and Classification , 1994 .

[13]  George W. Bohrnstedt,et al.  The Reliability of Products of Two Random Variables , 1978 .

[14]  M. E. Muller,et al.  A Note on the Generation of Random Normal Deviates , 1958 .

[15]  Jia Lin Xie,et al.  Job Scope and Stress: Can Job Scope Be Too High? , 1995 .

[16]  A. S. Otis.  A Method of Inferring the Change in a Coefficient of Correlation Resulting from a Change in the Heterogeneity of the Group. , 2022 .

[17]  R. Darlington,et al.  Regression and Linear Models , 1990 .

[18]  E. F. Stone-Romero,et al.  Estimating the Power to Detect Dichotomous Moderators with Moderated Multiple Regression , 1994 .

[19]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[20]  L. Cronbach,et al.  Aptitudes and instructional methods: A handbook for research on interactions , 1977 .

[21]  Roger E. Millsap,et al.  Sampling variance in the correlation coefficient under range restriction: A Monte Carlo study. , 1989 .

[22]  S. Zedeck.  Problems with the use of "moderator" variables. , 1971 .

[23]  Steffanie L. Wilk,et al.  Within-group norming and other forms of score adjustment in preemployment testing. , 1994, The American psychologist.

[24]  H. G. Osburn,et al.  Unbiased estimation of sampling variance of correlations. , 1988 .

[25]  E. F. Stone-Romero,et al.  Relative power of moderated multiple regression and the comparison of subgroup correlation coefficients for detecting moderating effects. , 1994 .

[26]  Meni Koslowsky,et al.  DETECTING MODERATORS WITH META‐ANALYSIS: AN EVALUATION AND COMPARISON OF TECHNIQUES , 1993 .

[27]  J. P. Guilford,et al.  Fundamental statistics in psychology and education, 3rd ed. , 1956 .

[28]  K. Murphy.  "When your top choice turns you down: Effect of rejected offers on the utility of selection tests": Correction to Murphy. , 1987 .

[29]  Robert C. MacCallum,et al.  DISTINGUISHING BETWEEN MODERATOR AND QUADRATIC EFFECTS IN MULTIPLE REGRESSION , 1995 .

[30]  S. West,et al.  Multiple Regression: Testing and Interpreting Interactions. , 1994 .

[31]  Herman Aguinis,et al.  Statistical power problems with moderated multiple regression in management research. , 1995 .

[32]  Herman Aguinis,et al.  Sampling variance in the correlation coefficient under indirect range restriction : Implications for validity generalization , 1997 .

[33]  P. Walmsley,et al.  Statistical Method , 1923, Nature.

[34]  John E. Hunter,et al.  FORTY QUESTIONS ABOUT VALIDITY GENERALIZATION AND META‐ANALYSIS , 1985 .

[35]  L. E. Jones,et al.  Analysis of multiplicative combination rules when the causal variables are measured with error. , 1983 .

[36]  J W Hedge,et al.  Personnel selection. , 1997, Annual review of psychology.

[37]  Robert L. Linn,et al.  Range restriction problems in the use of self-selected groups for test validation. , 1968 .

[38]  Brian S. O'Leary,et al.  Sample sizes for specified statistical power in testing for differential validity. , 1980 .

[39]  Donald G. Gardner,et al.  MODERATION BY ORGANIZATION-BASED SELF-ESTEEM OF ROLE CONDITION-EMPLOYEE RESPONSE RELATIONSHIPS , 1993 .

[40]  Karl Pearson,et al.  Mathematical Contributions to the Theory of Evolution. XI. On the Influence of Natural Selection on the Variability and Correlation of Organs , 1903 .

[41]  J. Tedeschi,et al.  The Effect of Credibility on Perceived Power1 , 1993 .

[42]  R. Landis,et al.  ITEM BIAS INDICES BASED ON TOTAL TEST SCORE AND JOB PERFORMANCE ESTIMATES OF ABILITY , 1993 .

[43]  Jose M. Cortina,et al.  THE “BIG FIVE” PERSONALITY FACTORS IN THE IPI AND MMPI: PREDICTORS OF POLICE PERFORMANCE , 1992 .

[44]  Herman Aguinis,et al.  Type II Error Problems in the Use of Moderated Multiple Regression for the Detection of Moderating Effects of Dichotomous Variables , 1994 .

[45]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[46]  M. Evans.  A Monte Carlo study of the effects of correlated method variance in moderated multiple regression analysis , 1985 .

[47]  R. L. Thorndike.  RESEARCH PROBLEMS AND TECHNIQUES , 1947 .

[48]  Cynthia L. Cordes,et al.  A Review and an Integration of Research on Job Burnout , 1993 .

[49]  Karl Pearson,et al.  Mathematical contributions to the theory of evolution. XI.—On the influence of natural selection on the variability and correlation of organs , 1902, Proceedings of the Royal Society of London.

[50]  Herman Aguinis,et al.  A New Procedure for Computing Equivalence Bands in Personnel Selection , 1998 .

[51]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[52]  D. R. Saunders.  Moderator Variables in Prediction , 1956 .

[53]  L. Schmetterer.  On Unbiased Estimation , 1960 .

[54]  Stephen B. Dunbar,et al.  On predicting success in training for men and women: Examples from Marine Corps clerical specialties. , 1988 .

[55]  L. Delbeke.  Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[56]  R. Linn.  PEARSON SELECTION FORMULAS: IMPLICATIONS FOR STUDIES OF PREDICTIVE BIAS AND ESTIMATES OF EDUCATIONAL EFFECTS IN SELECTED SAMPLES , 1983 .

[57]  Generating Correlated Bivariate Random Normal Standard Scores in Quickbasic , 1992 .

[58]  N. Betz,et al.  Gender as a moderator of the relationships of stress and social support to physical health in college students , 1991 .

[59]  Kevin R. Murphy.  When your top choice turns you down: Effect of rejected offers on the utility of selection tests. , 1986 .

[60]  Herman Aguinis,et al.  Improving The Estimation of Moderating Effects by Using Computer-Administered Questionnaires , 1996 .

[61]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[62]  Robert M. Guion,et al.  Personnel assessment, selection, and placement. , 1991 .

[63]  S. Lohr.  Statistics (2nd Ed.) , 1994 .

[64]  Randall P. Settoon,et al.  Investigator characteristics as moderators of personnel selection research: a meta-analysis , 1994 .

[65]  N. Schmitt,et al.  PREDICTION OF TRADES APPRENTICES’PERFORMANCE ON JOB SAMPLE CRITERIA , 1990 .

[66]  L. Cronbach.  Statistical tests for moderator variables: flaws in analyses recently proposed , 1987 .

[67]  J. Guilford.  Fundamental statistics in psychology and education , 1943 .

[68]  L. Sechrest,et al.  Treatment of Aptitude X Treatment interactions. , 1991, Journal of consulting and clinical psychology.

[69]  B. J. Winer.  Fundamental Statistics in Psychology and Education. 5th ed. , 1973 .