In Pursuit of Moderation: Nine Common Errors and Their Solutions

One result of the increasing sophistication and complexity of MIS theory and research is the number of studies hypothesizing and testing for moderation effects. A review of the MIS and broader management literatures suggests researchers investigating moderated relationships often commit one or more errors falling into three broad categories: inappropriate use or interpretation of statistics, misalignment of research design with phenomena of interest, and measurement or scaling issues. Examples of nine common errors are presented. Commission of these errors is expected to yield literatures characterized by mixed results at best, and thoroughly erroneous results at worse. Procedures representing examples of best practice and reporting guidelines are provided to help MIS investigators avoid or minimize these errors.

[1]  M. Bartlett,et al.  The use of transformations. , 1947, Biometrics.

[2]  Peter Weill,et al.  The Relationship Between Investment in Information Technology and Firm Performance: A Study of the Valve Manufacturing Sector , 1992, Inf. Syst. Res..

[3]  Subhash Sharma,et al.  Identification and Analysis of Moderator Variables , 1981 .

[4]  Bartlett Ms.  The use of transformations. , 1947 .

[5]  H. J. Arnold,et al.  A Test of the Validity of the Multiplicative Hypothesis of Expectancy- Valence Theories of Work Motivation , 1981 .

[6]  Detmar W. Straub,et al.  Validation in Information Systems Research: A State-of-the-Art Assessment , 2001, MIS Q..

[7]  E. B. Andersen,et al.  Modern factor analysis , 1961 .

[8]  IgbariaMagid,et al.  Antecedents and consequences of job satisfaction among information center employees , 1993 .

[9]  William H. Bommer,et al.  Searching for a Needle in a Haystack: Trying to Identify the Illusive Moderators of Leadership Behaviors , 1995 .

[10]  Wynne W. Chin,et al.  A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte Carlo Simulation Study and Voice Mail Emotion/Adoption Study , 1996, ICIS.

[11]  Izak Benbasat,et al.  Evaluating the Impact of DSS, Cognitive Effort, and Incentives on Strategy Selection , 1999, Inf. Syst. Res..

[12]  Tor Guimaraes,et al.  Successful Strategies for User Participation in Systems Development , 1997, J. Manag. Inf. Syst..

[13]  L. E. Jones,et al.  Analysis of multiplicative combination rules when the causal variables are measured with error. , 1983 .

[14]  H. J. Arnold,et al.  Testing moderator variable hypotheses: A reply to stone and hollenbeck , 1984 .

[15]  C. Brodsky.  The Discovery of Grounded Theory: Strategies for Qualitative Research , 1968 .

[16]  J. A. Adams,et al.  Psychological bulletin. , 1962, Psychological bulletin.

[17]  Sridhar Narasimhan,et al.  Communication and Coordination in the Virtual Office , 1998, J. Manag. Inf. Syst..

[18]  J. M. Cortina,et al.  Interaction, Nonlinearity, and Multicollinearity: implications for Multiple Regression: , 1993 .

[19]  Christopher A. Higgins,et al.  After-Hours Telecommuting and Work-Family Conflict: A Comparative Analysis , 1992, Inf. Syst. Res..

[20]  Valerie Anderson,et al.  Research Methods in Human Resource Management , 2004 .

[21]  Philip Bobko,et al.  A solution to some dilemmas when testing hypotheses about ordinal interactions. , 1986 .

[22]  Ritu Agarwal,et al.  A Conceptual and Operational Definition of Personal Innovativeness in the Domain of Information Technology , 1998, Inf. Syst. Res..

[23]  R. Landis,et al.  Moderated Multiple Regression Tests are Criterion Specific , 2000 .

[24]  Michael H. Birnbaum,et al.  To resolve Fechner versus Stevens: Settle the dispute concerning “ratios” and “differences” , 1989, Behavioral and Brain Sciences.

[25]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[26]  W DicksonGary,et al.  Methodological issues in experimental IS research , 1985 .

[27]  C. Coulton,et al.  Interaction Effects in Multiple Regression , 1993 .

[28]  Michael H. Birnbaum,et al.  Relationships Among Models of Salary Bias , 1985 .

[29]  N. Schmitt,et al.  The Use Of Analysis Of Covariance Structures To Assess Beta And Gamma Change. , 1982, Multivariate behavioral research.

[30]  Robert L. Mason,et al.  Statistical Principles in Experimental Design , 2003 .

[31]  Viswanath Venkatesh,et al.  Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model , 2000, Inf. Syst. Res..

[32]  A. Schepanski.  THE PREDICTIVE ABILITY CRITERION IN EXPERIMENTAL JUDGMENT RESEARCH IN ACCOUNTING , 1983 .

[33]  Graham K. Rand,et al.  Quantitative Applications in the Social Sciences , 1983 .

[34]  Varun Grover,et al.  The Effect of Service Quality and Partnership on the Outsourcing of Information Systems Functions , 1996, J. Manag. Inf. Syst..

[35]  Frank L. Schmidt,et al.  Implications of a measurement problem for expectancy theory research , 1973 .

[36]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[37]  Tor Guimaraes,et al.  Exploring Differences in Employee Turnover Intentions and Its Determinants Among Telecommuters and Non-Telecommuters , 1999, J. Manag. Inf. Syst..

[38]  Bernard C. Y. Tan,et al.  A Cross-Cultural Study on Escalation of Commitment Behavior in Software Projects , 2000, MIS Q..

[39]  Rajiv D. Banker,et al.  The Moderating Effects of Structure on Volatility and Complexity in Software Enhancement , 2000, Inf. Syst. Res..

[40]  Jane M. Howell,et al.  Influence of Experience on Personal Computer Utilization: Testing a Conceptual Model , 1994, J. Manag. Inf. Syst..

[41]  Larry E. Toothaker,et al.  Multiple Regression: Testing and Interpreting Interactions , 1991 .

[42]  Michael H. Birnbaum,et al.  REPLY TO THE DEVIL'S ADVOCATES: DON'T CONFOUND MODEL TESTING AND MEASUREMENT , 1974 .

[43]  Peter A. Todd,et al.  Assessing IT usage: the role of prior experience , 1995 .

[44]  G. Chow.  Tests of equality between sets of coefficients in two linear regressions (econometrics voi 28 , 1960 .

[45]  Susan J. Harrington,et al.  The Effect of Codes of Ethics and Personal Denial of Responsibility on Computer Abuse Judgments and Intentions , 1996, MIS Q..

[46]  Philip Bobko,et al.  Correlation and Regression: Principles and Applications for Industrial/Organizational Psychology and Management , 1995 .

[47]  Tor Guimaraes,et al.  The Relationship Between User Participation and User Satisfaction: An Investigation of Four Contingency Factors , 1994, MIS Q..

[48]  L. Humphreys,et al.  Assessing spurious "moderator effects": Illustrated substantively with the hypothesized ("synergistic") relation between spatial and mathematical ability. , 1990, Psychological bulletin.

[49]  D. R. Saunders.  Moderator Variables in Prediction , 1956 .

[50]  Ronald E. Rice,et al.  Conceptualizing Effects of Office Information Systems: A Methodology and Application for the Study of Alpha, Beta, and Gamma Changes* , 1990 .

[51]  Magid Igbaria,et al.  Work Experiences, Job Involvement, and Quality of Work Life Among Information Systems Personnel , 1994, MIS Q..

[52]  IgbariaMagid,et al.  The effects of time pressure and completeness of information on decision making , 1998 .

[53]  V. Sambamurthy,et al.  Information Technology Assimilation in Firms: The Influence of Senior Leadership and IT Infrastructures , 1999, Inf. Syst. Res..

[54]  Rajiv Kohli,et al.  Information Technology Payoff in the Health-Care Industry: A Longitudinal Study , 2000, J. Manag. Inf. Syst..

[55]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[56]  Cynthia K. Riemenschneider,et al.  Executive Decisions About Adoption of Information Technology in Small Business: Theory and Empirical Tests , 1997, Inf. Syst. Res..

[57]  P. Bobko,et al.  Moderated regression analysis and Likert scales: too coarse for comfort. , 1992, The Journal of applied psychology.

[58]  Joseph A. Raelin.  Work patterns in the professional life‐cycle* , 1985 .

[59]  Peter Tyrer,et al.  The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability : A Monte Carlo Investigation , 1985 .

[60]  Michael H. Birnbaum,et al.  The devil rides again: Correlation as an index of fit. , 1973 .

[61]  ScienceDirect.  Organizational behavior and human performance , 1984 .

[62]  Salvatore T. March,et al.  Information Systems Research , 2004, IFIP International Federation for Information Processing.

[63]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[64]  Tor Guimaraes,et al.  Antecedents and Consequences of Job Satisfaction among Information Center Employees , 1993, J. Manag. Inf. Syst..

[65]  H. J. Arnold.  Moderator variables: A clarification of conceptual, analytic, and psychometric issues , 1982 .

[66]  Herman Aguinis,et al.  Statistical power problems with moderated multiple regression in management research. , 1995 .

[67]  S. S. Stevens,et al.  Problems and methods of psychophysics. , 1958, Psychological bulletin.

[68]  Viswanath Venkatesh,et al.  Why Don't Men Ever Stop to Ask for Directions? Gender, Social Influence, and Their Role in Technology Acceptance and Usage Behavior , 2000, MIS Q..

[69]  Jacob Cohen.  Partialed products are interactions; partialed powers are curve components. , 1978 .

[70]  Timothy Paul Cronan,et al.  Modeling IT Ethics: A Study in Situational Ethics , 1998, MIS Q..

[71]  W. J. Russell.  The Technique of Measurement , 1974 .

[72]  Andrew B. Whinston,et al.  Discovery and Representation of Causal Relationships in MIS Research: A Methodological Framework , 1997, MIS Q..

[73]  M. Dean,et al.  To Log or Not to Log: Bootstrap as an Alternative to the Parametric Estimation of Moderation Effects in the Presence of Skewed Dependent Variables , 2000 .

[74]  Detmar W. Straub,et al.  Structural Equation Modeling and Regression: Guidelines for Research Practice , 2000, Commun. Assoc. Inf. Syst..

[75]  John A. Wagner,et al.  The organizational tenure — Job involvement relationship: A job‐career experience explanation , 1987 .

[76]  P. Bobko.  Correlation and Regression: Applications for Industrial Organizational Psychology and Management , 2001 .

[77]  Journal of Marketing Research , 1968 .

[78]  V. Vroom.  Work and motivation , 1964 .

[79]  Jong-min Choe,et al.  The Relationships among Performance of Accounting Information Systems, Influence Factors, and Evolution Level of Information Systems , 1996, J. Manag. Inf. Syst..

[80]  Bill C. Hardgrave,et al.  Toward a Contingency Model for Selecting an Information System Prototyping Strategy , 1999, J. Manag. Inf. Syst..

[81]  Neal Schmitt,et al.  Comparison of Three Techniques to Assess Group-Level Beta and Gamma Change , 1984 .

[82]  Wanda J. Orlikowski,et al.  The Problem of Statistical Power in MIS Research , 1989, MIS Q..

[83]  C. Higgins,et al.  Gender differences in work-family conflict. , 1991 .

[84]  L. James,et al.  Mediators, Moderators, and Tests for Mediation. , 1984 .

[85]  Naveed Saleem,et al.  An Empirical Test of the Contingency Approach to User Participation in Information Systems Development , 1996, J. Manag. Inf. Syst..

[86]  Gerardine DeSanctis,et al.  ICIS Paper: Methodological Issues in Experimental IS Research: Experiences and Recommendations , 1985, MIS Q..

[87]  C. Judd,et al.  Statistical difficulties of detecting interactions and moderator effects. , 1993, Psychological bulletin.