How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice

Multiplicative interaction models are widely used in social science to test whether the relationship between an outcome and an independent variable changes with a moderating variable. Current empirical practice overlooks two important problems. First, these models assume a linear interaction effect that changes at a constant rate with the moderator. Second, reliably estimating the conditional effects of the independent variable at all values of the moderator requires sufficient common support. Replicating nearly 50 interaction effects from 22 recent publications in five top political science journals, we find that these core assumptions fail in a majority of cases, suggesting that a large portion of findings across all subfields based on interaction models are modeling artifacts or are at best highly model dependent. We propose simple diagnostics to assess the validity of these assumptions and offer flexible estimation strategies that allow for nonlinear interaction effects and safeguard against excessive extrapolation.

[1]  Jianqing Fan,et al.  Local polynomial kernel regression for generalized linear models and quasi-likelihood functions , 1995 .

[2]  Chad Hazlett,et al.  Kernel Regularized Least Squares: Reducing Misspecification Bias with a Flexible and Interpretable Machine Learning Approach , 2014, Political Analysis.

[3]  Qi Li,et al.  SMOOTH VARYING-COEFFICIENT ESTIMATION AND INFERENCE FOR QUALITATIVE AND QUANTITATIVE DATA , 2010, Econometric Theory.

[4]  Krzysztof J. Pelc.  Why Do Some Countries Get Better WTO Accession Terms Than Others? , 2011, International Organization.

[5]  Nicholas A. Valentino,et al.  Emotional Substrates of White Racial Attitudes , 2012 .

[6]  A. Hicken,et al.  The Personal Vote and the Efficacy of Education Spending , 2008 .

[7]  R. Truex.  The Returns to Office in a “Rubber Stamp” Parliament , 2014, American Political Science Review.

[8]  Margit Tavits.  Policy Positions, Issue Importance, and Party Competition in New Democracies , 2008 .

[9]  Bear F. Braumoeller.  Hypothesis Testing and Multiplicative Interaction Terms , 2004, International Organization.

[10]  Cindy D. Kam,et al.  Modeling and Interpreting Interactive Hypotheses in Regression Analysis , 2007 .

[11]  Money, Reputation, and Incumbency in U.S. House Elections, or Why Marginals Have Become More Expensive , 2013, American Political Science Review.

[12]  W. Lin,et al.  Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique , 2012, 1208.2301.

[13]  C. Bodea,et al.  Price Stability and Central Bank Independence: Discipline, Credibility, and Democratic Institutions , 2014, International Organization.

[14]  Justin Grimmer,et al.  Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods , 2017, Political Analysis.

[15]  Matt Golder,et al.  Rehabilitating Duverger’s Theory , 2006 .

[16]  William D. Berry,et al.  Improving Tests of Theories Positing Interaction , 2012 .

[17]  Luke W. Miratrix,et al.  Adjusting treatment effect estimates by post‐stratification in randomized experiments , 2013 .

[18]  L. Ezrow,et al.  Are Niche Parties Fundamentally Different from Mainstream Parties? The Causes and the Electoral Consequences of Western European Parties' Policy Shifts, 1976-1998 , 2006 .

[19]  G. King,et al.  Improving Quantitative Studies of International Conflict: A Conjecture , 2000, American Political Science Review.

[20]  J. Jaccard,et al.  Interaction effects in multiple regression , 1992 .

[21]  Stefan Wager,et al.  Estimation and Inference of Heterogeneous Treatment Effects using Random Forests , 2015, Journal of the American Statistical Association.

[22]  Somer-TopcuZeynep.  Timely Decisions: The Effects of Past National Elections on Party Policy Change , 2015 .

[23]  J. Sinacore.  Multiple regression: Testing and interpreting interactions , 1993 .

[24]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[25]  L. Keele,et al.  Identification, Inference and Sensitivity Analysis for Causal Mediation Effects , 2010, 1011.1079.

[26]  Kåre Vernby.  Inclusion and Public Policy: Evidence from Sweden’s Introduction of Noncitizen Suffrage , 2013 .

[27]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[28]  John W. Tukey,et al.  Data Analysis and Regression: A Second Course in Statistics , 1977 .

[29]  Zeynep Somer-Topcu.  Timely Decisions: The Effects of Past National Elections on Party Policy Change , 2009, The Journal of Politics.

[30]  James H. Anderson.  Sunshine Works -- Comment On"The Adverse Effects of Sunshine: A Field Experiment on Legislative Transparency in an Authoritarian Assembly" , 2013 .

[31]  D. Carpenter,et al.  When Canvassers Became Activists: Antislavery Petitioning and the Political Mobilization of American Women , 2014, American Political Science Review.

[32]  International Finance and Central Bank Independence: Institutional Diffusion and the Flow and Cost of Capital , 2015 .

[33]  Michaël Aklin,et al.  Political Competition, Path Dependence, and the Strategy of Sustainable Energy Transitions , 2013 .

[34]  S. West,et al.  Multiple Regression: Testing and Interpreting Interactions. , 1994 .

[35]  M. Petersen,et al.  Politics in the Mind's Eye: Imagination as a Link between Social and Political Cognition , 2013, American Political Science Review.

[36]  Edmund J. Malesky,et al.  The Adverse Effects of Sunshine: A Field Experiment on Legislative Transparency in an Authoritarian Assembly , 2012, American Political Science Review.

[37]  Gary King,et al.  The Dangers of Extreme Counterfactuals , 2006, Political Analysis.

[38]  Thomas Brambor,et al.  Understanding Interaction Models: Improving Empirical Analyses , 2006, Political Analysis.

[39]  Douglas L. Miller,et al.  A Practitioner’s Guide to Cluster-Robust Inference , 2015, The Journal of Human Resources.

[40]  L. Huddy,et al.  Expressive Partisanship: Campaign Involvement, Political Emotion, and Partisan Identity , 2015, American Political Science Review.

[41]  Marc Ratkovic,et al.  Estimating treatment effect heterogeneity in randomized program evaluation , 2013, 1305.5682.

[42]  Terrence L. Chapman.  Audience Beliefs and International Organization Legitimacy , 2009, International Organization.

[43]  R. Friedrich.  In Defense of Multiplicative Terms In Multiple Regression Equations , 1982 .

[44]  S. Wood.  Thin plate regression splines , 2003 .

[45]  W. Cleveland,et al.  Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting , 1988 .

[46]  J. Hosking.  L‐Moments: Analysis and Estimation of Distributions Using Linear Combinations of Order Statistics , 1990 .

[47]  David J. Samuels,et al.  Voting in Open Economies , 2007 .

[48]  L. K. Williams.  Unsuccessful Success? Failed No-Confidence Motions, Competence Signals, and Electoral Support , 2011 .

[49]  D. Leiter,et al.  Does the Ideological Dispersion of Parties Mediate the Electoral Impact of Valence? A Cross-National Study of Party Support in Nine Western European Democracies , 2010 .

[50]  Richard Olmsted Truex.  The Returns to Office in a “ Rubber Stamp ” Parliament , 2013 .

[51]  David Lazer,et al.  Who Wants to Deliberate - and Why? , 2009 .