Standards for Experimental Research: Encouraging a Better Understanding of Experimental Methods

In this essay, we closely examine three aspects of the Reporting Guidelines for this journal, as described by Gerber et al. ( 2014 , Journal of Experimental Political Science 1(1): 81–98) in the inaugural issue of the Journal of Experimental Political Science. These include manipulation checks and when the reporting of response rates is appropriate. The third, most critical, issue concerns the committee's recommendations for detecting errors in randomization. This is an area where there is evidence of widespread confusion about experimental methods throughout our major journals. Given that a goal of the Journal of Experimental Political Science is promoting best practices and a better understanding of experimental methods across the discipline, we recommend changes to the Standards that will allow the journal to play a leading role in correcting these misunderstandings.

[1]  S. Senn.  Seven myths of randomisation in clinical trials , 2011, Statistics in medicine.

[2]  M. Hiscox,et al.  Through a Glass and Darkly: Attitudes Toward International Trade and the Curious Effects of Issue Framing , 2006, International Organization.

[3]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, Journal of clinical epidemiology.

[4]  Judd B. Kessler,et al.  Learning from (failed) replications: Cognitive load manipulations and charitable giving , 2014 .

[5]  Neil Malhotra,et al.  Public Preferences for Bipartisanship in the Policymaking Process , 2014 .

[6]  R. Abelson.  Statistics As Principled Argument , 1995 .

[7]  Susan D. Hyde.  Experimenting in Democracy Promotion: International Observers and the 2004 Presidential Elections in Indonesia , 2010, Perspectives on Politics.

[8]  Adam J. Berinsky,et al.  The Indirect Effects of Discredited Stereotypes in Judgments of Jewish Leaders , 2005 .

[9]  T. Permutt.  Testing for imbalance of covariates in controlled experiments. , 1990, Statistics in medicine.

[10]  Kari Lock Morgan,et al.  Rerandomization to improve covariate balance in experiments , 2012, 1207.5625.

[11]  William M. Pollock,et al.  Informed Consent: Roll-Call Knowledge, the Mass Media, and Political Representation , 2011 .

[12]  John O. Summers,et al.  Checking the Success of Manipulations in Marketing Experiments , 1986 .

[13]  David E. Broockman,et al.  Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators , 2011 .

[14]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[15]  A. T. Panter,et al.  The SAGE andbook of methods in social psychology , 2004 .

[16]  Albert D. Cover,et al.  Baby Books and Ballots: The Impact of Congressional Mail on Constituent Opinion , 1982 .

[17]  Please do not cite without the authors ’ permission , 2006 .

[18]  Jane M. Carey,et al.  Human factors in management information systems , 1988 .

[19]  J. I.  The Design of Experiments , 1936, Nature.

[20]  G. Catlin,et al.  The Social Sciences , 1942, Nature.

[21]  J. Ladd.  The Neglected Power of Elite Opinion Leadership to Produce Antipathy Toward the News Media: Evidence from a Survey Experiment , 2010 .

[22]  Nicholas A. Valentino,et al.  Cues that Matter: How Political Ads Prime Racial Attitudes During Campaigns , 2002, American Political Science Review.

[23]  C. R. Barr.  Grass roots politics , 1962 .

[24]  S. Senn.  Testing for baseline balance in clinical trials. , 1994, Statistics in medicine.

[25]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, BMC medicine.

[26]  Adam J. Berinsky,et al.  Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys , 2014 .

[27]  Nicholas A. Valentino,et al.  The Compassion Strategy Race and the Gender Gap in Campaign 2000 , 2004 .

[28]  Andrew M. Penner,et al.  Racial Fluidity and Inequality in the United States1 , 2012, American Journal of Sociology.

[29]  P. Himmel.  Political Science The State Of The Discipline , 2016 .

[30]  Diana C. Mutz,et al.  The Perils of Randomization Checks in the Analysis of Experiments , 2012 .

[31]  Jane Sell,et al.  Laboratory experiments in the social sciences , 2007 .

[32]  Is Money in Politics Harming Trust in Government? Evidence from Two Survey Experiments , 2012 .

[33]  Neil Malhotra,et al.  Bridging Partisan Divisions over Antiterrorism Policies , 2012 .

[34]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[35]  Kevin Arceneaux.  Cognitive Biases and the Strength of Political Arguments , 2012 .

[36]  Gary King,et al.  Misunderstandings between experimentalists and observationalists about causal inference , 2008 .

[37]  Neil Malhotra,et al.  Publication Bias in the Social Sciences , 2014 .

[38]  Ira Katznelson,et al.  Political science : the state of the discipline , 2002 .

[39]  Vincent L. Hutchings.  The Race Card: Campaign Strategy, Implicit Messages, and the Norm of Equality By Tali Mendelberg. Princeton: Princeton University Press, 2001. 320p. $52.50 cloth, $17.95 paper. , 2002, American Political Science Review.

[40]  SchererNancy,et al.  Does Descriptive Race Representation Enhance Institutional Legitimacy? The Case of the U.S. Courts , 2015 .

[41]  M. Prior,et al.  Improving Media Effects Research through Better Measurement of News Exposure , 2009, The Journal of Politics.

[42]  Richard E. Matland,et al.  Doing Rawls Justice: An Experimental Study of Income Distribution Norms , 2003 .

[43]  Kevin Arceneaux,et al.  Reporting Guidelines for Experimental Research: A Report from the Experimental Research Section Standards Committee , 2014, Journal of Experimental Political Science.

[44]  G. Gurtner,et al.  Statistics in medicine. , 2011, Plastic and reconstructive surgery.

[45]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[46]  P. Cozby,et al.  Methods in behavioral research , 1977 .

[47]  Martha Foschi,et al.  Hypotheses, Operationalizations, and Manipulation Checks , 2014 .

[48]  Kevin Arceneaux,et al.  Educating the Least Informed: Group Endorsements in a Grassroots Campaign , 2009 .