Cross-Sample Comparisons and External Validity

Abstract Experimentation is an increasingly popular method among political scientists. While experiments are highly advantageous for creating internally valid conclusions, they are often criticized for being low on external validity. Critical to questions of external validity are the types of subjects who participate in a given experiment, with scholars typically arguing that samples of adults are more externally valid then student samples. Despite the vociferousness of such arguments, these claims have received little empirical treatment. In this paper we empirically test for key differences between student and adult samples by conducting four parallel experiments on each of the three samples commonly used by political scientists. We find that our student and diverse, national adult sample behave consistently and in line with theoretical predictions once relevant moderators are taken into account. The same is not true for our adult convenience sample.

[1]  R. McDermott Experimental Methodology in Political Science , 2002, Political Analysis.

[2]  D. O. Sears College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. , 1986 .

[3]  F. Shayan Experimental Political Science and the Study of Causality: From Nature to the Lab by Rebecca B. Morton and Kenneth C. Williams. Cambridge: Cambridge University Press, 2010. 606pp., £25.99, ISBN 978 0521136488 , 2013 .

[4]  Douglas Rivers,et al.  The 2006 Cooperative Congressional Election Study , 2008 .

[5]  D. Kinder,et al.  Communication and politics in the age of information. , 2003 .

[6]  Henry E. Brady,et al.  The Oxford Handbook of Political Methodology , 2010 .

[7]  Mark R. Joslyn,et al.  Gun Policy, Opinion, Tragedy, and Blame Attribution: The Conditional Influence of Issue Frames , 2001, The Journal of Politics.

[8]  James N. Druckman,et al.  F RAMING T HEORY , 2007 .

[9]  M. Steffens Is the implicit association test immune to faking? , 2004, Experimental psychology.

[10]  J. Geer,et al.  Beyond Negativity: The Effects of Incivility on the Electorate , 2007 .

[11]  K. Mcgraw Cambridge Handbook of Experimental Political Science: Candidate Impressions and Evaluations , 2011 .

[12]  Diana C. Mutz Impersonal influence: Effects of representations of public opinion on political attitudes , 1992 .

[13]  Harold Sigall,et al.  The bogus pipeline: A new paradigm for measuring affect and attitude. , 1971 .

[14]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[15]  John H. Aldrich,et al.  Treatment Spillover Effects across Survey Experiments , 2009, Political Analysis.

[16]  M. H. Ensom,et al.  Post Hoc Power Analysis: An Idea Whose Time Has Passed? , 2001, Pharmacotherapy.

[17]  P. Henry College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice , 2008 .

[18]  David Strömberg Media and Politics , 2015 .

[19]  Jennifer Jerit,et al.  Are Survey Experiments Externally Valid? , 2010, American Political Science Review.

[20]  G. Huber,et al.  Assessing the Programmatic Equivalence Assumption in Question Wording Experiments Understanding Why Americans Like Assistance to the Poor More Than Welfare , 2013 .

[21]  James A. Stimson,et al.  Laboratory Experiments in Political Science , 2009 .

[22]  Brian J. Gaines,et al.  The Logic of the Survey Experiment Reexamined , 2007, Political Analysis.

[23]  Diana C. Mutz Population-Based Survey Experiments , 2011 .

[24]  Thomas R. Palfrey,et al.  Experimental foundations of political science , 1993 .

[25]  D. Green,et al.  Field Experiments and Natural Experiments , 2008 .

[26]  Thomas J. Leeper,et al.  A Source of Bias in Public Opinion Stability , 2012, American Political Science Review.

[27]  R. Mcdermott,et al.  Cambridge Handbook of Experimental Political Science: Internal and External Validity , 2011 .

[28]  Jon A. Krosnick,et al.  News Media Impact on the Ingredients of Presidential Evaluations: Politically Knowledgeable Citizens Are Guided by a Trusted Source , 2000 .

[29]  Philip Sedgwick,et al.  Internal and external validity , 2010, BMJ : British Medical Journal.

[30]  A. Aragón,et al.  Participants' Understanding of the Process of Psychological Research: Debriefing , 2000, Ethics & behavior.

[31]  Gary King,et al.  Improving Anchoring Vignettes Designing Surveys to Correct Interpersonal Incomparability , 2010 .

[32]  Norbert Schwarz,et al.  Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments , 1985 .

[33]  Jennifer Jerit How Predictive Appeals Affect Policy Opinions , 2009 .

[34]  D. O. Sears,et al.  The Oxford handbook of political psychology , 2013 .

[35]  Neal J. Roese,et al.  Twenty years of bogus pipeline research : a critical review and meta-analysis , 1993 .

[36]  Bobby J. Calder,et al.  Demand characteristics and three conceptions of the frequently deceived subject. , 1970 .

[37]  Adam J. Berinsky,et al.  Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk , 2012, Political Analysis.

[38]  Henry E. Brady Contributions of Survey Research to Political Science* , 2000, PS: Political Science & Politics.

[39]  Nicholas A. Valentino,et al.  Emotional Substrates of White Racial Attitudes , 2012 .

[40]  Cindy D. Kam,et al.  Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental Research , 2007 .

[41]  D KamCindy,et al.  Risk Orientations and Policy Frames , 2015 .

[42]  Thomas Ostrom The bogus pipeline: A new ignis fatuus? , 1973 .

[43]  H. Sigall,et al.  Reducing attenuation in the expression of interpersonal affect via the bogus pipeline. , 1972, Sociometry.

[44]  Thomas E. Nelson,et al.  Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance , 1997, American Political Science Review.

[45]  Charles S. Taber,et al.  Motivated Skepticism in the Evaluation of Political Beliefs , 2006 .

[46]  Scott Sigmund Gartner,et al.  The Multiple Effects of Casualties on Public Support for War: An Experimental Approach , 2008, American Political Science Review.

[47]  Cindy D. Kam Implicit Attitudes, Explicit Choices: When Subliminal Priming Predicts Candidate Preference , 2007 .

[48]  A. Gerber,et al.  Citizens’ Policy Confidence and Electoral Punishment: A Neglected Dimension of Electoral Accountability , 2011 .

[49]  N. Stanton,et al.  Faking personality questionnaires in personnel selection , 2001 .

[50]  Kevin Arceneaux Cognitive Biases and the Strength of Political Arguments , 2012 .

[51]  James N. Druckman,et al.  Framing Public Opinion in Competitive Democracies , 2007, American Political Science Review.

[52]  Thomas J. Leeper,et al.  Learning More from Political Communication Experiments: Pretreatment and Its Effects , 2012 .

[53]  Cindy D. Kam,et al.  Risk Orientations and Policy Frames , 2010, The Journal of Politics.

[54]  L. Huddy,et al.  American Patriotism, National Identity, and Political Involvement , 2007 .