How Measurement Error in Content Analysis and Self-Reported Media Use Leads to Minimal Media Effect Findings in Linkage Analyses: A Simulation Study

In the debate on minimal media effects and their causes, methodological concerns about measurement are rarely discussed. We argue that even in state-of-the-art media-effects studies that combine measures of media messages and media use (i.e., linkage analyses), measurement error in both the media content analysis and the media use self-reports will typically lead to severely downward-biased effect estimates. We demonstrate this phenomenon using a large Monte Carlo simulation with varying parameters of the content analysis and the survey study. Results show that measurement error in the content analysis and media use variables does indeed lead to smaller effect estimates, especially when the media messages of interest are relatively rare. We discuss these findings as well as possible remedies and implications for future research.

[1]  N. Schwarz,et al.  Asking Questions About Behavior: Cognition, Communication, and Questionnaire Construction , 2001 .

[2]  William J. McGuire,et al.  The Myth of Massive Media Impact: Savagings and Salvagings , 1986 .

[3]  Anke Wonneberger,et al.  Explaining Response Errors of Self-Reported Frequency and Duration of TV Exposure Through Individual and Contextual Factors , 2017 .

[4]  K. Goldstein,et al.  Measuring Media Exposure and the Effects of Negative Campaign Ads , 1999 .

[5]  Travis N. Ridout,et al.  Does Political Advertising Persuade? , 2007 .

[6]  M. Slater Operationalizing and Analyzing Exposure: The Foundation of Media Effects Research , 2004 .

[7]  J. Kleinnijenhuis Newspaper Complexity and the Knowledge Gap , 1991 .

[8]  Lynn Vavreck,et al.  The Exaggerated Effects of Advertising on Turnout: The Dangers of Self-Reports , 2007 .

[9]  Jochen Peter,et al.  Country Characteristics as Contingent Conditions of Agenda Setting , 2003, Commun. Res..

[10]  Jörg Matthes,et al.  Exposure to Counterattitudinal News Coverage and the Timing of Voting Decisions , 2012, Commun. Res..

[11]  P. Lazarsfeld,et al.  The People's Choice: How the Voter Makes Up His Mind in a Presidential Campaign , 1968 .

[12]  C. D. Vreese,et al.  Discussing the democratic deficit: effects of media and interpersonal communication on satisfaction with EU democracy , 2015 .

[13]  Wouter van Atteveldt,et al.  Party Leaders in the Media and Voting Behavior: Priming Rather Than Learning or Projection , 2015 .

[14]  L. Guest The People's Choice: How the Voter Makes Up His Mind in a Presidential Campaign. , 1946 .

[15]  Rens Vliegenthart,et al.  Matching Theory and Data: Why Combining Media Content with Survey Data Matters , 2015, British Journal of Political Science.

[16]  Hans Mathias Kepplinger,et al.  Framing Scandals: Cognitive and Emotional Media Effects , 2012 .

[17]  M. Prior,et al.  The Immensely Inflated News Audience: Assessing Bias in Self-Reported News Exposure , 2009 .

[18]  Patti M. Valkenburg,et al.  Comm Research—Views from Europe| Five Challenges for the Future of Media-Effects Research , 2013 .

[19]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[20]  David E. Wiley,et al.  The Estimation of Measurement Error in Panel Data , 1970 .

[21]  C. D. Vreese,et al.  Owning the issues of crime and immigration: The relation between immigration and crime news and anti-immigrant voting in 11 countries , 2015 .

[22]  Hans-Bernd Brosius,et al.  OPINION FORMATION IN MEDIATED CONFLICTS AND CRISES: A THEORY OF COGNITIVE–AFFECTIVE MEDIA EFFECTS , 1991 .

[23]  Z. Fazekas,et al.  Media Content and Political Behavior in Observational Research: A Critical Assessment , 2015, British Journal of Political Science.

[24]  J. Zaller The statistical power of election studies to detect media exposure effects in political campaigns , 2002 .

[25]  D. Hillard,et al.  Computer Assisted Topic Classification , 2008 .

[26]  D. R. Heise Separating reliability and stability in test-retest correlation. , 1969 .

[27]  J. Zaller,et al.  WHO GETS THE NEWS? ALTERNATIVE MEASURES OF NEWS RECEPTION AND THEIR IMPLICATIONS FOR RESEARCH , 1993 .

[28]  Claes H. de Vreese,et al.  Who's Afraid of Conflict? The Mobilizing Effect of Conflict Framing in Campaign News , 2014, British Journal of Political Science.

[29]  Chul-joo Lee,et al.  The Reliability and Stability of General Media Exposure Measures , 2008 .

[30]  Michael D. Slater,et al.  Combining Content Analysis and Assessment of Exposure through Self-Report, Spatial, or Temporal Variation in Media Effects Research , 2016 .

[31]  Christian Schemer,et al.  The Influence of News Media on Stereotypic Attitudes Toward Immigrants in a Political Campaign , 2012 .

[32]  C. D. Vreese,et al.  Discussing the Democratic Deficit: Effects of Media and Interpersonal Communication on Satisfaction with Democracy in the European Union , 2015 .

[33]  Richard L. Allen,et al.  The Reliability and Stability of Television Exposure , 1981 .

[34]  C. D. Vreese,et al.  The Effects of Human Interest and Conflict News Frames on the Dynamics of Political Knowledge Gains: Evidence from a Cross‐national Study , 2013 .

[35]  Joseph E. Schwartz,et al.  The Neglected Problem of Measurement Error in Categorical Data , 1985 .

[36]  M Dosemeci,et al.  Does nondifferential misclassification of exposure always bias a true effect toward the null value? , 1990, American journal of epidemiology.

[37]  James W. Hardin,et al.  Measurement Error, GLMs, and Notational Conventions , 2003 .

[38]  B. Avolio,et al.  . , 2005, The Leadership Quarterly.

[39]  Michael Scharkow,et al.  The Accuracy of Self-Reported Internet Use—A Validation Study Using Client Log Data , 2016 .

[40]  Peter Neijens,et al.  Measuring Media Exposure in a Changing Communications Environment , 2016 .

[41]  Kevin Arceneaux The Benefits of Experimental Methods for the Study of Campaign Effects , 2010 .

[42]  Kenneth Benoit,et al.  Treating Words as Data with Error: Uncertainty in Text Statements of Policy Positions , 2009 .

[43]  S. Iyengar The Method is the Message: The Current State of Political Communication Research , 2001 .

[44]  Hajo G. Boomgaarden,et al.  Cynics All Around? The Impact of Election News on Political Cynicism in Comparative Perspective , 2013 .

[45]  M. Prior,et al.  The Challenge of Measuring Media Exposure: Reply to Dilliplane, Goldman, and Mutz , 2013 .

[46]  S. Iyengar Experimental Designs for Political Communication Research: Using New Technology and Online Participant Pools to Overcome the Problem of Generalizability , 2014 .

[47]  Patrick Roessler,et al.  The Individual Agenda-Designing Process , 1999 .

[48]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[49]  Rens Vliegenthart,et al.  Covering the crisis: Media coverage of the economic crisis and citizens’ economic expectations , 2011 .

[50]  Pamela J. Shoemaker,et al.  Exposure to What? Integrating Media Content and Effects Studies , 1990 .

[51]  Larry M. Bartels Messages Received: The Political Impact of Media Exposure , 1993, American Political Science Review.

[52]  Wayne A. Fuller,et al.  Measurement Error Models , 1988 .

[53]  Robert Huckfeldt,et al.  Partisan Cues and the Media: Information Flows in the 1992 Presidential Election , 1998, American Political Science Review.

[54]  Arthur H. Miller,et al.  Type-Set Politics: Impact of Newspapers on Public Confidence * , 1979 .

[55]  Claes H. de Vreese,et al.  News matters: Influences on the vote in the Danish 2000 euro referendum campaign , 2004 .

[56]  S. Iyengar,et al.  New perspectives and evidence on political communication and campaign effects. , 2000, Annual review of psychology.

[57]  K. Goldstein,et al.  Lessons Learned: Campaign Advertising in the 2000 Elections , 2002 .

[58]  Diana C. Mutz,et al.  Televised Exposure to Politics: New Measures for a Fragmented Media Environment , 2013 .

[59]  J. R. Cook,et al.  Simulation-Extrapolation Estimation in Parametric Measurement Error Models , 1994 .

[60]  Claes H. de Vreese,et al.  The Effects of Strategic News on Political Cynicism and Vote Choice Among Young Voters , 2008 .

[61]  Diana C. Mutz,et al.  All Virtue Is Relative: A Response to Prior , 2013 .

[62]  Mustafa Dosemeci,et al.  RE: “DOES NONDIFFERENTIAL MISCLASSIFICATION OF EXPOSURE ALWAYS BIAS A TRUE EFFECT TOWARD THE NULL VALUE?” , 1991 .

[63]  Diana C. Mutz Mass Media and the Depoliticization of Personal Experience , 1992 .

[64]  Hans-Jürgen Weiss,et al.  Structural Equation Models for the Analysis of the Agenda-Setting Process , 1989 .

[65]  S. Chaffee,et al.  Measurement and Effects of Attention to Media News , 1986 .

[66]  J. Zaller,et al.  Evaluation of Media Exposure Items In the 1989 NES Pilot Study , 1999 .

[67]  Lynn Vavreck,et al.  Improving Media Measurement: Evidence From the Field , 2014 .

[68]  Claes H. de Vreese,et al.  The Impact of Media Coverage and Motivation on Performance-Relevant Information , 2013 .

[69]  W. Bennett,et al.  A New Era of Minimal Effects? The Changing Foundations of Political Communication , 2008 .

[70]  Arthur H. Miller,et al.  Front-Page News and Real-World Cues: A New Look at Agenda-Setting by the Media , 1980 .

[71]  C. H. de Vreese,et al.  Europhile media and Eurosceptic voting: effects of news media coverage on Eurosceptic voting in the 2009 European Parliamentary elections , 2014 .

[72]  Joseph N. Cappella,et al.  A Comparative Analysis of the Performance of Alternative Measures of Exposure , 2008 .