An Agenda for Studying Credibility Perceptions of Visual Misinformation

ABSTRACT Today’s political misinformation has increasingly been created and consumed in visual formats, such as photographs, memes, and videos. Despite the ubiquity of visual media and the growing scholarly attention to misinformation, there is a relative dearth of research on visual misinformation. It remains unclear which specific visual formats (e.g., memes, visualizations) and features (e.g., color, human faces) contribute to visual misinformation's influence, either on their own or in combination with non-visual features and heuristics, and through what mechanisms. In response to these gaps, we identify a theoretical framework that explains the persuasive mechanisms and pathways of visual features in lending credibility (e.g., as arguments, heuristics, and attention determinants). We propose a list of relevant visual attributes to credibility perceptions and a research agenda that integrates methods including computational visual analysis, crowdsourced annotations, and experiments to advance our understanding of visual misinformation.

[1]  Yingdan Lu,et al.  Unpacking Multimodal Fact-Checking: Features and Engagement of Fact-Checking Videos on Chinese TikTok (Douyin) , 2023, Social Media + Society.

[2]  M. Kirchhof,et al.  YouTube as a source of misinformation on COVID-19 vaccination: a systematic analysis , 2022, BMJ Global Health.

[3]  Zachary C. Steinert-Threlkeld,et al.  Image as Data: Automated Content Analysis for Visual Presentations of Political Actors and Events , 2022, Computational Communication Research.

[4]  Eugene Cho,et al.  Seeing Is Believing: Is Video Modality More Powerful in Spreading Fake News via Online Messaging Apps? , 2021, J. Comput. Mediat. Commun..

[5]  Dan Mercea,et al.  Guy next door and implausibly attractive young women: The visual frames of social media propaganda , 2021, New Media Soc..

[6]  Ullrich K. H. Ecker,et al.  A comparison of prebunking and debunking interventions for implied versus explicit misinformation. , 2021, British journal of psychology.

[7]  Iman A. Wahle,et al.  Aesthetic preference for art can be predicted from a mixture of low- and high-level visual features , 2021, Nature Human Behaviour.

[8]  Jennifer Pan,et al.  The Pervasive Presence of Chinese Government Content on Douyin Trending Videos , 2021, SSRN Electronic Journal.

[9]  Felix M. Simon,et al.  Beyond (Mis)Representation: Visuals in COVID-19 Misinformation , 2020, The international journal of press/politics.

[10]  Yilang Peng Athec: A Python Library for Computational Aesthetic Analysis of Visual Media in Social Science Research , 2021, Social Science Research Network.

[11]  Yilang Peng What Makes Politicians’ Instagram Posts Popular? Analyzing Social Media Strategies of Candidates and Office Holders with Computer Vision , 2020, The International Journal of Press/Politics.

[12]  Claes H. de Vreese,et al.  Do (Microtargeted) Deepfakes Have Real Effects on Political Attitudes? , 2020, The International Journal of Press/Politics.

[13]  F. Arendt,et al.  Visual Cues to the Hidden Agenda: Investigating the Effects of Ideology-Related Visual Subtle Backdrop Cues in Political Communication , 2020, The International Journal of Press/Politics.

[14]  Jeffrey T. Hancock,et al.  Credibility Perceptions and Detection Accuracy of Fake News Headlines on Social Media: Effects of Truth-Bias and Endorsement Cues , 2020, Communication Research.

[15]  Dean Eckles,et al.  Images and Misinformation in Political Groups: Evidence from WhatsApp in India , 2020, ArXiv.

[16]  Eedan R. Amit-Danhi,et al.  Off the charts: user engagement enhancers in election infographics , 2020, Information, Communication & Society.

[17]  Thomas E. Powell,et al.  A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated via Social Media , 2020, Political Communication.

[18]  Loes Aaldering,et al.  Gender Differences in Political Media Coverage: A Meta-Analysis , 2020, Journal of Communication.

[19]  Andrew Chadwick,et al.  Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News , 2020, Social Media + Society.

[20]  Jungseock Joo,et al.  Understanding the Political Ideology of Legislators from Social Media Images , 2019, ICWSM.

[21]  E. Ert,et al.  What do Airbnb hosts reveal by posting photographs online and how does it affect their perceived trustworthiness? , 2020 .

[22]  Jakob Henke,et al.  How can Journalists Promote News Credibility? Effects of Evidences on Trust and Credibility , 2020, Journalism Practice.

[23]  Ying Xie,et al.  Is a Picture Worth a Thousand Words? An Empirical Study of Image Content and Social Media Engagement , 2019, Journal of Marketing Research.

[24]  Emily K. Vraga,et al.  Testing Logic-based and Humor-based Corrections for Science, Health, and Political Misinformation on Social Media , 2019, Journal of Broadcasting & Electronic Media.

[25]  Maarten W. Bos,et al.  Predicting the Personal Appeal of Marketing Images Using Computational Methods , 2019, Journal of Consumer Psychology.

[26]  Min Xu,et al.  Multi-level region-based Convolutional Neural Network for image emotion classification , 2019, Neurocomputing.

[27]  Cuihua Shen,et al.  Fake images: The effects of source, intermediary, and digital media literacy on contextual assessment of image credibility online , 2018, New Media Soc..

[28]  Yilang Peng,et al.  Same Candidates, Different Faces: Uncovering Media Bias in Visual Portrayals of Presidential Candidates with Computer Vision , 2018, Journal of Communication.

[29]  Q. Lei,et al.  Saturation and brightness modulate the effect of depth on visual working memory. , 2018, Journal of vision.

[30]  Jeeyun Oh,et al.  Harnessing the persuasive potential of data: The combinatory effects of data visualization and interactive narratives on obesity perceptions and policy attitudes , 2018, Telematics Informatics.

[31]  Nan Li,et al.  Communicating data: interactive infographics, scientific data and credibility , 2018, Journal of Science Communication.

[32]  Bartosz W. Wojdynski,et al.  Picture This: The Influence of Emotionally Valenced Images, On Attention, Selection, and Sharing of Social Media News , 2018 .

[33]  K. Jamieson,et al.  Fact-Checking Effectiveness as a Function of Format and Tone: Evaluating FactCheck.org and FlackCheck.org , 2018 .

[34]  Yilang Peng,et al.  Feast for the Eyes: Effects of Food Perceptions and Computer Vision Features on Food Photo Popularity , 2018 .

[35]  Emily K. Vraga,et al.  Using Expert Sources to Correct Health Misinformation in Social Media , 2017 .

[36]  John Axelsson,et al.  Eyelid-openness and mouth curvature influence perceived intelligence beyond attractiveness. , 2016, Journal of experimental psychology. General.

[37]  Thomas E. Powell,et al.  A Clearer Picture: The Contribution of Visuals and Text to Framing Effects , 2015 .

[38]  Hyun Suk Kim Attracting Views and Going Viral: How Message Features and News-Sharing Channels Affect Health News Diffusion. , 2015, The Journal of communication.

[39]  Eric Gilbert,et al.  Why We Filter Our Photos and How It Impacts Engagement , 2015, ICWSM.

[40]  Miriam J. Metzger,et al.  Psychological Approaches to Credibility Assessment Online , 2015 .

[41]  Sid Harris,et al.  The clearer picture , 2014 .

[42]  Oded Nov,et al.  The Persuasive Power of Data Visualization , 2014, IEEE Transactions on Visualization and Computer Graphics.

[43]  Virgílio A. F. Almeida,et al.  The impact of visual attributes on online image diffusion , 2014, WebSci '14.

[44]  Song-Chun Zhu,et al.  Visual Persuasion: Inferring Communicative Intents of Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Robert G. Magee,et al.  Red-Hot and Ice-Cold Web Ads: The Influence of Web Ads’ Warm and Cool Colors on Click-Through Rates , 2014 .

[46]  Edward Vul,et al.  Hierarchical Encoding Makes Individuals in a Group Seem More Attractive , 2014, Psychological science.

[47]  Anthony M. Limperos,et al.  Uses and Grats 2.0: New Gratifications for New Media , 2013 .

[48]  Scott Fortmann-Roe Effects of hue, saturation, and brightness on color preference in social networks: Gender‐based color preference on the social networking site Twitter , 2013 .

[49]  Limor Shifman,et al.  Memes in a Digital World: Reconciling with a Conceptual Troublemaker , 2013, J. Comput. Mediat. Commun..

[50]  Jean-Éric Pelet,et al.  The effect of colors of e-commerce websites on consumer mood, memorization and buying intention , 2012, Eur. J. Inf. Syst..

[51]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[52]  Allan Hanbury,et al.  Affective image classification using features inspired by psychology and art theory , 2010, ACM Multimedia.

[53]  M. Grabe,et al.  Image Bite Politics , 2009 .

[54]  Tsuhan Chen,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < , 2022 .

[55]  M. Grabe,et al.  Image Bite Politics: News and the Visual Framing of Elections , 2009 .

[56]  Miriam J. Metzger Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research , 2007, J. Assoc. Inf. Sci. Technol..

[57]  Chen-Chao Tao,et al.  Conceptualizing Media Stimuli in Experimental Research: Psychological Versus Attribute‐Based Definitions , 2007 .

[58]  A. Manstead,et al.  Temporal Aspects of Facial Displays in Person and Expression Perception: The Effects of Smile Dynamics, Head-tilt, and Gender , 2007 .

[59]  Marie‐Christine Lichtlé,et al.  The effect of an advertisement’s colour on emotions evoked by attitude towards the ad , 2007 .

[60]  S. Sundar The MAIN Model : A Heuristic Approach to Understanding Technology Effects on Credibility , 2007 .

[61]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[62]  N. Camgoz,et al.  Effects of Hue, Saturation, and Brightness: Part 2: Attention. , 2004 .

[63]  Susan E. Morgan,et al.  Associations Between Message Features and Subjective Evaluations of the Sensation Value of Antidrug Public Service Announcements , 2003 .

[64]  Daniel J. O'Keefe,et al.  Message Properties, Mediating States, and Manipulation Checks: Claims, Evidence, and Data Analysis in Experimental Persuasive Message Effects Research , 2003 .

[65]  Jan P. Allebach,et al.  Human vision and electronic imaging , 1996, J. Electronic Imaging.

[66]  Zhexue Huang,et al.  CLUSTERING LARGE DATA SETS WITH MIXED NUMERIC AND CATEGORICAL VALUES , 1997 .

[67]  Paul Messaris,et al.  Visual Persuasion: The Role of Images in Advertising , 1996 .

[68]  John T. Cacioppo,et al.  The Elaboration Likelihood Model of Persuasion , 1986, Advances in Experimental Social Psychology.