“It’s common and a part of being a content creator”: Understanding How Creators Experience and Cope with Hate and Harassment Online

Content creators—social media personalities with large audiences on platforms like Instagram, TikTok, and YouTube—face a heightened risk of online hate and harassment. We surveyed 135 creators to understand their personal experiences with attacks (including toxic comments, impersonation, stalking, and more), the coping practices they employ, and gaps they experience with existing solutions (such as moderation or reporting). We find that while a majority of creators view audience interactions favorably, nearly every creator could recall at least one incident of hate and harassment, and attacks are a regular occurrence for one in three creators. As a result of hate and harassment, creators report self-censoring their content and leaving platforms. Through their personal stories, their attitudes towards platform-provided tools, and their strategies for coping with attacks and harms, we inform the broader design space for how to better protect people online from hate and harassment.

[1]  Tara Matthews,et al.  SoK: A Framework for Unifying At-Risk User Research , 2021, 2022 IEEE Symposium on Security and Privacy (SP).

[2]  Sunny Consolvo,et al.  Designing Toxic Content Classification for a Diversity of Perspectives , 2021, SOUPS @ USENIX Security Symposium.

[3]  Tara Matthews,et al.  "Why wouldn't someone think of democracy as a target?": Security practices & challenges of people involved with U.S. political campaigns , 2021, USENIX Security Symposium.

[4]  Transgender People’s Technology Needs to Support Health and Transition , 2021, CHI.

[5]  Yubo Kou,et al.  Flag and Flaggability in Automated Moderation: The Case of Reporting Toxic Behavior in an Online Game Community , 2021, CHI.

[6]  “Disadvantaged in the American-dominated Internet”: Sex, Work, and Technology , 2021, CHI.

[7]  Nicola Dell,et al.  A Digital Safety Dilemma: Analysis of Computer-Mediated Computer Security Interventions for Intimate Partner Violence During COVID-19 , 2021, CHI.

[8]  Michael S. Bernstein,et al.  The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality , 2021, CHI.

[9]  Tadayoshi Kohno,et al.  Defensive Technology Use by Political Activists During the Sudanese Revolution , 2021, 2021 IEEE Symposium on Security and Privacy (SP).

[10]  Nicola Dell,et al.  The Role of Computer Security Customer Support in Helping Survivors of Intimate Partner Violence , 2021, USENIX Security Symposium.

[11]  Elissa M. Redmiles,et al.  "It's stressful having all these phones": Investigating Sex Workers' Safety Goals, Risks, and Practices Online , 2021, USENIX Security Symposium.

[12]  Thomas Ristenpart,et al.  The Tools and Tactics Used in Intimate Partner Surveillance: An Analysis of Online Infidelity Forums , 2020, USENIX Security Symposium.

[13]  Ada Lerner,et al.  Privacy and Activism in the Transgender Community , 2020, CHI.

[14]  Sonia Chiasson,et al.  'Lime', 'Open Lock', and 'Blocked': Children's Perception of Colors, Symbols, and Words in Cybersecurity Warnings , 2020, CHI.

[15]  Bruce Schneier,et al.  Privacy threats in intimate relationships , 2020, J. Cybersecur..

[16]  Apu Kapadia,et al.  "I am uncomfortable sharing what I can't see": Privacy Concerns of the Visually Impaired with Camera Based Assistive Applications , 2020, USENIX Security Symposium.

[17]  Donghee Yvette Wohn,et al.  What are Effective Strategies of Handling Harassment on Twitch?: Users' Perspectives , 2019, CSCW Companion.

[18]  Casey Fiesler,et al.  Moderation Challenges in Voice-based Online Communities on Discord , 2019, Proc. ACM Hum. Comput. Interact..

[19]  Nicola Dell,et al.  Computer Security and Privacy in the Interactions Between Victim Service Providers and Human Trafficking Survivors , 2019, USENIX Security Symposium.

[20]  David Nemer,et al.  "They Don't Leave Us Alone Anywhere We Go": Gender and Digital Abuse in South Asia , 2019, CHI.

[21]  Bryan C. Semaan,et al.  Moderation Practices as Emotional Labor in Sustaining Online Communities: The Case of AAPI Identity Work on Reddit , 2019, CHI.

[22]  Helena M. Mentis,et al.  Upside and Downside Risk in Online Security for Older Adults with Mild Cognitive Impairment , 2019, CHI.

[23]  Tara Matthews,et al.  Tough Times at Transitional Homeless Shelters: Considering the Impact of Financial Insecurity on Digital Security and Privacy , 2019, CHI.

[24]  Angelika Strohmayer,et al.  Technologies for Social Justice: Lessons from Sex Workers on the Front Lines , 2019, CHI.

[25]  Jay Chen,et al.  "Everyone Has Some Personal Stuff": Designing to Support Digital Privacy with Shared Mobile Phone Use in Bangladesh , 2019, CHI.

[26]  Donghee Yvette Wohn,et al.  Volunteer Moderators in Twitch Micro Communities: How They Get Involved, the Roles They Play, and the Emotional Labor They Experience , 2019, CHI.

[27]  Jun Zhao,et al.  `I make up a silly name': Understanding Children's Perception of Privacy Risks Online , 2019, CHI.

[28]  Gianluca Stringhini,et al.  "You Know What to Do" , 2018, Proceedings of the ACM on Human-Computer Interaction.

[29]  Jeremy Blackburn,et al.  "You Know What to Do" , 2018, Proc. ACM Hum. Comput. Interact..

[30]  Yang Wang,et al.  Cooperative Privacy and Security: Learning from People with Visual Impairments and Their Allies , 2019, SOUPS @ USENIX Security Symposium.

[31]  A. Bruckman,et al.  Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator , 2019, ACM Trans. Comput. Hum. Interact..

[32]  Nicola Dell,et al.  Clinical Computer Security for Victims of Intimate Partner Violence , 2019, USENIX Security Symposium.

[33]  Serge Egelman,et al.  Privacy and Security Threat Models and Mitigation Strategies of Older Adults , 2019, SOUPS @ USENIX Security Symposium.

[34]  Lucy Vasserman,et al.  Measuring and Mitigating Unintended Bias in Text Classification , 2018, AIES.

[35]  Sarah Myers West.  Censored, suspended, shadowbanned: User interpretations of content moderation on social media platforms , 2018 .

[36]  Priya Kumar,et al.  'I Knew It Was Too Good to Be True" , 2018, Proc. ACM Hum. Comput. Interact..

[37]  Morgan Klaus Scheuerman,et al.  Safe Spaces and Safe Places , 2018, Proc. ACM Hum. Comput. Interact..

[38]  David R. Karger,et al.  Squadbox: A Tool to Combat Email Harassment Using Friendsourced Moderation , 2018, CHI.

[39]  David R. Karger,et al.  Squadbox: A Tool To Combat Online Harassment Using Friendsourced Moderation , 2018, CHI Extended Abstracts.

[40]  Kentaro Toyama,et al.  Keeping a Low Profile?: Technology, Risk and Privacy among Undocumented Immigrants , 2018, CHI.

[41]  Ada Lerner,et al.  Computer Security and Privacy for Refugees in the United States , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[42]  A. Bruckman,et al.  Online Harassment and Content Moderation , 2018 .

[43]  Hongfei Fan,et al.  Computer Supported Cooperative Work and Social Computing , 2018, Communications in Computer and Information Science.

[44]  David Nemer,et al.  "Privacy is not for me, it's for those rich women": Performative Privacy Practices on Mobile Phones by Women in South Asia , 2018, SOUPS @ USENIX Security Symposium.

[45]  J. Golbeck.  Online Harassment , 2020, Human Resource Management International Digest.

[46]  Eric Gilbert,et al.  Online Harassment and Content Moderation: The Case of Blocklists , 2018, ACM Trans. Comput. Hum. Interact..

[47]  S. Hewitt,et al.  2017 , 2017, Les 25 ans de l’OMC: Une rétrospective en photos.

[48]  Cliff Lampe,et al.  Classification and Its Consequences for Online Harassment , 2017, Proc. ACM Hum. Comput. Interact..

[49]  Lynne M. Coventry,et al.  Privacy Considerations when Designing Social Network Systems to Support Successful Ageing , 2017, CHI.

[50]  Tara Matthews,et al.  Stories from Survivors: Privacy & Security Practices when Coping with Intimate Partner Abuse , 2017, CHI.

[51]  Zahra Ashktorab,et al.  Identifying Women's Experiences With and Strategies for Mitigating Negative Effects of Online Harassment , 2017, CSCW.

[52]  Lucas Dixon,et al.  Ex Machina: Personal Attacks Seen at Scale , 2016, WWW.

[53]  Mahdi N. Al-Ameen,et al.  Learning System-assigned Passwords: A Preliminary Study on the People with Learning Disabilities , 2017, SOUPS.

[54]  Elizabeth D. Mynatt,et al.  Characterizations of Online Harassment: Comparing Policies Across Social Media Platforms , 2016, GROUP.

[55]  Franziska Roesner,et al.  Individual versus Organizational Computer Security and Privacy Concerns in Journalism , 2016, Proc. Priv. Enhancing Technol..

[56]  Zahra Ashktorab,et al.  Designing Cyberbullying Mitigation and Prevention Solutions through Participatory Design With Teenagers , 2016, CHI.

[57]  Xuan Zhao,et al.  The Social Media Ecology: User Perceptions, Strategies and Challenges , 2016, CHI.

[58]  Volker Wulf,et al.  ICT Use by Prominent Activists in Republika Srpska , 2016, CHI.

[59]  Kate Crawford,et al.  What is a flag for? Social media reporting tools and the vocabulary of complaint , 2016, New Media Soc..

[60]  Franziska Roesner,et al.  Investigating the Computer Security Practices and Needs of Journalists , 2015, USENIX Security Symposium.

[61]  David J. Crandall,et al.  Privacy Concerns and Behaviors of People with Visual Impairments , 2015, CHI.