Supporting Comment Moderators in Identifying High Quality Online News Comments

Online comments submitted by readers of news articles can provide valuable feedback and critique, personal views and perspectives, and opportunities for discussion. The varying quality of these comments necessitates that publishers remove the low quality ones, but there is also a growing awareness that by identifying and highlighting high quality contributions this can promote the general quality of the community. In this paper we take a user-centered design approach towards developing a system, CommentIQ, which supports comment moderators in interactively identifying high quality comments using a combination of comment analytic scores as well as visualizations and flexible UI components. We evaluated this system with professional comment moderators working at local and national news outlets and provide insights into the utility and appropriateness of features for journalistic tasks, as well as how the system may enable or transform journalistic practices around online comments.

[1]  W. A. Shewhart,et al.  Quality control , 1927 .

[2]  Dominique Brossard,et al.  The "Nasty Effect: " Online Incivility and Risk Perceptions of Emerging Technologies , 2014, J. Comput. Mediat. Commun..

[3]  Brink van der Merwe,et al.  Comment classification for an online news domain , 2014 .

[4]  Stephen A. Rains,et al.  Online and Uncivil? Patterns and Determinants of Incivility in Newspaper Website Comments , 2014 .

[5]  A AndersonAshley,et al.  The "Nasty Effect , 2014 .

[6]  Tamara Munzner,et al.  Design Study Methodology: Reflections from the Trenches and the Stacks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[7]  Nicholas A. Diakopoulos.  The Editor's Eye: Curation and Comment Relevance on the New York Times , 2015, CSCW.

[8]  Martin Wattenberg,et al.  Flash forums and forumReader: navigating a new kind of large-scale online discussion , 2004, CSCW.

[9]  Mor Naaman,et al.  Towards quality discourse in online news comments , 2011, CSCW.

[10]  Kenneth Y. Goldberg,et al.  Opinion space: a scalable tool for browsing online comments , 2010, CHI.

[11]  Paul Resnick,et al.  Follow the reader: filtering comments on slashdot , 2007, CHI.

[12]  Giuseppe Carenini,et al.  ConVisIT: Interactive Topic Modeling for Exploring Asynchronous Online Conversations , 2015, IUI.

[13]  N. Diakopoulos,et al.  Data-Driven Rankings : The Design and Development of the IEEE Top Programming Languages News App , 2014 .

[14]  Kathleen McElroy,et al.  Where Old (Gatekeepers) Meets New (Media) , 2013 .

[15]  Paul Resnick,et al.  Slash(dot) and burn: distributed moderation in a large online conversation space , 2004, CHI.

[16]  Clifford Nass,et al.  Normative influences on thoughtful online participation , 2011, CHI.

[17]  Jane B. Singer,et al.  QUALITY CONTROL , 2010 .

[18]  Karin Wahl Jorgensen.  Understanding the Conditions for Public Discourse: four rules for selecting letters to the editor , 2002 .

[19]  Dan M. Brown,et al.  Communicating Design: Developing Web Site Documentation for Design and Planning , 2006 .

[20]  Elizabeth F. Churchill,et al.  Automatic identification of personal insults on social news sites , 2012, J. Assoc. Inf. Sci. Technol..

[21]  Ani Nenkova,et al.  Revisiting Readability: A Unified Framework for Predicting Text Quality , 2008, EMNLP.

[22]  Nayer M. Wanas,et al.  Automatic scoring of online discussion posts , 2008, WICOW '08.

[23]  Joshua M. Scacco,et al.  Changing Deliberative Norms on News Organizations' Facebook Sites , 2015, J. Comput. Mediat. Commun..

[24]  Tamara Munzner,et al.  Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool for Investigative Journalists , 2014, IEEE Transactions on Visualization and Computer Graphics.

[25]  Mark S. Ackerman,et al.  Arkose: reusing informal information from online discussions , 2007, GROUP.

[26]  James Caverlee,et al.  Ranking Comments on the Social Web , 2009, 2009 International Conference on Computational Science and Engineering.

[27]  Kevin Wise,et al.  Moderation, Response Rate, and Message Interactivity: Features of Online Communities and Their Effects on Intent to Participate , 2006, J. Comput. Mediat. Commun..

[28]  A R Forrest,et al.  Quality control. , 1978, British medical journal.

[29]  Ani Nenkova,et al.  What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain , 2013, TACL.

[30]  Hanspeter Pfister,et al.  LineUp: Visual Analysis of Multi-Attribute Rankings , 2013, IEEE Transactions on Visualization and Computer Graphics.

[31]  Bill Reader,et al.  Air Mail: NPR Sees "Community" in Letters From Listeners , 2007 .

[32]  Nicholas Diakopoulos.  Picking the NYT Picks : Editorial Criteria and Automation in the Curation of Online News , 2015 .

[33]  Mor Naaman,et al.  Finding and assessing social media information sources in the context of journalism , 2012, CHI.

[34]  Mor Naaman,et al.  Editorial Algorithms: Using Social Media to Discover and Report Local News , 2015, ICWSM.

[35]  G. Harry McLaughlin,et al.  SMOG Grading - A New Readability Formula. , 1969 .

[36]  David A. Shamma,et al.  Understanding Online Reviews: Funny, Cool or Useful? , 2015, CSCW.