On the Role of Score, Genre and Text in Helpfulness of Video Game Reviews on Metacritic

Evaluating others' reviews of products via helpfulness votes is a crucial aspect of purchase decision-making in online marketplaces. Previous work studied key determinant factors of review helpfulness, such as product metadata and review text. However, understanding the extent to which review helpfulness depends on product context, rather than the inherent textual value of a review, remains an open question. In this work, we study how genre, score and review text relate to the helpfulness of 319 017 video game reviews on Metacritic via correlational analyses and prediction experiments. We find significant, genre-dependent correlations between score and helpfulness. This underlies the strength of text vs. score features in our helpfulness prediction models, which achieve up to 0.64 F1. This work is thus a stepping stone towards causal disentanglement of product context from text in review helpfulness estimation.

[1]  Christian Gütl,et al.  Investigating Helpfulness of Video Game Reviews on the Steam Platform , 2018, 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS).

[2]  R. Flesch.  A new readability yardstick. , 1948, The Journal of applied psychology.

[3]  Jahna Otterbacher,et al.  'Helpfulness' in online communities: a measure of message quality , 2009, CHI.

[4]  Panagiotis G. Ipeirotis,et al.  Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics , 2010, IEEE Transactions on Knowledge and Data Engineering.

[5]  Panagiotis G. Ipeirotis,et al.  Designing novel review ranking systems: predicting the usefulness and impact of reviews , 2007, ICEC.

[6]  M. Coleman,et al.  A computer readability formula designed for machine scoring. , 1975 .

[7]  David Schuff,et al.  What Makes a Helpful Review? A Study of Customer Reviews on Amazon.com , 2010 .

[8]  Haohan Wang,et al.  Predicting usefulness of Yelp reviews with localized linear regression models , 2016, 2016 7th IEEE International Conference on Software Engineering and Service Science (ICSESS).

[9]  Miao Fan,et al.  Product-Aware Helpfulness Prediction of Online Reviews , 2019, WWW.

[10]  David Schuff,et al.  Is It the Review or the Reviewer? a Multi-Method Approach to Determine the Antecedents of Online Review Helpfulness , 2011, 2011 44th Hawaii International Conference on System Sciences.

[11]  R. Schindler,et al.  Perceived Helpfulness of Online Consumer Reviews: The Role of Message Content and Style , 2010 .

[12]  Elena García Barriocanal,et al.  Evaluating content quality and helpfulness of online product reviews: The interplay of review helpfulness vs. review content , 2012, Electron. Commer. Res. Appl..

[13]  Soo-Min Kim,et al.  Automatically Assessing Review Helpfulness , 2006, EMNLP.

[14]  G. Harry McLaughlin,et al.  SMOG Grading - A New Readability Formula. , 1969 .

[15]  X. Zhang,et al.  Impact of Online Consumer Reviews on Sales: The Moderating Role of Product and Consumer Characteristics , 2010 .

[16]  Raimundo Santos Moura,et al.  Predicting Portuguese Steam Review Helpfulness Using Artificial Neural Networks , 2016, WebMedia.

[17]  Jon M. Kleinberg,et al.  WWW 2009 MADRID! Track: Data Mining / Session: Opinions How Opinions are Received by Online Communities: A Case Study on Amazon.com Helpfulness Votes , 2022 .

[18]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[19]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[20]  Anindya Ghose,et al.  Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets , 2008, Inf. Syst. Res..

[21]  David N. Laband,et al.  Advertising as Information: An Empirical Note , 1986 .

[22]  P. Nelson.  Advertising as Information , 1974, Journal of Political Economy.

[23]  Daniel M. Johnson,et al.  The edge of glory: the relationship between metacritic scores and player experience , 2014, CHI PLAY.

[24]  Jure Leskovec,et al.  How Community Feedback Shapes User Behavior , 2014, ICWSM.

[25]  R. Gunning.  The Technique of Clear Writing. , 1968 .

[26]  Jun Zhou,et al.  Multi-Domain Gated CNN for Review Helpfulness Prediction , 2019, WWW.

[27]  Mia Consalvo,et al.  Digital pitchforks and virtual torches: Fan responses to the Mass Effect news debacle , 2011 .

[28]  P. Nelson.  Information and Consumer Behavior , 1970, Journal of Political Economy.

[29]  E A Smith,et al.  Automated readability index. , 1967, AMRL-TR. Aerospace Medical Research Laboratories.

[30]  Eric Gilbert,et al.  VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text , 2014, ICWSM.

[31]  Anders Drachen,et al.  Only the good... get pirated: game piracy activity vs. metacritic score , 2011, FDG.