Investigating Helpfulness of Video Game Reviews on the Steam Platform

Digital retail platforms, such as Steam, offer an easy way for potential customers to evaluate video games before buying them and many rely on the experiences other users expressed in reviews. However, not all reviews are helpful making it difficult for users to extract useful information. Hence, Steam allows for users to label reviews as helpful or unhelpful and ultimately to sort by most helpful reviews. Naturally, these community-sourced labels are not available at the time of writing. In this paper, we analyze differences between helpful and unhelpful reviews by investigating a large number of video game reviews on Steam. To that end, we crawl over one hundred thousand reviews, extract numerous features, apply a statistical hypothesis test and conduct a prediction experiment. We find that there are significant differences between the two groups. For example, review length and time spent playing a game strongly influence the helpfulness of reviews. Our results reveal valuable insights for developers on how to support the community by, for example, providing immediate feedback to authors when writing reviews.

[1]  R. Gunning.  The Technique of Clear Writing. , 1968 .

[2]  Elena García Barriocanal,et al.  Evaluating content quality and helpfulness of online product reviews: The interplay of review helpfulness vs. review content , 2012, Electron. Commer. Res. Appl..

[3]  G. Harry McLaughlin,et al.  SMOG Grading - A New Readability Formula. , 1969 .

[4]  M. Coleman,et al.  A computer readability formula designed for machine scoring. , 1975 .

[5]  Jaideep Srivastava,et al.  Impact of social influence in e-commerce decision making , 2007, ICEC.

[6]  Raimundo Santos Moura,et al.  Predicting Portuguese Steam Review Helpfulness Using Artificial Neural Networks , 2016, WebMedia.

[7]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[8]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .

[9]  Aphra Kerr.  Global Games: Production, Circulation and Policy in the Networked Era , 2016 .

[10]  R. Flesch.  A new readability yardstick. , 1948, The Journal of applied psychology.

[11]  P. Nelson.  Advertising as Information , 1974, Journal of Political Economy.

[12]  Jahna Otterbacher,et al.  'Helpfulness' in online communities: a measure of message quality , 2009, CHI.

[13]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[14]  Robert J. Kauffman,et al.  Proceedings of the ninth international conference on Electronic commerce , 2003, ICEC 2007.

[15]  P. Nelson.  Information and Consumer Behavior , 1970, Journal of Political Economy.

[16]  Xiaohui Yu,et al.  Modeling and Predicting the Helpfulness of Online Reviews , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[17]  Haohan Wang,et al.  Predicting usefulness of Yelp reviews with localized linear regression models , 2016, 2016 7th IEEE International Conference on Software Engineering and Service Science (ICSESS).

[18]  B. Pan,et al.  A retrospective view of electronic word-of-mouth in hospitality and tourism management , 2017 .

[19]  E A Smith,et al.  Automated readability index. , 1967, AMRL-TR. Aerospace Medical Research Laboratories.

[20]  David Schuff,et al.  Is It the Review or the Reviewer? a Multi-Method Approach to Determine the Antecedents of Online Review Helpfulness , 2011, 2011 44th Hawaii International Conference on System Sciences.

[21]  David Schuff,et al.  What Makes a Helpful Review? A Study of Customer Reviews on Amazon.com , 2010 .

[22]  Panagiotis G. Ipeirotis,et al.  Designing novel review ranking systems: predicting the usefulness and impact of reviews , 2007, ICEC.

[23]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[24]  Jon M. Kleinberg,et al.  WWW 2009 MADRID! Track: Data Mining / Session: Opinions How Opinions are Received by Online Communities: A Case Study on Amazon.com Helpfulness Votes , 2022 .

[25]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[26]  Soo-Min Kim,et al.  Automatically Assessing Review Helpfulness , 2006, EMNLP.

[27]  David N. Laband,et al.  Advertising as Information: An Empirical Note , 1986 .

[28]  R. Schindler,et al.  Perceived Helpfulness of Online Consumer Reviews: The Role of Message Content and Style , 2010 .

[29]  Eric Gilbert,et al.  VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text , 2014, ICWSM.

[30]  Panagiotis G. Ipeirotis,et al.  Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics , 2010, IEEE Transactions on Knowledge and Data Engineering.

[31]  Anindya Ghose,et al.  Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets , 2008, Inf. Syst. Res..