Evaluation of features to predict the usefulness of online reviews

Checking online reviews before purchasing goods or using services has become increasingly common. However, it is difficult to select useful reviews, and concerns about fake reviews are growing. Many online review systems use recency and user‐generated usefulness votes in order to prioritize reviews for users, but there is much room for improvement. In this work, we focus on evaluating the effectiveness of a large number of features for predicting the usefulness of online reviews, including features that have not been commonly evaluated in prior work (e.g., social network measures). Features were grouped into hierarchical categories that might represent factors impacting perceived usefulness of Yelp users. Using all features, a binary classifier achieved a high level of accuracy (0.889). Additionally, a feature ablation study found that several feature groups yielded statistically significant improvements. Interestingly, many of the features that improved performance are not the types of measures that are displayed to users in commercial online review services such as Yelp and are the measures that are rarely used to prioritize reviews for users. Our study results suggest different types of information that online review services might want to use in ranking and displaying reviews for users.

[1]  R. Cialdini,et al.  New Ways to Promote Proenvironmental Behavior: The Application of Persuasion Theory to the Development Of Effective Proenvironmental Public Service Announcements , 2000 .

[2]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[3]  Daniela B. Friedman,et al.  A Systematic Review of Readability and Comprehension Instruments Used for Print and Web-Based Cancer Information , 2006, Health education & behavior : the official publication of the Society for Public Health Education.

[4]  Franziska Marquart,et al.  Communication and persuasion : central and peripheral routes to attitude change , 1988 .

[5]  Pranjal Gupta,et al.  How e-WOM recommendations influence product consideration and quality of choice: A motivation to process information perspective , 2010 .

[6]  J. Powell,et al.  Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. , 2002, JAMA.

[7]  Allen M. Weiss,et al.  Listening to Strangers: Whose Responses are Valuable, how Valuable are They, and Why? , 2008 .

[8]  Dietrich Klakow,et al.  A survey on the role of negation in sentiment analysis , 2010, NeSp-NLP@ACL.

[9]  Michael Luca,et al.  Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud , 2015 .

[10]  Yue Lu,et al.  Exploiting social context for review quality prediction , 2010, WWW '10.

[11]  Vladimir Batagelj,et al.  Pajek - Program for Large Network Analysis , 1999 .

[12]  Tim Mazzarol,et al.  Factors influencing word of mouth effectiveness: receiver perspectives , 2008 .

[13]  Eric K. Clemons,et al.  When Online Reviews Meet Hyperdifferentiation: A Study of Craft Beer Industry , 2006, Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06).

[14]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[15]  Abhijit Biswas,et al.  The diagnostic role of signals in the context of perceived risks in online shopping: Do signals matter more on the Web? , 2004 .

[16]  Don Fallis,et al.  On Verifying the Accuracy of Information: Philosophical Perspectives , 2003, Libr. Trends.

[17]  Soo-Min Kim,et al.  Automatically Assessing Review Helpfulness , 2006, EMNLP.

[18]  N. Zhang,et al.  Predicting Yelp Ratings Using User Friendship Network Information , 2015 .

[19]  John T. Cacioppo,et al.  The Elaboration Likelihood Model of Persuasion , 1986, Advances in Experimental Social Psychology.

[20]  Huaping Chen,et al.  Credibility of Electronic Word-of-Mouth: Informational and Normative Determinants of On-line Consumer Recommendations , 2009, Int. J. Electron. Commer..

[21]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[22]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[23]  Michael L. Anderson,et al.  Learning from the Crowd: Regression Discontinuity Estimates of the Effects of an Online Review Database , 2012 .

[24]  Wesley W. Chu,et al.  A Social Network-Based Recommender System (SNRS) , 2010, Data Mining for Social Network Data.

[25]  Iryna Pentina,et al.  Exploring effects of source similarity, message valence, and receiver regulatory focus on yelp review persuasiveness and purchase intentions , 2018 .

[26]  Panagiotis G. Ipeirotis,et al.  Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics , 2010, IEEE Transactions on Knowledge and Data Engineering.

[27]  Mehdi Behboudi,et al.  USING SOCIAL NETWORK PARADIGM FOR DEVELOPING A CONCEPTUAL FRAMEWORK IN CRM , 2011, Australian Journal of Business and Management Research.

[28]  Pradeep Racherla,et al.  Perceived 'usefulness' of online consumer reviews: An exploratory investigation across three services categories , 2012, Electron. Commer. Res. Appl..

[29]  Soo Young Rieh,et al.  Credibility: A multidisciplinary framework , 2007, Annu. Rev. Inf. Sci. Technol..

[30]  Asher Levi,et al.  The Social Aspect of Voting for Useful Reviews , 2014, SBP.

[31]  R. Cialdini,et al.  The Application of Persuasion Theory to the Development Of Effective Proenvironmental Public Service Announcements , 2000 .

[32]  Soo Young Rieh.  Judgment of information quality and cognitive authority in the Web , 2002, J. Assoc. Inf. Sci. Technol..

[33]  J. Swait,et al.  Brand Credibility, Brand Consideration, and Choice , 2004 .

[34]  Martin G. Everett,et al.  Analyzing social networks , 2013 .

[35]  Wonjoon Kim,et al.  The Heterogeneous Effect of WOM on Product Sales: Why the Effect of WOM Valence Is Mixed? , 2011 .

[36]  Donnavieve N. Smith,et al.  Online Peer and Editorial Recommendations, Trust, and Choice in Virtual Markets , 2005 .

[37]  Carlo Strapparava,et al.  WordNet Affect: an Affective Extension of WordNet , 2004, LREC.

[38]  N. Roberts,et al.  Second-hand knowledge: An inquiry into cognitive authority: P. Wilson. London: Greenwood Press, 1983. 210 pp. ISBN 0 313 23763 8. £28.95. , 1985 .

[39]  Richard L. Daft,et al.  Organizational information requirements, media richness and structural design , 1986 .

[40]  L. Mui,et al.  A computational model of trust and reputation , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[41]  J. Cacioppo,et al.  The Elaboration Likelihood Model of Persuasion , 1986 .

[42]  Soo Young Rieh,et al.  Interaction on the Web: Scholars' Judgment of Information Quality and Cognitive Authority , 2000 .

[43]  R. Hogarth.  Judgement and choice: The psychology of decision , 1982 .

[44]  Ann E. Schlosser.  Posting versus Lurking: Communicating in a Multiple Audience Context , 2005 .

[45]  C. Chung,et al.  The impact of word-of-mouth communication on attribute evaluation , 2011 .

[46]  Claudia López,et al.  Analysis of local online review systems as digital word-of-mouth , 2014, WWW.

[47]  Anindya Ghose,et al.  Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets , 2008, Inf. Syst. Res..

[48]  Troy V. Mumford.  Whom to Believe: Recruiting Information Source Credibility and Organizational Attractiveness , 2012 .

[49]  Steven Bird,et al.  NLTK: The Natural Language Toolkit , 2002, ACL.

[50]  Jaime Arguello,et al.  Predicting Speech Acts in MOOC Forum Posts , 2015, ICWSM.

[51]  B. J. Fogg,et al.  Web credibility research: a method for online experiments and early study results , 2001, CHI Extended Abstracts.