Extracting usability and user experience information from online user reviews

Internet review sites allow consumers to write detailed reviews of products potentially containing information related to user experience (UX) and usability. Using 5198 sentences from 3492 online reviews of software and video games, we investigate the content of online reviews with the aims of (i) charting the distribution of information in reviews among different dimensions of usability and UX, and (ii) extracting an associated vocabulary for each dimension using techniques from natural language processing and machine learning. We (a) find that 13%-49% of sentences in our online reviews pool contain usability or UX information; (b) chart the distribution of four sets of dimensions of usability and UX across reviews from two product categories; (c) extract a catalogue of important word stems for a number of dimensions. Our results suggest that a greater understanding of users' preoccupation with different dimensions of usability and UX may be inferred from the large volume of self-reported experiences online, and that research focused on identifying pertinent dimensions of usability and UX may benefit further from empirical studies of user-generated experience reports.

[1]  Jan Bosch,et al.  A framework for capturing the relationship between usability and software architecture , 2003, Softw. Process. Improv. Pract..

[2]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[3]  Ben Shneiderman,et al.  Designing the user interface - strategies for effective human-computer interaction, 3rd Edition , 1997 .

[4]  Jonathan Earthy,et al.  The Benefits of Using ISO 13407: Human Centred Design Process for Interactive Systems , 2001, INTERACT.

[5]  Brian Shackel,et al.  Usability - Context, framework, definition, design and evaluation , 1991, Interact. Comput..

[6]  Anders Bruun,et al.  The effect of task assignments and instruction types on remote asynchronous usability testing , 2012, CHI.

[7]  Eric K. Ringger,et al.  Pulse: Mining Customer Opinions from Free Text , 2005, IDA.

[8]  Paul A. Pavlou,et al.  Can online reviews reveal a product's true quality?: empirical findings and analytical modeling of Online word-of-mouth communication , 2006, EC '06.

[9]  Sarah Diefenbach,et al.  Needs, affect, and interactive products - Facets of user experience , 2010, Interact. Comput..

[10]  Marc Hassenzahl,et al.  User experience - a research agenda , 2006, Behav. Inf. Technol..

[11]  Kaisa Väänänen,et al.  Let users tell the story: evaluating user experience with experience reports , 2010, CHI Extended Abstracts.

[12]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[13]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[14]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .

[15]  Hakan Ferhatosmanoglu,et al.  Short text classification in twitter to improve information filtering , 2010, SIGIR.

[16]  Deborah Hix,et al.  Remote usability evaluation: can users report their own critical incidents? , 1998, CHI Conference Summary.

[17]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[18]  Marc Hassenzahl,et al.  User experience (UX): towards an experiential perspective on product quality , 2008, IHM '08.

[19]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[20]  Larry L. Constantine,et al.  Software for Use - A Practical Guide to the Models and Methods of Usage-Centered Design , 1999 .

[21]  C. Borror.  Nonparametric Statistical Methods, 2nd, Ed. , 2001 .

[22]  Ben Shneiderman,et al.  Designing the user interface (2nd ed.): strategies for effective human-computer interaction , 1992 .

[23]  Jurek Kirakowski,et al.  Functionality, usability, and user experience: three areas of concern , 2006, INTR.

[24]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[25]  Thomas Olsson,et al.  Narratives of satisfying and unsatisfying experiences of current mobile augmented reality applications , 2012, CHI.

[26]  Kasper Hornbæk,et al.  Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience , 2011, CHI.

[27]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[28]  Nigel Bevan,et al.  What is the difference between the purpose of usability and user experience evaluation methods , 2009 .

[29]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[30]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[31]  Dwayne D. Gremler,et al.  Electronic word-of-mouth via consumer-opinion platforms: What motivates consumers to articulate themselves on the Internet? , 2004 .

[32]  E. Anderson.  Customer Satisfaction and Word of Mouth , 1998 .

[33]  Soo-Min Kim,et al.  Automatic Identification of Pro and Con Reasons in Online Reviews , 2006, ACL.

[34]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[35]  R. Lutz.  Changing Brand Attitudes through Modification of Cognitive Structure , 1975 .

[36]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[37]  Nigel Bevan,et al.  Classifying and selecting UX and usability measures , 2008 .

[38]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[39]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[40]  Rex B. Kline,et al.  Usability measurement and metrics: A consolidated model , 2006, Software Quality Journal.