Measuring the Quality of the Website User Experience

Consumers spend an increasing amount of time and money online finding information, completing tasks, or making purchases. The quality of the website experience has become a key differentiator for organizations—affecting whether they purchase and their likelihood to return and recommend a website to friends. Two instruments were created to more effectively measure the quality of the website user experience to help improve the experience. Three studies used Classical Test Theory (CTT) to create a new instrument to measure the quality of the website user experience from the website visitor’s perspective. Data were collected over five years from more than 4,000 respondents reflecting on experiences with more than 100 websites. An eight-item questionnaire of website quality was created—the Standardized User Experience Percentile Rank Questionnaire (SUPRQ). The SUPR-Q contains four factors: usability, trust, appearance, and loyalty. The factor structure was replicated across three studies, with data collected both during usability tests and retrospectively in surveys. There was evidence of convergent validity with existing questionnaires, including the System Usability Scale (SUS). An initial distribution of scores across the websites generated a database used to produce percentile ranks and make scores more meaningful to researchers and practitioners. In Study 4, a new set of data and confirmatory factor analysis (CFA) confirmed the factor structure and generated alternative items that work on non–e-commerce websites. The SUPR-Q can be

[1]  R. Kline.  Principles and practice of structural equation modeling, 3rd ed. , 2011 .

[2]  Marc Hassenzahl,et al.  The Effect of Perceived Hedonic Quality on Product Appealingness , 2001, Int. J. Hum. Comput. Interact..

[3]  Kesten C. Green,et al.  Predictive Validity of Evidence-Based Persuasion Principles: An Application of the Index Method , 2015 .

[4]  Ann Heylighen,et al.  How relative absolute can be: SUMI and the impact of the nature of the task in measuring perceived software usability , 2007, AI & SOCIETY.

[5]  Alexandre N. Tuch,et al.  Is beautiful really usable? Toward understanding the relation between usability, aesthetics, and affect in HCI , 2012, Comput. Hum. Behav..

[6]  Qimei Chen,et al.  Attitude Toward the Site , 1999 .

[7]  Robert E. Boostrom,et al.  Plenty of attitude: evaluating measures of attitude toward the site , 2013 .

[8]  Sylvain Sénécal,et al.  Measuring Perceived Website Usability , 2007 .

[9]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[10]  Jinwoo Kim,et al.  Architectural criteria for website evaluation – conceptual framework and empirical validation , 2004, Behav. Inf. Technol..

[11]  Herbjørn Nysveen,et al.  Drivers of Intention to Revisit the Websites of Well-Known Companies: The Role of Corporate Brand Loyalty , 2001 .

[12]  F. Reichheld.  The one number you need to grow. , 2003, Harvard business review.

[13]  Steve Krug,et al.  Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability , 2013 .

[14]  Jeff Sauro,et al.  Quantifying the User Experience: Practical Statistics for User Research , 2012 .

[15]  Richard H. Gutsche.  User experience , 1976, PERV.

[16]  A. Parasuraman,et al.  A conceptual framework for understanding e-service quality : implications for future research and managerial practice , 2000 .

[17]  Ann Macintosh,et al.  Human interface guidelines: the apple desktop interface , 1987 .

[18]  Qimei Chen,et al.  Attitude Toward The Site II: New Information , 2002, Journal of Advertising Research.

[19]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[20]  Fred Reichheld,et al.  The Ultimate Question: Driving Good Profits and True Growth , 2006 .

[21]  J. Linacre,et al.  Many-facet Rasch measurement , 1994 .

[22]  Clare-Marie Karat,et al.  Comparison of empirical testing and walkthrough methods in user interface evaluation , 1992, CHI.

[23]  Erin V. Lehman,et al.  Development of an Instrument , 2005 .

[24]  Paul A. Pavlou,et al.  Understanding and Predicting Electronic Commerce Adoption: An Extension of the Theory of Planned Behavior , 2006, MIS Q..

[25]  Corrado Lo Storto.  Evaluating ecommerce websites cognitive efficiency: an integrative framework based on data envelopment analysis. , 2013, Applied ergonomics.

[26]  Jurek Kirakowski,et al.  The Software Usability Measurement Inventory: Background and Usage , 1996 .

[27]  Carl W. Turner,et al.  Validation of a Two Factor Structure for System Trust , 2005 .

[28]  W. Hays.  Using Multivariate Statistics , 1983 .

[29]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[30]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[31]  Gerald L. Lohse,et al.  Internet retail store design: How the user interface influences traffic and sales , 2006, J. Comput. Mediat. Commun..

[32]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[33]  Alhassan G. Abdul-Muhmin.  Repeat Purchase Intentions in Online Shopping: The Role of Satisfaction, Attitude, and Online Retailers' Performance , 2010 .

[34]  Dominique L. Scapin,et al.  Ergonomic criteria for evaluating the ergonomic quality of interactive systems , 1997, Behav. Inf. Technol..

[35]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[36]  Jeff Sauro,et al.  When designing usability questionnaires, does it hurt to be positive? , 2011, CHI.

[37]  Kenneth E. Clow,et al.  Web Site Interaction Satisfaction: Scale Development Considerations , 2008 .

[38]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[39]  Cathleen Wharton,et al.  Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces , 1990, CHI '90.

[40]  M. Gilly,et al.  .comQ: Dimensionalizing, Measuring, and Predicting Quality of the E-tail Experience , 2002 .

[41]  Jeff Sauro,et al.  Correlations among prototypical usability metrics: evidence for the construct of usability , 2009, CHI.

[42]  Joey F. George,et al.  Influences on the intent to make Internet purchases , 2002, Internet Res..

[43]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[44]  C. Fox,et al.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2001 .

[45]  Prashant C. Palvia,et al.  Developing and validating an instrument for measuring user-perceived web quality , 2002, Inf. Manag..

[46]  R. Thakur,et al.  A Parsimonious Model of the Antecedents and Consequence of Online Trust: An Uncertainty Perspective , 2008 .

[47]  R. Keeney,et al.  The Value of Internet Commerce to the Customer , 1999 .

[48]  Marc Hassenzahl,et al.  User experience - a research agenda , 2006, Behav. Inf. Technol..

[49]  イントランス.  Human interface guidelines : the apple desktop interface , 1989 .

[50]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[51]  Gordon C. Bruner.  SIMILARITY ANALYSIS OF THREE ATTITUDE-TOWARD-THE-WEBSITE SCALES , 2001 .

[52]  Sidney L. Smith,et al.  Guidelines for Designing User Interface Software , 1986 .

[53]  Pradeep Korgaonkar,et al.  Web usage, advertising, and shopping: relationship patterns , 2002, Internet Res..

[54]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[55]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[56]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[57]  Philip Bobko,et al.  Correlation and Regression: Applications for Industrial Organizational Psychology and Management , 2001 .

[58]  Ebba Þóra Hvannberg,et al.  Analysis of strategies for improving and estimating the effectiveness of heuristic evaluation , 2004, NordiCHI '04.

[59]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[60]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[61]  Klaus Opwis,et al.  Intranet satisfaction questionnaire: Development and validationof a questionnaire to measure user satisfaction with the Intranet , 2009, Comput. Hum. Behav..

[62]  Stefano Federici,et al.  On the dimensionality of the System Usability Scale: a test of alternative measurement models , 2009, Cognitive Processing.

[63]  Anne Marsden,et al.  International Organization for Standardization , 2014 .

[64]  Nigel Bevan,et al.  Extending Quality in Use to Provide a Framework for Usability Measurement , 2009, HCI.

[65]  Thomas S. Tullis,et al.  A Comparison of Questionnaires for Assessing Website Usability , 2004 .

[66]  Angela M. Cirucci,et al.  Usability Testing , 2021, UX Research Methods for Media and Communication Studies.

[67]  James R. Lewis,et al.  UMUX-LITE: when there's no time for the SUS , 2013, CHI.

[68]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[69]  David G. Novick,et al.  Usability inspection methods after 15 years of research and practice , 2007, SIGDOC '07.

[70]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[71]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[72]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[73]  Ingoo Han,et al.  The Impact of Customer Trust and Perception of Security Control on the Acceptance of Electronic Commerce , 2003, Int. J. Electron. Commer..

[74]  B. Tabachnick,et al.  Using multivariate statistics, 5th ed. , 2007 .

[75]  Bruce Cooil,et al.  A Longitudinal Examination of Net Promoter and Firm Revenue Growth , 2007 .

[76]  Bonnie E. John,et al.  Tracking the effectiveness of usability evaluation methods , 1997, Behav. Inf. Technol..

[77]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[78]  M. R. Espejo.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2004 .