The development and evaluation of a survey to measure user engagement

Facilitating engaging user experiences is essential in the design of interactive systems. To accomplish this, it is necessary to understand the composition of this construct and how to evaluate it. Building on previous work that posited a theory of engagement and identified a core set of attributes that operationalized this construct, we constructed and evaluated a multidimensional scale to measure user engagement. In this paper we describe the development of the scale, as well as two large-scale studies (N = 440 and N = 802) that were undertaken to assess its reliability and validity in online shopping environments. In the first we used Reliability Analysis and Exploratory Factor Analysis to identify six attributes of engagement: Perceived Usability, Aesthetics, Focused Attention, Felt Involvement, Novelty, and Endurability. In the second we tested the validity of and relationships among those attributes using Structural Equation Modeling. The result of this research is a multidimensional scale that may be used to test the engagement of software applications. In addition, findings indicate that attributes of engagement are highly intertwined, a complex interplay of user-system interaction variables. Notably, Perceived Usability played a mediating role in the relationship between Endurability and Novelty, Aesthetics, Felt Involvement, and Focused Attention.

[1]  D. Hebb.  Textbook of psychology , 1958 .

[2]  G. Boyle.  Critical review of state-trait curiosity test development , 1983 .

[3]  J. B. Kernan,et al.  On the Meaning of Leisure: An Investigation of Some Determinants of the Subjective Experience , 1983 .

[4]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[5]  James D. Hollan,et al.  Direct Manipulation Interfaces , 1985, Hum. Comput. Interact..

[6]  Craig A. Smith,et al.  Patterns of cognitive appraisal in emotion. , 1985, Journal of personality and social psychology.

[7]  S. Crown.  The Penguin Dictionary of Psychology. By Arthur S. Reber. London: Penguin Books. 1985. Pp 848. £5.95, $7.95. , 1987, British Journal of Psychiatry.

[8]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[9]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[10]  Jawaid A. Ghani,et al.  The Experience Of Flow In Computer-Mediated And In Face-To-Face Groups , 1991, ICIS.

[11]  Brenda Laurel,et al.  Computers as theatre , 1991 .

[12]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[13]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[14]  Peter A. Todd,et al.  Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication , 1992, MIS Q..

[15]  J. M. Cortina,et al.  What Is Coefficient Alpha? An Examination of Theory and Applications , 1993 .

[16]  J. Webster,et al.  The Dimensionality and Correlates of Flow in Human-Computer Interactions. , 1993 .

[17]  William R. Darden,et al.  Work and/or Fun: Measuring Hedonic and Utilitarian Shopping Value , 1994 .

[18]  Clifford Nass,et al.  Seductive interfaces: satisfying a mass audience , 1994, CHI Conference Companion.

[19]  Leon A. Kappelman,et al.  Measuring user involvement: a diffusion of innovation perspective , 1995, DATB.

[20]  Tom Carey,et al.  Engagement as a Design Concept for Multimedia , 1995 .

[21]  Richard David Jacques,et al.  The nature of engagement and its role in hypermedia evaluation and design , 1996 .

[22]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[23]  Lloyd P. Rieber,et al.  Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games , 1996 .

[24]  Pekka Makkonen,et al.  Does collaborative hypertext support better engagement in learning of the basics in informatics? , 1997, ITiCSE '97.

[25]  Noam Tractinsky,et al.  Aesthetics and apparent usability: empirically assessing cultural and methodological issues , 1997, CHI.

[26]  T. Reio.  EFFECTS OF CURIOSITY ON SOCIALIZATION-RELATED LEARNING AND JOB PERFORMANCE IN ADULTS , 1997 .

[27]  Jane Webster,et al.  Audience engagement in multimedia presentations , 1997, DATB.

[28]  Gregory Schraw,et al.  The Role of Choice in Reader Engagement , 1998 .

[29]  E. Kelloway.  Using LISREL for Structural Equation Modeling: A Researcher′s Guide , 1998 .

[30]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[31]  G. Hutcheson.  The Multivariate Social Scientist: Introductory Statistics Using Generalized Linear Models , 1999 .

[32]  B. Joseph Pine,et al.  The Experience Economy , 2020, Journal of Orthopaedic Experience & Innovation.

[33]  R. Peterson.  Constructing Effective Questionnaires , 1999 .

[34]  Peter Chapman,et al.  Engagement in multimedia training systems , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[35]  Bo N. Schenkman,et al.  Aesthetics and preferences of web pages , 2000, Behav. Inf. Technol..

[36]  R. Vallerand,et al.  On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS) , 2000 .

[37]  Detmar W. Straub,et al.  The Relative Importance of Perceived Ease of Use in IS Adoption: A Study of E-Commerce Adoption , 2000, J. Assoc. Inf. Syst..

[38]  Donna L. Hoffman,et al.  Measuring the Customer Experience in Online Environments: A Structural Modeling Approach , 2000 .

[39]  Russel L. Thompson,et al.  A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys , 2000 .

[40]  Morgan Jennings,et al.  Theory and models for creating engaging and immersive ecommerce Websites , 2000, SIGCPR '00.

[41]  Ping Zhang,et al.  Satisfiers and dissatisfiers: a two-factor model for website design and evaluation , 2000 .

[42]  Andrew Hargadon,et al.  The pleasure principle: immersion, engagement, flow , 2000, HYPERTEXT '00.

[43]  N. Tractinsky,et al.  What is beautiful is usable , 2000, Interact. Comput..

[44]  T. Kline,et al.  Common Method Variance and Specification Errors: A Practical Approach to Detection , 2000, The Journal of psychology.

[45]  U. Konradt,et al.  The Experience of Flow in Interacting With a Hypermedia Learning Environment , 2001 .

[46]  Edward E. Rigdon,et al.  Experiential value: Conceptualization, measurement and application in the catalog and Internet shopping environment. , 2001 .

[47]  Terry L. Childers,et al.  HEDONIC AND UTILITARIAN MOTIVATIONS FOR ONLINE RETAIL SHOPPING BEHAVIOR , 2001 .

[48]  Jordan Litman,et al.  Measuring Epistemic Curiosity and Its Diversive and Specific Components , 2003, Journal of personality assessment.

[49]  Yuping Liu,et al.  Developing a scale to measure the interactivity of websites , 2003, Journal of Advertising Research.

[50]  K. Reynolds,et al.  Hedonic shopping motivations , 2003 .

[51]  David J. Shernoff,et al.  Student engagement in high school classrooms from the perspective of flow theory. , 2003 .

[52]  Ming-Hui Huang,et al.  Designing website attributes to induce experiential encounters , 2003, Comput. Hum. Behav..

[53]  Curtis S. Ikehara,et al.  Methodological issues of real time data acquisition from multiple sources of physiological data , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[54]  Erik Champion,et al.  Applying game design theory to virtual heritage environments , 2003, GRAPHITE '03.

[55]  Whitney Quesenbery,et al.  Dimensions of Usability: Defining the Conversation, Driving the Process , 2003 .

[56]  Edward E. Rigdon,et al.  Play, Flow, and the Online Search Experience , 2004 .

[57]  Gitte Lindgaard,et al.  Adventurers versus nit-pickers on affective computing , 2004, Interact. Comput..

[58]  Jinwoo Kim,et al.  Critical factors for the aesthetic fidelity of web pages: empirical studies with professional web designers and users , 2004, Interact. Comput..

[59]  Noam Tractinsky,et al.  Assessing dimensions of perceived visual aesthetics of web sites , 2004 .

[60]  Norma S. Said,et al.  An engaging multimedia design model , 2004, IDC '04.

[61]  Jinwoo Kim,et al.  Why People Continue to Play Online Games: In Search of Critical Design Factors to Increase Customer Loyalty to Online Contents , 2004, Cyberpsychology Behav. Soc. Netw..

[62]  R. Schwarzer,et al.  Measuring one component of dispositional self-regulation: attention control in goal pursuit , 2004 .

[63]  Steven Pace,et al.  A grounded theory of the flow experiences of Web users , 2004, Int. J. Hum. Comput. Stud..

[64]  Timo Saari,et al.  Transcendent experience in the use of computer-based media , 2004, NordiCHI '04.

[65]  Jodi Forlizzi,et al.  Understanding experience in interactive systems , 2004, DIS '04.

[66]  Regan L. Mandryk,et al.  Objectively evaluating entertainment technology , 2004, CHI EA '04.

[67]  L. Bannon,et al.  Understanding affect in design: an outline conceptual framework , 2004 .

[68]  Debbie Stone,et al.  User Interface Design and Evaluation , 2005 .

[69]  Liam J. Bannon,et al.  A human-centred perspective on interaction design , 2005 .

[70]  Paul Cairns,et al.  Engagement with an Interactive Museum Exhibit , 2005, BCS HCI.

[71]  Elizabeth Kelley.  Context and leadership in the remote environment , 2005 .

[72]  John C. McCarthy,et al.  The value of the novel in designing for experience , 2005 .

[73]  Richard Hull,et al.  Designing Engaging Experiences with Children and Artists , 2005, Funology.

[74]  Yu-Chen Chen,et al.  Extrinsic versus intrinsic motivations for consumers to shop on-line , 2005, Inf. Manag..

[75]  Caroline Hummels,et al.  Let's Make Things Engaging , 2005, Funology.

[76]  Jane Webster,et al.  Enhancing the Design of Web Navigation Systems: The Influence of User Disorientation on Engagement and Performance , 2006, MIS Q..

[77]  Agnieszka Bojko,et al.  Using eye tracking to compare web page designs: a case study , 2006 .

[78]  Gitte Lindgaard,et al.  Attention web designers: You have 50 milliseconds to make a good first impression! , 2006, Behav. Inf. Technol..

[79]  Guy H. Walker,et al.  Feedback and driver situation awareness (SA): A comparison of SA measures and contexts , 2008 .

[80]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[81]  P. Cairns,et al.  From immersion to addiction in videogames , 2008 .

[82]  D. Spence,et al.  Heterogeneous treatment effects of integrated soil fertility management on crop productivity: Evidence from Nigeria , 2011 .