Design and Evaluation of a User Experience Questionnaire for Remote Labs

Remote laboratories have been in use for 25 years now. Whereas several learning oriented meta-analyses exist, validated and agreed-upon tools for assessing the user experience are not readily available. The present paper fills this gap by designing and evaluating a questionnaire focused on the needs of remote labs developers and educators using them. Building from pre-existing tools, a first version of the User eXperience Questionnaire (UXQ version 20190308) was designed to contain four scales, usability, utility, satisfaction and immersion. A total of 180 completed responses were collected from two different remote labs (VISIR and FPGA), in different campuses and in different courses to evaluate the questionnaire. The questionnaire was analyzed in terms of reliability, using Cronbach’s alpha and McDonalds’ omega, and validity of construct, through factor analysis. The reliability of the questionnaire and its four subscales is acceptable, but its validity should be improved. Accordingly, the questionnaire was redesigned to obtain the User eXperience Questionnaire (UXQ version 20191126), which includes three scales: usability, utility and immersion, and three questions per scale. This questionnaire was assessed using the same data. Reliability coefficients are above 0.7 and construct validity is satisfactory. A new questionnaire to evaluate the user experience in remote laboratories has been designed and validated. The questionnaire, now renamed as UXQ4RL v. 1.0, is presented and made available in this paper.

[1]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[2]  William Revelle,et al.  Cronbach’s α, Revelle’s β, and Mcdonald’s ωH: their relations with each other and two alternative conceptualizations of reliability , 2005 .

[3]  Lawrence A. Crowl,et al.  Running Control Engineering Experiments Over the Internet , 1996 .

[4]  Enrique Herrera-Viedma,et al.  Virtual and remote labs in education: A bibliometric analysis , 2016, Comput. Educ..

[5]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[6]  Javier García-Zubía,et al.  LabsLand: A sharing economy platform to promote educational remote laboratories maintainability, sustainability and adoption , 2016, 2016 IEEE Frontiers in Education Conference (FIE).

[7]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[8]  J.V. Nickerson,et al.  Remote versus hands-on labs: a comparative study , 2004, 34th Annual Frontiers in Education, 2004. FIE 2004..

[9]  Jack P. Krichen,et al.  Networking Labs in the Online Environment: Indicators for Success. , 2010 .

[10]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[11]  James R. Brinson.  Learning outcome achievement in non-traditional (virtual and remote) versus traditional (hands-on) laboratories: A review of the empirical research , 2015, Comput. Educ..

[12]  Sebastián Dormido Bencomo,et al.  UNED Labs: A Network of Virtual and Remote Laboratories , 2011 .

[13]  Thomas C. Pentecost,et al.  Moving beyond Alpha: A Primer on Alternative Sources of Single-Administration Reliability Evidence for Quantitative Chemistry Education Research , 2018, Journal of Chemical Education.

[14]  Jeffrey V. Nickerson,et al.  Hands-on, simulated, and remote laboratories: A comparative literature review , 2006, CSUR.

[15]  Diego López-de-Ipiña,et al.  Application and user perceptions of using the WebLab-Deusto-PLD in technical education , 2011, 2011 First Global Online Laboratory Consortium Remote Laboratories Workshop.

[16]  Jeffrey E. Froyd,et al.  Five Major Shifts in 100 Years of Engineering Education , 2012, Proceedings of the IEEE.

[17]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[18]  David J. Woodruff,et al.  Statistical Inference for Coefficient Alpha , 1987 .

[19]  Javier García Zubía,et al.  Using remote labs in education: two little ducks in remote experimentation , 2011 .

[20]  Gustavo Ribeiro Alves,et al.  Impact of a remote lab on teaching practices and student learning , 2018, Comput. Educ..

[21]  Roderick P. McDonald,et al.  Generalizability in Factorable Domains: "Domain Validity and Generalizability"1 , 1978 .

[22]  J. Ramalho-Santos,et al.  Cronbach's alpha: a tool for assessing the reliability of scales , 1999 .

[23]  Wenshan Hu,et al.  NCSLab remote experimentation for control engineering education in Wuhan University , 2016, 2016 35th Chinese Control Conference (CCC).

[24]  Dilek Karahoca,et al.  Re-assessing the usability metric for user experience (UMUX) scale , 2016 .

[25]  James R. Lewis,et al.  UMUX-LITE: when there's no time for the SUS , 2013, CHI.

[26]  Javier García-Zubía,et al.  Acceptance, usability and usefulness of WebLab-Deusto from students point of view , 2008, 2008 Third International Conference on Digital Information Management.

[27]  W. Revelle,et al.  Reliability from α to ω: A tutorial. , 2019, Psychological assessment.

[28]  Ariadna Angulo-Brunet,et al.  A journey around alpha and omega to estimate internal consistency reliability , 2017 .

[29]  Angeliki Poulymenakou,et al.  Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability , 2009, Int. J. Hum. Comput. Interact..

[30]  Michel Billaud,et al.  Pedagogical evaluation of remote laboratories in eMerge project , 2007 .

[31]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[32]  Ingvar Gustavsson,et al.  Empirical Analysis of the Use of the VISIR Remote Lab in Teaching Analog Electronics , 2017, IEEE Transactions on Education.

[33]  Michael J. Singer,et al.  The Factor Structure of the Presence Questionnaire , 2005, Presence: Teleoperators & Virtual Environments.

[34]  Michael R. Mullen,et al.  Structural equation modelling: guidelines for determining model fit , 2008 .

[35]  Robert I. Jennrich,et al.  Gradient Projection Algorithms and Software for Arbitrary Rotation Criteria in Factor Analysis , 2005 .