The impact of self-efficacy and perceived system efficacy on effectiveness of virtual training systems

This study developed and tested a research model which examined the impact of user perceptions of self-efficacy (SE) and virtual environment (VE) efficacy on the effectiveness of VE training systems. The model distinguishes between the perceptions of one's own capability to perform trained tasks effectively and the perceptions of system performance, regarding the established parameters from literature. Specifically, the model posits that user perceptions will have positive effects on task performance and memory. Seventy-six adults participated in a VE in a controlled experiment, designed to empirically test the model. Each participant performed a series of object assembly tasks. The task involved selecting, rotating, releasing, inserting and manipulating 3D objects. Initially, the results of factor analysis demonstrated dimensionality of two user perception measures and produced a set of empirical validated factors underlining the VE efficacy. The results of regression analysis revealed that SE had a significant positive effect on perceived VE efficacy. No significant effects were found of perceptions on performance and memory. Furthermore, the study provided insights into the relationships between the perception measures and performance measures for assessing the efficacy of VE training systems. The study also addressed how well users learn, perform, adapt to and perceive the VE training, which provides valuable insight into the system efficacy. Research and practical implications are presented at the end of the paper.

[1]  Woodrow Barfield,et al.  A Review of Presence and Performance in Virtual Environments , 2000, Int. J. Hum. Comput. Interact..

[2]  William F. Moroney,et al.  A comparison of two scoring procedures with the NASA task load index in a simulated flight task , 1992, Proceedings of the IEEE 1992 National Aerospace and Electronics Conference@m_NAECON 1992.

[3]  Robert L. Craig,et al.  Training and development handbook: A guide to human resource development , 1976 .

[4]  Domenico Pascarella,et al.  Dependability issues in visual-haptic interfaces , 2010, J. Vis. Lang. Comput..

[5]  Anand K. Gramopadhye,et al.  The Added Usefulness of Process Measures Over Performance Measures in Interface Design , 2005 .

[6]  P. Boulanger,et al.  Hapto-audio-visual environments for collaborative training of ophthalmic surgery over optical network , 2006, 2006 IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE 2006).

[7]  Roland Brünken,et al.  Memory characteristics and modality in multimedia learning: An aptitude–treatment–interaction study , 2009 .

[8]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[9]  R. Tobias.  An Introduction to Partial Least Squares Regression , 1996 .

[10]  Arthur D. Fisk,et al.  Measuring Skill Acquisition and Retention with an ATM Simulator: The Need for Age-Specific Training , 1998, Hum. Factors.

[11]  Neil Charness,et al.  Light Pen Use and Practice Minimize Age and Hand Performance Differences in Pointing Tasks , 2004, Hum. Factors.

[12]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[13]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[14]  Fei Wang,et al.  Generic and systematic evaluation of haptic interfaces based on testbeds , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Jim X. Chen,et al.  A Model for Understanding How Virtual Reality Aids Complex Conceptual Learning , 1999, Presence: Teleoperators & Virtual Environments.

[16]  Glyn Lawson,et al.  Designing touchpad user-interfaces for vehicles: which tasks are most suitable? , 2011, Behav. Inf. Technol..

[17]  R. Mayer,et al.  Interactive Multimodal Learning Environments , 2007 .

[18]  Viswanath Venkatesh,et al.  Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model , 2000, Inf. Syst. Res..

[19]  V. Bacharach,et al.  Psychometrics : An Introduction , 2007 .

[20]  Jonathan Crellin,et al.  Evaluating evaluation: an empirical examination of novel and conventional usability evaluation methods , 1990 .

[21]  H. Abdi.  Partial Least Squares (PLS) Regression. , 2003 .

[22]  Mel Slater,et al.  Using Presence Questionnaires in Reality , 2000, Presence: Teleoperators & Virtual Environments.

[23]  Satyandra K. Gupta,et al.  Towards the development of a virtual environment-based training system for mechanical assembly operations , 2007, Virtual Reality.

[24]  Doug A. Bowman,et al.  A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods , 2002, Presence: Teleoperators & Virtual Environments.

[25]  J. Pallant,et al.  Development and Validation of a Scale to Measure Perceived Control of Internal States , 2000, Journal of personality assessment.

[26]  Martha Reeves,et al.  Evaluation of Training , 1993 .

[27]  Steve Clarke,et al.  End User Computing Challenges and Technologies: Emerging Tools and Applications , 2007 .

[28]  Fred D. Davis,et al.  Improving Computer Training Effectiveness for Decision Technologies: Behavior Modeling and Retention Enhancement , 2001, Decis. Sci..

[29]  Saeid Nahavandi,et al.  Design and evaluation of a haptically enable virtual environment for object assembly training , 2009, 2009 IEEE International Workshop on Haptic Audio visual Environments and Games.

[30]  Ana-Maria Marhan,et al.  Virtual Reality-Based Environments for Learning and Training , 2008 .

[31]  Henry Been-Lirn Duh,et al.  Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment , 2002, Proceedings IEEE Virtual Reality 2002.

[32]  Bassam Hasan.  The Impact of Multilevel Computer Self-Efficacy on Effectiveness of Computer Training , 2008 .

[33]  Mansooreh Mollaghasemi,et al.  Usability engineering of virtual environments (VEs): identifying multiple criteria that drive effective VE system design , 2003, Int. J. Hum. Comput. Stud..

[34]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[35]  Yong-Ho Yoo.  MOTOR SKILL LEARNING WITH FORCE FEEDBACK IN MIXED REALITY , 2004 .

[36]  Magid Igbaria,et al.  The effects of self-efficacy on computer usage , 1995 .

[37]  Eric N. Wiebe,et al.  A Comparison of Two Mental Workload Instruments in Multimedia Instruction , 2006 .

[38]  Deborah Compeau,et al.  Intentions to Use Information Technologies: An Integrative Model , 2006, J. Organ. End User Comput..

[39]  Peter Young.  Virtual Reality Systems , 2007 .

[40]  Debbie Stone,et al.  User Interface Design and Evaluation , 2005 .

[41]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[42]  Benjamin Watson,et al.  VR for the Elderly: Quantitative and Qualitative Differences in Performance with a Driving Simulator , 1999, Cyberpsychology Behav. Soc. Netw..

[43]  William Winn,et al.  Learning in Artificial Environments: Embodiment, Embeddedness and Dynamic Adaptation , 2003 .

[44]  Malcolm Munro,et al.  Assessing User Competence: Conceptualization and Measurement , 2000, Inf. Syst. Res..

[45]  Mel Slater,et al.  Immersion, presence and performance in virtual environments: an experiment with tri-dimensional chess , 1996, VRST.

[46]  Cheri Ostroff.  TRAINING EFFECTIVENESS MEASURES AND SCORING SCHEMES: A COMPARISON , 2006 .

[47]  I. Ajzen,et al.  Attitude-behavior relations: A theoretical analysis and review of empirical research. , 1977 .

[48]  Donald E. Parker,et al.  User Experience Modeling and Enhancement for Virtual Environments That Employ Wide-Field Displays , 2007, HCI.

[49]  Fabrizia Mantovani,et al.  VR Learning : Potential and Challenges for the Use of 3 D Environments in Education and Training , 2001 .

[50]  Marilyn E. Gist.  Training design and pedagogy: Implications for skill acquisition, maintenance, and generalization. , 1997 .

[51]  R. X. Liu,et al.  Principal component regression analysis with SPSS , 2003, Comput. Methods Programs Biomed..

[52]  Thomas E. Whalen,et al.  Measuring the human side of virtual reality , 2003, IEEE International Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2003. VECIMS '03. 2003.

[53]  Deborah Compeau,et al.  Computer Self-Efficacy: Development of a Measure and Initial Test , 1995, MIS Q..

[54]  Kay M. Stanney,et al.  Handbook of Virtual Environments - Design, Implementation, and Applications, Second Edition , 2014, Handbook of Virtual Environments, 2nd ed..

[55]  Saeid Nahavandi,et al.  User-Centered Evaluation of a Virtual Environment Training System: Utility of User Perception Measures , 2009, HCI.

[56]  Xing-Dong Yang,et al.  Validating the Performance of Haptic Motor Skill Training , 2008, 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[57]  Lin Chai,et al.  To have or have not: an examination of feedback, learner control knowledge type in online learning , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[58]  Jochen Müsseler,et al.  Effects of sensorimotor transformations with graphical input devices , 2011, Behav. Inf. Technol..

[59]  Judy M. Vance,et al.  Development of a Dual-Handed Haptic Assembly System: SHARP , 2008, J. Comput. Inf. Sci. Eng..

[60]  Deborah Compeau,et al.  Application of Social Cognitive Theory to Training for Computer Skills , 1995, Inf. Syst. Res..

[61]  Michalis Nik Xenos,et al.  A study on how usability flaws in GUI design increase mouse movements and consequently may affect users' health , 2011, Behav. Inf. Technol..

[62]  A. Salmoni.  Motor skill learning. , 1989 .

[63]  Alistair G. Sutcliffe.  Multimedia and virtual reality - designing multisensory user interfaces , 2003 .

[64]  A. Bandura.  Self-Efficacy: The Exercise of Control , 1997, Journal of Cognitive Psychotherapy.

[65]  D. Kirkpatrick.  Techniques for evaluating training programs , 1979 .

[66]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[67]  C. Sue-Chan,et al.  Goal assignment and performance: Assessing the mediating roles of goal commitment and self-efficacy and the moderating role of power distance , 2002 .

[68]  D. Ausubel,et al.  Chapter V: Meaningful Learning and Retention: Intrapersonal Cognitive Variables , 1961 .

[69]  Fred D. Davis.  User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts , 1993, Int. J. Man Mach. Stud..

[70]  Mun Y. Yi,et al.  Predicting Computer Task Performance: Personal Goal and Self-Efficacy , 2004, J. Organ. End User Comput..

[71]  Wei Liu,et al.  Mixed Reality Systems for Learning: A Pilot Study Understanding User Perceptions and Acceptance , 2007, HCI.

[72]  Richard D. Johnson,et al.  An empirical investigation of sources of application-specific computer-self-efficacy and mediators of the efficacy - performance relationship , 2005, Int. J. Hum. Comput. Stud..

[73]  Arthur D. Fisk,et al.  Application of Cognitive Theory to Training and Design Solutions for Age-related Computer Use , 1999 .

[74]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[75]  Mahdi Tavakoli,et al.  A haptic interface for computer-integrated endoscopic surgery and training , 2006, Virtual Reality.

[76]  Liesbeth Kester,et al.  Dynamic task selection: Effects of feedback and learner control on efficiency and motivation , 2009 .

[77]  Alistair G. Sutcliffe,et al.  Presence, memory and interaction in virtual environments , 2005, Int. J. Hum. Comput. Stud..

[78]  Detmar W. Straub,et al.  Structural Equation Modeling and Regression: Guidelines for Research Practice , 2000, Commun. Assoc. Inf. Syst..

[79]  Daniel B. Willingham,et al.  A Neuropsychological Theory of Motor Skill Learning , 2004 .

[80]  Joseph L. Gabbard.  Usability Engineering of Virtual Environments , 2014, Handbook of Virtual Environments, 2nd ed..

[81]  Alistair G. Sutcliffe,et al.  Evaluating the usability of virtual reality user interfaces , 2000, Behav. Inf. Technol..

[82]  Marc M. Sebrechts,et al.  HANDBOOK OF VIRTUAL ENVIRONMENTS , 2014 .

[83]  Robert R. Whelan.  Neuroimaging of cognitive load in instructional multimedia , 2007 .

[84]  L. Sjodahl.  Evaluation of Training. , 1975 .

[85]  Fred D. Davis,et al.  Developing and Validating an Observational Learning Model of Computer Software Training and Skill Acquisition , 2003, Inf. Syst. Res..

[86]  S. Nahavandi,et al.  Haptically enabled interactive virtual reality prototype for general assembly , 2008, 2008 World Automation Congress.

[87]  B. Elango,et al.  The Effect of Attitudes, Goal Setting and Self-Efficacy on End User Performance , 2001, J. Organ. End User Comput..

[88]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[89]  David B. Kaber,et al.  Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation , 2004 .

[90]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[91]  Richard D. Johnson,et al.  Research Report: The Role of Behavioral Modeling in Computer Skills Acquisition: Toward Refinement of the Model , 2000, Inf. Syst. Res..

[92]  Michael J. Smith,et al.  Usability evaluation and interface design : cognitive engineering, intelligent agents and virtual reality , 2001 .

[93]  Thomas Hill,et al.  Role of efficacy expectations in predicting the decision to use advanced technologies: The case of computers. , 1987 .

[94]  John R. Wilson,et al.  Measurement of presence and its consequences in virtual environments , 2000, Int. J. Hum. Comput. Stud..

[95]  Richard D. Johnson,et al.  The Multilevel and Multifaceted Character of Computer Self-Efficacy: Toward Clarification of the Construct and an Integrative Framework for Research , 1998, Inf. Syst. Res..

[96]  Matthew Turk,et al.  Perceptual user interfaces , 2000 .

[97]  Marilyn E. Gist,et al.  Self-Efficacy: Implications for Organizational Behavior and Human Resource Management , 1987 .

[98]  D. Ausubel,et al.  Meaningful Learning and Retention: Intrapersonal Cognitive Variables , 1961 .

[99]  Hokyoung Ryu,et al.  Interaction Unit Analysis: A New Interaction Design Framework , 2009, Hum. Comput. Interact..

[100]  J. J. Gallagher.  Meaningful Learning and Retention: Intrapersonal Cognitive Variables , 1961 .

[101]  Fred D. Davis,et al.  A Model of the Antecedents of Perceived Ease of Use: Development and Test† , 1996 .

[102]  Robert S. Kennedy,et al.  Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. , 1993 .

[103]  Ing. A. O. Brugman.  Survey on Behavioral Observation Methods in Virtual Environments , 2003 .

[104]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[105]  Alvin Y. Wang,et al.  Predictors of web-student performance: the role of self-efficacy and reasons for taking an on-line class , 2002, Comput. Hum. Behav..

[106]  Abu Saleh Md. Mahfujur Rahman,et al.  Suitability of Searching and Representing Multimedia Learning Resources in a 3-D Virtual Gaming Environment , 2008, IEEE Transactions on Instrumentation and Measurement.