Motivation, Usability and Their Interrelationships in a Self-paced Online Learning Environment

This study addressed how usability improvement and motivational design affect learners’ motivation and learning performance in a self-paced, online learning environment. The study also investigated the interrelationships between commonly-used usability measures and the motivation measures based on Keller’s ARCS model. A two-phase study approach was used. In Phase I, an existing self-paced, online safety training tutorial was used as the baseline. Two alternative designs were developed with improved usability and motivational design based on the ARCS model. In Phase II, the effects of the three interface designs were evaluated through a three-group, generalized randomized block covariate design experiment. A total of seventy-two college students (48 males and 24 females) participated in the usability testing and the online training session using one of the interface designs. Both quantitative and qualitative data were collected and analyzed. Results suggested a significant interface design effect on learner motivation. Learners who used the interface design with both usability improvement and motivational design applied (the UM group) showed the highest level of motivation. In particular, the attention level of the learners in the UM group was significantly higher than the baseline group. Results also indicated motivation differences between genders. Females showed higher scores than males in overall motivation score and in each of the four subscales of attention, relevance, confidence, and satisfaction. No significant difference in learning performance was found among the three treatment groups using different interface designs. None of the usability or motivation measures was a significant predictor of learning performance. Small to medium positive correlations were found between usability satisfaction and three motivation measures, i.e., attention, relevance and satisfaction. Content analysis identified a number of interface design components to be relevant to learners’ motivation components: overall appearance, graphics/multimedia, text appearance, page layout, navigation, and paging/scrolling. Implications and design recommendations for online tutorial interface design were discussed. Additional discussion was provided regarding the online learning environment and the integration of usability, motivation, and instructional design and technology.

[1]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[2]  Lih-Juan ChanLin.  A Case for Assessing Motivation from Learning a Computer-Assisted Instruction. , 1994 .

[3]  Robert G. Main.  Integrating Motivation into the Instructional Design Process. , 1993 .

[4]  Lyman W. Porter,et al.  Managerial Attitudes and Performance. , 1969 .

[5]  D. H. Lim,et al.  Motivation and Learner Characteristics Affecting Online Learning and Learning Application , 2003 .

[6]  Vincent G. Duffy.  Toward Online Virtual Safety Training in the Manufacturing and Service Industries , 2000 .

[7]  L. Price,et al.  Gender differences and similarities in online courses: challenging stereotypical views of women , 2006, J. Comput. Assist. Learn..

[8]  T. Karsenti,et al.  The Influence of Gender Differences on Within-Term Changes in Junior-College Student Motivation. , 1994 .

[9]  B. Weiner.  Achievement Motivation and Attribution Theory , 1974 .

[10]  John M. Keller,et al.  Motivation and instructional design: A theoretical perspective , 1979 .

[11]  Andrew Gemino,et al.  Investigating Coherence and Multimedia Effects of a Technology-Mediated Collaborative Environment , 2006, J. Manag. Inf. Syst..

[12]  Lisa Gualtieri,et al.  Designing usable, self-paced e-learning courses: a practical guide , 2006, ELERN.

[13]  J. Keller.  Using the ARCS Motivational Process in Computer-Based Instruction and Distance Education , 1999 .

[14]  J. Rotter.  Generalized expectancies for internal versus external control of reinforcement. , 1966, Psychological monographs.

[15]  William A. Horton,et al.  Designing Web-Based Training , 2000 .

[16]  L. L Lohr,et al.  Designing the instructional interface , 2000 .

[17]  Szu-Yuan Sun,et al.  Usability, quality, value and e-learning continuance decisions , 2005, Comput. Educ..

[18]  Sang Ho Song,et al.  Research Issues of Motivation in Web-based Instruction , 2000 .

[19]  Stefanie B. Waschull.  Predicting Success in Online Psychology Courses: Self-Discipline and Motivation , 2005 .

[20]  Michael Feldstein.  What is usable e-learning? , 2002, ELERN.

[21]  Debra L. Shapiro,et al.  The Future of Work Motivation Theory , 2004 .

[22]  Daniel C. A. Hillman,et al.  Learner-Interface Interaction in Distance Education: An Extension of Contemporary Models and Strategies for Practitioners , 1994 .

[23]  Marion Goldstein,et al.  Web-based nutrition education for college students: Is it feasible? , 2006, Evaluation and program planning.

[24]  D. Cartwright.  Social Learning and Clinical Psychology. , 1955 .

[25]  Tonya L. Smith-Jackson,et al.  Journal of Information Technology Education Web-based Learning Environment: a Theory-based Design Process for Development and Evaluation Web-based Learning Environment Background Overview of Gps Education , 2022 .

[26]  Wilhelmina C. Savenye,et al.  Applying Social Cognitive Constructs of Motivation To Enhance Student Success in Online Distance Education. , 2003 .

[27]  Raymond J. Wlodkowski.  Enhancing adult motivation to learn : a guide to improving instruction and increasing learner achievement , 1987 .

[28]  Wei‐Tao Tai.  Effects of training framing, general self‐efficacy and training motivation on trainees' training effectiveness , 2006 .

[29]  J. Keller.  The Arcs Model of Motivational Design , 2010 .

[30]  Slava Kalyuga,et al.  The Expertise Reversal Effect , 2003 .

[31]  Kenneth G. Brown,et al.  A field study of employee e-learning activity and outcomes , 2005 .

[32]  L. Gomez,et al.  Motivation in Project-based Classrooms: New Measures Better Coupled to Students' Experiences , 2006, ICLS.

[33]  D. Schunk,et al.  Risk Taking: Theoretical, Empirical, and Educational Considerations , 1991 .

[34]  S. Parker.  Work and the Nature of man , 1969 .

[35]  Sanna Järvelä,et al.  The changes in learning theory and the topicality of the recent research on motivation , 1999 .

[36]  Etienne Wenger,et al.  Situated Learning: Legitimate Peripheral Participation , 1991 .

[37]  Raymond J. Wlodkowski.  Motivation and Diversity: A Framework for Teaching , 1999 .

[38]  Patricia G. Wojahn,et al.  Computer-Mediated Communication: The Great Equalizer between Men and Women?. , 1994 .

[39]  Thomas C. Reeves,et al.  Usability and Instructional Design Heuristics for E-Learning Evaluation , 2002 .

[40]  P. Pintrich,et al.  Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq) , 1993 .

[41]  Charles W. Davidson,et al.  The Effects of Easy-Listening Background Music on the On-Task-Performance of Fifth-Grade Children , 1986 .

[42]  J. Saylor,et al.  Usability, learning, and subjective experience: user evaluation of K-MODDL in an undergraduate class , 2004, Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004..

[43]  A. Bandura.  Self-efficacy: toward a unifying theory of behavioral change. , 1977, Psychology Review.

[44]  M. Olson,et al.  Enhancing Adult Motivation to Learn: A Comprehensive Guide for Teaching All Adults , 2010 .

[45]  Joanna Wolfe,et al.  Why do women feel lgnored? gender differences in computer-mediated classroom interactions , 1999 .

[46]  Mark Notess,et al.  Usability, user experience, and learner experience , 2001, ELERN.

[47]  Brad Mehlenbacher.  Assessing the usability of on-line instructional materials , 2002 .

[48]  D. Schunk,et al.  Self-Efficacy and Academic Motivation , 2002 .

[49]  W. McKeachie,et al.  The Making of the Motivated Strategies for Learning Questionnaire , 2005 .

[50]  Nishikant Sonwalkar,et al.  A New Methodology for Evaluation: The Pedagogical Rating of Online Courses. , 2002 .

[51]  Tjeerd Plomp,et al.  Development research applied to improve motivation in distance education , 1999 .

[52]  Martyn Wild,et al.  Motivation and interface design: maximising learning opportunities , 1998, J. Comput. Assist. Learn..

[53]  Sebastiano Bagnara,et al.  Multimedia systems in distance education: effects of usability on learning , 1999, Interact. Comput..

[54]  Bruce W. Speck,et al.  Assessment strategies for the on-line class : from theory to practice , 2002 .

[55]  Robert Irizarry,et al.  Self-Efficacy & Motivation Effects on Online Psychology Student Retention. , 2002 .

[56]  J. Brophy,et al.  Conceptualizing student motivation , 1983 .

[57]  Panagiotis Zaharias.  Usability and e-learning: the road towards integration , 2004, ELERN.

[58]  Donald R. Brown,et al.  Gender Differences in Values and Their Impact on Academic Achievement. , 1987 .

[59]  Dannelle D. Stevens,et al.  Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning , 2004 .

[60]  P. Pintrich,et al.  Motivation in Education: Theory, Research, and Applications , 1995 .

[61]  Adams Js.  Towards an understanding of inequity , 1963 .

[62]  Michael Allen.  Adventures with "Robin Hood": Gender and Conflict on a First-Year Bulletin Board , 1994 .

[63]  Marilyn P. Arnone,et al.  Awesome Web Sites: How To Find and Use Them. , 2001 .

[64]  Cynthia L. Selfe,et al.  The Politics of the Interface: Power and Its Exercise in Electronic Contact Zones. , 1994 .

[65]  E. A. Locke.  Toward a theory of task motivation and incentives , 1968 .

[66]  Sherria L. Hoskins,et al.  Motivation and ability: which students use online learning and what influence does it have on their achievement? , 2005, Br. J. Educ. Technol..

[67]  Young Sam Ryu,et al.  Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods , 2005 .

[68]  J Hogge,et al.  Arrive online®: a web based road safety and injury prevention resource , 2005 .

[69]  Frances A. Clem.  Culture and Motivation in Online Learning Environments , 2004 .

[70]  J. Eccles.  Expectancies, values and academic behaviors , 1983 .

[71]  P.K. Imbrie,et al.  Learning motivation evaluation for a computer-based instructional tutorial using ARCS model of motivational design , 2004, 34th Annual Frontiers in Education, 2004. FIE 2004..

[72]  U. Neisser,et al.  Language-dependent recall of autobiographical memories. , 2000, Journal of experimental psychology. General.

[73]  Panagiotis Zaharias,et al.  A usability evaluation method for e-learning: focus on motivation to learn , 2006, CHI Extended Abstracts.

[74]  Wei-Chi Tsai,et al.  Perceived importance as a mediator of the relationship between training assignment and training motivation , 2003 .

[75]  J. Keller,et al.  An application of the ARCS Model of Motivational Design. , 1987 .

[76]  Martyn Sloman,et al.  The E-Learning Revolution: How Technology is Driving a New Training Paradigm , 2002 .

[77]  J. Keller.  Strategies for stimulating the motivation to learn , 1987 .

[78]  M F Seabrook,et al.  Motivation and performance. , 1982, Ergonomics.

[79]  Larry F. Hodges,et al.  Can a virtual cat persuade you?: the role of gender and realism in speaker persuasiveness , 2006, CHI.

[80]  P. Pintrich.  A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). , 1991 .

[81]  Rita Pavolka,et al.  From waterfall to rapid prototyping: supporting enterprise-wide adoption of the oncourse collaboration and learning (CL) environment at Indiana University , 2005, SIGUCCS '05.

[82]  Charalambos Vrasidas.  Issues of pedagogy and design in e-learning systems , 2004, SAC '04.

[83]  Toru Ishida,et al.  Cross-cultural study of avatar expression interpretations , 2006, International Symposium on Applications and the Internet (SAINT'06).

[84]  Lisa Gualtieri,et al.  The basics of e-learning: an excerpt from handbook of human factors in web design , 2005, ELERN.

[85]  Susan Hallam,et al.  The Effects of Background Music on Primary School Pupils' Task Performance , 2002 .

[86]  A. Maslow.  Motivation and Personality , 1954 .

[87]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[88]  Samer Khasawneh,et al.  A Path Analytic Study of the Determinants of College Students' Motivation to Use Online Learning Technologies. , 2004 .

[89]  J. Eccles,et al.  In the Mind of the Actor: The Structure of Adolescents' Achievement Task Values and Expectancy-Related Beliefs , 1995 .

[90]  Curtis J. Bonk,et al.  Surveying the future of workplace e-learning: the rise of blending, interactivity, and authentic learning , 2005, ELERN.

[91]  Andrew Dillon,et al.  Designing web-based instruction: A human-computer interaction perspective , 1997 .

[92]  Donald G. Perrin.  Creative Online Learning Environments , 2005 .

[93]  S. G. Brainard,et al.  A longitudinal study of undergraduate women in engineering and science , 1997, Proceedings Frontiers in Education 1997 27th Annual Conference. Teaching and Learning in an Era of Change.

[94]  J. Colquitt,et al.  Toward an integrative theory of training motivation: a meta-analytic path analysis of 20 years of research. , 2000, The Journal of applied psychology.

[95]  F. Paas.  Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. , 1992 .

[96]  J. Keller.  Development and use of the ARCS model of instructional design , 1987 .

[97]  Christian Rauh,et al.  The Influence of the Avatar on Online Perceptions of Anthropomorphism, Androgyny, Credibility, Homophily, and Attraction , 2005, J. Comput. Mediat. Commun..

[98]  Henry L. Tosi.  A Theory of Goal Setting and Task Performance , 1991 .

[99]  Khaled Sabry,et al.  Evaluation of the interactivity of web-based learning systems: principles and process , 2003 .

[100]  James O. Carey,et al.  The systematic design of instruction , 1978 .

[101]  P. Pintrich.  The role of goal orientation in self-regulated learning. , 2000 .

[102]  Brian Shackel,et al.  Usability - Context, framework, definition, design and evaluation , 1991, Interact. Comput..

[103]  William Y. Lan,et al.  The relationship between problems with technology and graduate students' evaluations of online teaching , 2005, Internet High. Educ..

[104]  Linda L. Lohr,et al.  Learner-Centered Usability: Tools for Creating a Learner-Friendly Instructional Environment. , 2001 .

[105]  Kenneth S Bordens,et al.  Instructor's Manual to Accompany Research Design and Methods: A Process Approach Methods: A Process Approach (3rd ed.) , 1996 .

[106]  Michael Griffin.  Background Music and the Learning Environment: Borrowing from other Disciplines. , 2006 .

[107]  C. Alderfer.  Existence, Relatedness, and Growth; Human Needs in Organizational Settings , 1972 .

[108]  A. Kitsantas,et al.  Supporting Self-Regulation in Student-Centered Web-Based Learning Environments , 2002 .

[109]  A. Bandura.  Self-efficacy. I: V. S. Ramachandran (red.), Encyclopedia of human behavior (vol. s. ). New York: Academic Press. , 1994 .

[110]  Terri Gullickson,et al.  Encyclopedia of human behavior , 1995 .

[111]  Teresa K. Debacker,et al.  Extending the Traditional Classroom Through Online Discussion: The Role of Student Motivation , 2006 .

[112]  F. W. Irwin,et al.  Motivation and performance. , 1961, Annual review of psychology.

[113]  Janet G. Donald.  Motivation for Higher‐Order Learning , 1999 .

[114]  Hermann Astleitner,et al.  The Effects of Situation-Outcome-Expectancies and of ARCS-Strategies on Self-Regulated Learning with Web-Lectures , 2003 .

[115]  S. Patrick.  GENDER DIFFERENCES AND THE ONLINE CLASSROOM: MALE AND FEMALE COLLEGE STUDENTS EVALUATE THEIR EXPERIENCES , 2001 .

[116]  Harold E. Fiske.  Music Matters: A New Philosophy of Music Education , 1997 .

[117]  A. Hamid.  e-Learning: Is it the "e" or the learning that matters? , 2001, Internet High. Educ..

[118]  Kar Yan Tam,et al.  Does Animation Attract Online Users' Attention? The Effects of Flash on Information Search Performance and Perceptions , 2004, Inf. Syst. Res..

[119]  Kyong-Jee Kim.  Motivational Influences in Self-Directed Online Learning Environments: A Qualitative Case Study , 2005 .

[120]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[121]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[122]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[123]  S. Thompson.  Social Learning Theory , 2008 .

[124]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[125]  Yu-Hui Tao,et al.  The design and the formative evaluation of a web-based course for simulation analysis experiences , 2004, Comput. Educ..

[126]  Heidi A. Diefes-Dux,et al.  A preliminary validation of Attention, Relevance, Confidence and Satisfaction model-based Instructional Material Motivational Survey in a computer-based tutorial setting , 2006, Br. J. Educ. Technol..

[127]  H. Klein,et al.  MOTIVATION TO LEARN AND COURSE OUTCOMES: THE IMPACT OF DELIVERY MODE, LEARNING GOAL ORIENTATION, AND PERCEIVED BARRIERS AND ENABLERS , 2006 .

[128]  V. Vroom.  Work and motivation , 1964 .

[129]  Patrick M. Wright,et al.  Training Motivation in Organizations: An Analysis of Individual-Level Antecedents [*] , 2000 .

[130]  Roy M. Bohlin.  Factor Analyses of the Instructional Motivation Needs of Adult Learners. , 1993 .

[131]  Dan S. Chiaburu,et al.  Individual and contextual influences on multiple dimensions of training effectiveness , 2005 .

[132]  Thérèse Bouffard,et al.  The impact of goal orientation on self‐regulation and performance among college students , 1995 .

[133]  Eddie Blass,et al.  E‐learner experiences: what is the future for e‐learning? , 2006 .

[134]  J. Keller.  Motivational Design of Instruction , 1983 .

[135]  Charles M. Reigeluth.  Instructional theories in action : lessons illustrating selected theories and models , 2009 .

[136]  S. Handel.  Listening As Introduction to the Perception of Auditory Events , 1989 .

[137]  Does Background Music Impact Computer Task Performance , 2008 .

[138]  Charles B. Hodges.  Designing to Motivate: Motivational Techniques to Incorporate in E-Learning Experiences , 2004 .

[139]  G. VandenBos.  APA dictionary of psychology , 2007 .

[140]  G. Huang,et al.  Entry and Persistence of Women and Minorities in College Science and Engineering Education. , 2000 .

[141]  G. Dobbins,et al.  Exploratory Field Study of Training Motivation , 1993 .

[142]  Christian Werner,et al.  Designing Online Instruction for Success: Future Oriented Motivation and Self-Regulation. , 2007 .

[143]  Thomas Fox McManus,et al.  Individualizing instruction in a web-based hypermedia learning environment: nonlinearity, advance organizers, and self-regulated learners , 2000 .

[144]  Linda Serra Hagedorn,et al.  A Social Cognitive Construct Validation: Determining Women's and Men's Success in Engineering Programs , 2007 .

[145]  Lucy Suchman.  Plans and situated actions: the problem of human-machine communication , 1987 .

[146]  Malcolm L. Fleming,et al.  Instructional Message Design: Principles from the Behavioral and Cognitive Sciences , 1993 .

[147]  John M. Keller,et al.  The systematic process of motivational design , 1987 .

[148]  Tung Bui,et al.  Impact of Learning Strategies and Motivation on Performance: A Study in Web-Based Instruction. , 2001 .

[149]  Vic Lally,et al.  Gender differences in an on-line learning environment , 1999, J. Comput. Assist. Learn..

[150]  Manuscript title Adult learners and academic achievement: The roles of self-efficacy, self-regulation, and motivation Author's contact information , 2006 .

[151]  John M. Keller,et al.  Effectiveness of motivationally adaptive computer-assisted instruction on the dynamic aspects of motivation , 2001 .

[152]  Kenneth A Johnson,et al.  A Job Safety Program for Construction Workers Designed to Reduce the Potential for Occupational Injury Using Tool Box Training Sessions and Computer-Assisted Biofeedback Stress Management Techniques , 2002, International journal of occupational safety and ergonomics : JOSE.

[153]  Michael Simonson.  Online training in an online world , 2002 .

[154]  Hermann Astleitner,et al.  A Model for Motivationally Adaptive Computer-Assisted Instruction , 1995 .

[155]  Dean R. Spitzer,et al.  Motivation: The Neglected Factor in Instructional Design. , 1996 .

[156]  Jennifer Preece,et al.  Predicting quality in educational software: Evaluating For Learning, Usability and the Synergy between Them , 1999, Interact. Comput..

[157]  Irwin L. Goldstein,et al.  Training in organizations: Needs assessment, development, and evaluation , 1986 .

[158]  Raymond MacDonald,et al.  The effect of background music and background noise on the task performance of introverts and extraverts , 2007 .

[159]  Ching-Chun Shih,et al.  Web-based Learning: Relationships among Student Motivation, Attitudes, Learning Styles, and Achievement. , 2001 .

[160]  Michael B. Paulsen,et al.  Student Motivation and Epistemological Beliefs , 1999 .

[161]  Choenjit Rumpradit.  An evaluation of the effect of user interface elements and user learning styles on user performance, confidence and satisfaction on the world wide web , 1999 .

[162]  Donna M. Gabrielle,et al.  Effects of Technology-Mediated Instructional Strategies on Motivation, Performance, and Self-Directed Learning , 2003 .

[163]  Mei Wang,et al.  Evaluating the usability of Web-based learning tools , 2002, J. Educ. Technol. Soc..

[164]  Toni L. Doolen,et al.  A usability comparison of PDA-based quizzes and paper-and-pencil quizzes , 2005, Comput. Educ..

[165]  G. Gibbs,et al.  Conditions Under Which Assessment Supports Students’ Learning , 2005 .

[166]  S. Chyung.  Systematic and systemic approaches to reducing attrition rates in online higher education , 2001 .

[167]  Phyllis Tharenou.  THE RELATIONSHIP OF TRAINING MOTIVATION TO PARTICIPATION IN TRAINING AND DEVELOPMENT. , 2001 .

[168]  J. Peter Kincaid,et al.  Simulation techniques for training emergency response , 2003 .

[169]  R. Bauer,et al.  State-dependent learning in older and younger adults. , 1997, Psychology and aging.

[170]  John Keller,et al.  Learner motivation and E-learning design: A multinationally validated process , 2004 .

[171]  Paula M. Bach,et al.  Usability and learning in a speech-enabled reading tutor: a field study , 2006, CHI EA '06.

[172]  Geri Gay,et al.  The laptop and the lecture: The effects of multitasking in learning environments , 2003, J. Comput. High. Educ..

[173]  Yehudi Menuhin,et al.  The music of man , 1979 .

[174]  Edward L. Meyen,et al.  The Online Academy Formative Evaluation Approach to Evaluating Online Instruction , 2002 .

[175]  A. Friederici,et al.  Investigating emotion with music: An fMRI study , 2006, Human brain mapping.

[176]  Doo H. Lim,et al.  Cross Cultural Differences in Online Learning Motivation , 2004 .

[177]  T Cockerton,et al.  Cognitive Test Performance and Background Music , 1997, Perceptual and motor skills.

[178]  I. E. Allen,et al.  Making the Grade: Online Education in the United States, 2006. , 2006 .

[179]  Rebecca Kathryn Spiros.  Individual differences in motivation during distance training: The influence of goal orientation and self-efficacy on learning outcomes. , 2003 .

[180]  Sandra J. Milberg,et al.  Arousal cues arousal-related material in memory: implications for understanding effects of mood on memory , 1983 .

[181]  Dennis Adams,et al.  Wireless laptops in the classroom (and the Sesame Street syndrome) , 2006, CACM.