Developing an Assessment Model for Evaluating Software Tools in Education

This paper presents a research model to evaluate the usability of computer-mediated learning technology, user acceptance of the technology, and the effect of the technology on learning outcomes. We also introduce the use of innovative research techniques for the evaluation, including eye tracking and think-aloud protocol. Think-aloud protocol allows us to understand a user’s state of mind while he/she is interacting with the system. Eye tracking also provides additional insights into a user’s cognitive state while using the system. Both qualitative and quantitative data can be collected following the research model, thus providing more in-depth understanding on the effect of the system. We demonstrated the application of the research model and research techniques in evaluating visualization software designed in a project funded by the National Science Foundation.

[1]  C. C. Bissell.  Revitalising the engineering curriculum: the role of information technology , 1996 .

[2]  Jurek Kirakowski,et al.  What is Usability , 1991 .

[3]  Thomas S. Tullis,et al.  Gender Preferences in Web Design: Usability Testing through Eye Tracking , 2007, AMCIS.

[4]  Raduan Che Rose,et al.  Teachers' readiness to use technology in the classroom: an empirical study , 2008 .

[5]  Maarten van Someren,et al.  The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes , 1994 .

[6]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2003, Int. J. Hum. Comput. Interact..

[7]  Detmar W. Straub,et al.  Trust and TAM in Online Shopping: An Integrated Model , 2003, MIS Q..

[8]  Timothy A. Philpot,et al.  Comprehensive Assessment of a Software Development Project for Engineering Instruction , 2006 .

[9]  Jacquelyn F. Sullivan,et al.  The Integrated Teaching and Learning Program: A Pioneering Learning Environment for 21 st Century Engineering Education , 1998 .

[10]  I. Ajzen,et al.  Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research , 1977 .

[11]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[12]  Richard M. Felder,et al.  MATTERS OF STYLE , 2004 .

[13]  Albert L. Lederer,et al.  The technology acceptance model and the World Wide Web , 2000, Decis. Support Syst..

[14]  Nigel Bevan,et al.  International standards for HCI and usability , 2001, Int. J. Hum. Comput. Stud..

[15]  Samuli Kolari,et al.  Visualisation Promotes Apprehension and Comprehension , 2004 .

[16]  R. Felder,et al.  Applications, Reliability and Validity of the Index of Learning Styles* , 2005 .

[17]  Peter A. Todd,et al.  Understanding Information Technology Usage: A Test of Competing Models , 1995, Inf. Syst. Res..

[18]  R. Felder,et al.  Understanding Student Differences , 2005 .

[19]  R. Felder,et al.  Learning and Teaching Styles in Engineering Education. , 1988 .

[20]  Clayton Lewis,et al.  TASK-CENTERED USER INTERFACE DESIGN A Practical Introduction , 2006 .

[21]  Panos Markopoulos,et al.  A comparison of think-aloud, questionnaires and interviews for testing usability with children , 2002 .

[22]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[23]  Deborah I. Fels,et al.  Methods for inclusion: Employing think aloud protocols in software usability studies with individuals who are deaf , 2006, Int. J. Hum. Comput. Stud..

[24]  Denis Gillet,et al.  Collaborative web-based experimentation in flexible engineering education , 2005, IEEE Transactions on Education.

[25]  Raquel Benbunan-Fich.  Improving education and training with IT , 2002, CACM.