Evaluating Subject Matter Experts' Learning and Use of an ITS Authoring Tool

Because ITSs offer knowledge-based instruction, ITS authoring shells can use knowledge acquisition systems to bring courseware development within the reach of subject-matter experts unskilled in instructional design and development. The Experimental Advanced Instructional Design Advisor (XAIDA), a simple ITS shell, was evaluated in a study of 17 Air Force medical-technician trainers. The study examined XAIDA's learnability, usability, and acceptability to the participants using a one-week combined training and evaluation program. All participants were able to produce courseware using XAIDA. Measures of cognitive structure indicated that their conception of XAIDA's features became more similar to that of an expert as the result of training. Performance tests, skill ratings and usability ratings indicated that XAIDA is both usable and learnable. Participant's attitudes were generally positive except at a point when training required a difficult cognitive shift. Attitudes remained focused througout training on personal and low-level concerns, indicating the participants were not likely to be early adapters of the technology. The results have implications for the deployment of ITS shells in operational environments.

[1]  J. Michael Spector,et al.  Automating instructional design : concepts and issues , 1993 .

[2]  D. Bailey,et al.  Qualities of the Stages of Concern Questionnaire and Implications for Educational Innovations , 1992 .

[3]  E. Salas,et al.  Toward an Integration of Training Theory and Technique , 1991 .

[4]  G. Hall.  The Concerns-Based Approach to Facilitating Change. , 1979 .

[5]  Willian Geoghegan,et al.  Whatever Happened to Instructional Technology , 1994 .

[6]  Henry M. Halff,et al.  Four Easy Pieces: Development Systems for Knowledge-Based Generative Instruction , 1999 .

[7]  Frank Linton,et al.  Using evaluation in the design of an intelligent tutoring system , 1997 .

[8]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[9]  Susan Palmiter,et al.  Animated Demonstrations for Learning Procedural Computer-Based Tasks , 1993, Hum. Comput. Interact..

[10]  James M. Royer,et al.  Techniques and Procedures for Assessing Cognitive Skills , 1993 .

[11]  Nancy J. Cooke,et al.  Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing , 1996 .

[12]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[13]  Roger W. Schvaneveldt,et al.  Pathfinder associative networks: studies in knowledge organization , 1990 .

[14]  Eduardo Salas,et al.  Measuring Knowledge Organization as a Method for Assessing Learning during Training , 1995, Hum. Factors.

[15]  J. M. Spector.  Designing and Developing an Advanced Instructional Design Advisor , 1990 .