Development of a User Model Evaluation Technique for Hypermedia Based Interfaces

The model underlying the interface and its interpretation by the user is a key element of successful interface usage (Chignell and Hancock, 1986; Norman, 1986). Some of the critical characteristics of the system model that determine its effectiveness are: 1) the proximity between the user's model and the system model, 2) how well the system model is matched to its application, and 3) how easily the system model lends itself to navigation through the interface. Several interesting issues arise from studying these system model characteristics. One of these is how the system model can be built in order to meet the needs of all users. Ideally, a system model must be flexible to meet the requirements of users from varied knowledge bases and computing experience, yet must be specific enough to perform its function in an efficient manner. Evaluation of human computer interfaces has traditionally been limited to determining the effectiveness of linear text and graphics. Evaluation tools have generally been formulated as checklists which assess predominantly static characteristics of interfaces either through user (e.g., Chin, Diehl, and Norman, 1988) or expert evaluation (Hamel and Clark, 1986). The use of a hypermedia interface, with its many nested levels and pathways, may produce varied user models. Thus it is particularly important to evaluate a user's perceived system model versus the actual system model in order to determine the effectiveness and deficiencies of the interface when it involves a hypermedia application. Traditional evaluation techniques include user evaluation and feedback, analysis of user learning, analysis of user interface usage patterns. However, modifications to these traditional methods must be considered to meet the different challenges that a hypermedia interface presents. Methods of evaluation are needed for hypermedia, and for other applications where the human factors characteristics of the interface are not obvious from an assessment of its static characteristics. Through comparison of the user's perceived system model versus the actual system model it is possible to evaluate the effectiveness of a hypermedia interface in large part, and identify obvious deficiencies. The interface designer can then determine why idiosyncratic or inappropriate user models occurred and then alter the interface or user thinking (through training or online help) to improve the interface. This paper discusses two experiments carried out at the University of Southern California using the Jefferson prototype interface (a computer assisted assignment/online retrieval tool). These experiments use a novel method of evaluating the proximity between student models of the interface and the intended system model through the comparison of hierarchies built by students before and after using the software.