A Strategy for Evaluating Web-Based Discretionary Decision Support Systems

The World Wide Web facilitates user access to knowledge-based decision support systems. Such web-enabled systems can provide users with advice about how decision-makers exercise discretion. GetAid, developed using the web-based shell environment WebShell, is an example of a web-based decision support system operating in a discretionary legal domain. This paper presents the Context, Criteria, Contingency evaluation framework for knowledge- based systems, general in design but geared towards the evaluation of legal knowledge-based systems. Central to this framework is a hierarchical model of evaluation criteria arranged in four quadrants: verification and validation, user credibility, technical infrastructure and the impact of the system upon its environment. This framework frames an evaluation both in terms of the context of use of the system and the context of its evaluation and includes guidelines for the selection of appropriate evaluation criteria under differing contingencies. A case study is presented describing the use of this evaluation framework in planning the evaluation of the web-deployed GetAid system.

[1]  Frank F. Land,et al.  The moving staircase - Problems of appraisal and evaluation in a turbulent environment , 1999, Inf. Technol. People.

[2]  Andrew Stranieri,et al.  The evaluation of legal knowledge based systems , 1999, ICAIL '99.

[3]  E. Shortliffe.  Computer-based medical consultations: mycin (elsevier north holland , 1976 .

[4]  S. Toulmin.  The uses of argument , 1960 .

[5]  John Zeleznikow,et al.  Acknowledging insufficiency in the evaluation of legal knowledge-based systems: strategies towards a broadbased evaluation model , 2001, ICAIL '01.

[6]  Barbara Ann Kitchenham.  Evaluating software engineering methods and tool—part 2: selecting an appropriate evaluation method—technical criteria , 1996, SOEN.

[7]  KitchenhamBarbara Ann,et al.  Evaluating software engineering methods and tools , 1998 .

[8]  Michael Jackson,et al.  Evaluation Methodologies: A System for Use , 1992 .

[9]  Michael Goul,et al.  Validating expert systems , 1990, IEEE Expert.

[10]  Andrew Stranieri,et al.  Argumentation structures that integrate dialectical and non-dialectical reasoning , 2001, The Knowledge Engineering Review.

[11]  Denis Borenstein,et al.  Towards a practical method to validate decision support systems , 1998, Decis. Support Syst..

[12]  Pierre N. Robillard,et al.  A Software System Evaluation Framework , 1995, Computer.

[13]  Walter Woolfolk,et al.  The problem of the dynamic organization and the static system: principles and techniques for achieving flexibility , 1996, Proceedings of HICSS-29: 29th Hawaii International Conference on System Sciences.

[14]  Edward H. Shortliffe,et al.  Computer-based medical consultations, MYCIN , 1976 .

[15]  Brian E. Mennecke,et al.  The Problem of the Dynamic Organization and the Static System: Principles and Techniques for Achieving Flexibility , 1996, HICSS.

[16]  Andrew Stranieri,et al.  A hybrid rule – neural approach for the automation of legal reasoning in the discretionary domain of family law in Australia , 1999, Artificial Intelligence and Law.

[17]  Dustin Huntington,et al.  Web-based expert systems are on the way: Java-based Web delivery , 2000 .

[18]  Steve Smithson,et al.  Rethinking the approaches to information systems investment evaluation , 1999 .

[19]  Harald Reiterer,et al.  Software evaluation using the 9241 evaluator , 1997, Behav. Inf. Technol..