Brittleness in the design of cooperative problem-solving systems: the effects on user performance

One of the critical problems in the design and use of advanced decision-support systems is their potential "brittleness". This brittleness can arise because of the inability of the designer to anticipate and design for all of the scenarios that could arise during the use of the system. The typical "safety valve" to deal with this problem is to keep a person "in the loop", requiring that person to apply his/her expertise in making the final decision on what actions to take. This paper provides empirical data on how the role of the decision support system can have a major impact on the effectiveness of this design strategy. Using flight planning for commercial airlines as a testbed, three alternative designs for a graphical flight planning tool were evaluated, using 27 dispatchers and 30 pilots as subjects. The results show that the presentation of a suggestion or recommendation by the computer early in the person's own problem evaluation can have a significant impact on that person's decision processes, influencing situation assessment and the evaluation of alternative solutions.

[1]  Lucy Suchman.  Plans and situated actions: the problem of human-machine communication , 1987 .

[2]  David Chapman,et al.  What are plans for? , 1990, Robotics Auton. Syst..

[3]  William B. Rouse,et al.  Studies of planning behavior of aircraft pilots in normal, abnormal, and emergency situations , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  D. Isenberg.  Thinking and Managing: A Verbal Protocol Analysis of Managerial Problem Solving , 1986 .

[5]  James L. Alty,et al.  Expert Systems: An Alternative Paradigm , 1984, Int. J. Man Mach. Stud..

[6]  Kevin B. Bennett,et al.  Human Interaction with an "Intelligent" Machine , 1987, Int. J. Man Mach. Stud..

[7]  Paul E. Lehner,et al.  Cognitive Factors in User/Expert-System Interaction , 1987 .

[8]  G. Miller,et al.  Plans and the structure of behavior , 1960 .

[9]  Ernest Davis,et al.  A logical framework for commonsense predictions of solid object behaviour , 1988, Artif. Intell. Eng..

[10]  J. B. Black,et al.  Cognition, computing, and cooperation , 1990 .

[11]  Earl D. Sacerdoti,et al.  Planning in a Hierarchy of Abstraction Spaces , 1974, IJCAI.

[12]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987 .

[13]  John W. Payne,et al.  Task complexity and contingent processing in decision making: An information search and protocol analysis☆ , 1976 .

[14]  E M Roth,et al.  Cognitive simulation as a tool for cognitive task analysis. , 1992, Ergonomics.

[15]  Philip J. Smith,et al.  Design concepts for the development of cooperative problem-solving systems , 1992 .

[16]  Stephen J. Andriole,et al.  Handbook of Decision Support Systems , 1989 .

[17]  H. Simon,et al.  A Behavioral Model of Rational Choice , 1955 .

[18]  G. Klein,et al.  A recognition-primed decision (RPD) model of rapid decision making. , 1993 .

[19]  John M. Carroll,et al.  Interface design issues for advice-giving expert systems , 1987, CACM.

[20]  P J Smith,et al.  An empirical evaluation of the performance of antibody identification tasks , 1991, Transfusion.

[21]  Philip J. Smith,et al.  Design of a Cooperative Problem-Solving System for En-Route Flight Planning: An Empirical Evaluation , 1994 .

[22]  Christine M. Mitchell,et al.  Use of Model-Based Qualitative Icons and Adaptive Windows in Workstations for Supervisory Control Systems , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[23]  John L. Bennett,et al.  Learning from user experience with groupware , 1990, CSCW '90.

[24]  Robert Thierauf,et al.  User-oriented decision support systems: Accent on problem finding , 1988 .

[25]  Barbara Hayes-Roth,et al.  A Cognitive Model of Planning , 1979, Cogn. Sci..

[26]  M. McCloskey.  Naive Theories of Motion. , 1982 .

[27]  Randall Davis,et al.  Model-based reasoning: troubleshooting , 1988 .

[28]  Ian Benson,et al.  Some social and economic consequences of groupware for flight crew , 1990, CSCW '90.

[29]  Philip J. Smith,et al.  A Catalog of Errors , 1992, Int. J. Man Mach. Stud..

[30]  G. Miller,et al.  Plans and the structure of behavior , 1960 .

[31]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[32]  Mark Stefik,et al.  Planning with Constraints (MOLGEN: Part 1) , 1981, Artif. Intell..

[33]  Jonathan Grudin,et al.  CSCW: the convergence of two development contexts , 1991, CHI.

[34]  Allan Collins,et al.  Misconceptions in student's understanding , 1979 .

[35]  R. Wilensky.  Planning and Understanding: A Computational Approach to Human Reasoning , 1983 .

[36]  George A. Sexton,et al.  Diverter Decision Aiding for In-Flight Diversions , 1990 .

[37]  John A. Sorensen.  Generation and Evaluation of Near-Optimum Vertical Flight Profiles , 1983, 1983 American Control Conference.

[38]  Janet L. Kolodner,et al.  Using Experience in Clinical Problem Solving: Introduction and Framework , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[39]  Nancy G. Hall.  Diagnosing problems with the user interface for a strategic planning fuzzy DSS , 1988, IEEE Trans. Syst. Man Cybern..

[40]  Etienne Wenger,et al.  Artificial Intelligence and Tutoring Systems: Computational and Cognitive Approaches to the Communication of Knowledge , 1987 .

[41]  A. Tversky.  Elimination by aspects: A theory of choice. , 1972 .

[42]  Lucy A. Suchman,et al.  Plans and Situated Actions: The Problem of Human-Machine Communication (Learning in Doing: Social, , 1987 .

[43]  Philip J. Smith,et al.  Knowledge-Based Search Tactics , 1993, Inf. Process. Manag..

[44]  Donald A. Norman,et al.  Some observations on mental models , 1987 .

[45]  J. Hoc,et al.  Cognitive psychology of planning , 1988 .