A Theory of Decision Support System Design for User Calibration

A theory is proposed for designing decision support systems (DSS) so that the confidence a decision maker has in a decision made using the aid equals the quality of that decision. The DSS design theory for user calibration prescribes properties of a DSS needed for users to achieve perfect calibration. Relevant calibration, decision making, and DSS literatures are synthesized; and related behavioral theories are borrowed to identify the properties of expressiveness, visibility, and inquirability as requisite components of the DSS design theory for user calibration.

[1]  P. Juslin.  The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items , 1994 .

[2]  Peter R. Keller,et al.  Visual cues - practical data visualization , 1993 .

[3]  Ralph H. Sprague,et al.  Building Effective Decision Support Systems , 1982 .

[4]  E. Shortliffe,et al.  An analysis of physician attitudes regarding computer-based clinical consultation systems. , 1981, Computers and biomedical research, an international journal.

[5]  Norman L. Chervany,et al.  An Experimental Evaluation of Information Overload in a Production Environment , 1974 .

[6]  Izak Benbasat,et al.  An evaluation of empirical research in managerial support systems , 1990, Decis. Support Syst..

[7]  G. Gigerenzer.  Why the distinction between single-event probabilities and frequencies is important for psychology (and vice versa). , 1994 .

[8]  Shelby H. McIntyre,et al.  An Experimental Study of the Impact of Judgment-Based Marketing Models , 1982 .

[9]  Andrew B. Whinston,et al.  Foundations of Decision Support Systems , 1981 .

[10]  Ben Shneiderman,et al.  Designing the user interface (videotape) , 1987 .

[11]  James A. Mason,et al.  Evaluating the Intelligence in Dialogue Systems , 1988, Int. J. Man Mach. Stud..

[12]  Erik Hollnagel,et al.  Cognitive Systems Engineering: New Wine in New Bottles , 1983, Int. J. Man Mach. Stud..

[13]  A. H. Murphy,et al.  The expected value of frequency calibration , 1990 .

[14]  B. H. McCormick,et al.  Visualization in scientific computing , 1995 .

[15]  J. Frank Yates,et al.  Subjective probability accuracy analysis. , 1994 .

[16]  Elizabeth C. Hirschman,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[17]  P. Booth.  Errors and theory in human-computer interaction , 1991 .

[18]  Rebecca A. Henry,et al.  Accuracy and confidence in group judgment , 1989 .

[19]  Rami Zwick,et al.  Comparing the calibration and coherence of numerical and verbal probability judgments , 1993 .

[20]  U. Neisser.  John Dean's memory: A case study , 1981, Cognition.

[21]  Janet A. Sniezek,et al.  Influences on the appropriateness of confidence in judgment: Practice, effort, information, and decision-making , 1991 .

[22]  Jeffrey E. Kottemann,et al.  What-if analysis and the illusion of control , 1991, Proceedings of the Twenty-Fourth Annual Hawaii International Conference on System Sciences.

[23]  B. M. Staw.  The Escalation of Commitment To a Course of Action , 1981 .

[24]  Joseph Weizenbaum,et al.  and Machine , 1977 .

[25]  Ab McClelland,et al.  The calibration of subjective probabilities: Theories and models 1980-1993 , 1994 .

[26]  H. Arkes,et al.  Two methods of reducing overconfidence , 1987 .

[27]  Barry M. Staw,et al.  Knee-deep in the Big Muddy: A study of escalating commitment to a chosen course of action. , 1976 .

[28]  John M. Carroll,et al.  Interface design issues for advice-giving expert systems , 1987, CACM.

[29]  Charles R. Schwenk,et al.  Agreement and thinking alike: ingredients for poor decisions , 1990 .

[30]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[31]  J. Keith Murnighan,et al.  Group Confidence Pressures in Iterative Decisions , 1982 .

[32]  S. Oskamp.  OVERCONFIDENCE IN CASE-STUDY JUDGMENTS. , 1965, Journal of consulting psychology.

[33]  P. Christopher Earley,et al.  Technology, credibility, and feedback use , 1989 .

[34]  Daniel J. Power,et al.  AN EMPIRICAL ASSESSMENT OF COMPUTER‐ASSISTED DECISION ANALYSIS* , 1986 .

[35]  Randall Steeb,et al.  A Computer-Based Interactive System for Group Decisionmaking , 1981, IEEE Transactions on Systems, Man, and Cybernetics.

[36]  H. J. Einhorn,et al.  Accepting error to make less error. , 1986, Journal of personality assessment.

[37]  G. Keren.  Calibration and probability judgements: Conceptual and methodological issues , 1991 .

[38]  Donald A. Norman,et al.  Approaches to the Study of Intelligence , 1991, Artif. Intell..

[39]  Thomas W. Malone,et al.  Expert systems: the next challenge for managers , 1986 .

[40]  R. Hogarth,et al.  Confidence in judgment: Persistence of the illusion of validity. , 1978 .

[41]  Neff Walker,et al.  A classification of visual representations , 1994, CACM.

[42]  G. Gigerenzer,et al.  Probabilistic mental models: a Brunswikian theory of confidence. , 1991, Psychological review.

[43]  B. Kemal Büyükkurt,et al.  An Experimental Study of the Effectiveness of Three Debiasing Techniques , 1991 .

[44]  Samuel G. Trull.  Some Factors Involved in Determining Total Decision Success , 1966 .

[45]  R. Hogarth.  Judgement and choice: The psychology of decision , 1982 .

[46]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[47]  Sid L. Huff.  DSS Development: Promise and Practice , 1986 .

[48]  George M. Kasper,et al.  The design of joint cognitive systems: the effect of cognitive coupling on performance , 1994, Int. J. Hum. Comput. Stud..

[49]  Richard P. Will,et al.  Individual Differences in the Performance and Use of an Expert System , 1992, Int. J. Man Mach. Stud..

[50]  E. Jantsch.  The design of inquiring systems,: by C. West Churchman New York, Basic Books, 1972. , 1972 .

[51]  Gerardine DeSanctis,et al.  A foundation for the study of group decision support systems , 1987 .

[52]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[53]  N. M. Morris,et al.  On Looking into the Black Box: Prospects and Limits in the Search for Mental Models , 1986 .

[54]  Richard O. Mason,et al.  A Dialectical Approach to Strategic Planning , 1969 .

[55]  Ben Shneiderman,et al.  Designing the user interface (2nd ed.): strategies for effective human-computer interaction , 1992 .

[56]  Richard A. Stevick,et al.  Importance of Decision and Postdecision Dissonance: A Return to the Racetrack , 1991 .

[57]  Charles R. Crowell,et al.  Human-computer interaction: A preliminary social psychological analysis , 1982 .

[58]  Jayashree Mahajan.  The Overconfidence Effect in Marketing Management Predictions , 1992 .

[59]  Gerardine DeSanctis,et al.  Understanding the effectiveness of computer graphics for decision support: a cumulative experimental approach , 1986, CACM.

[60]  John D. Gould,et al.  An experimental study of people creating spreadsheets , 1987, TOIS.

[61]  William Remus,et al.  Future Directions: Toward Intelligent Decision Support Systems: An Artificially Intelligent Statistician , 1986, MIS Q..

[62]  Michael J. Ginzberg,et al.  Decision Support Systems: Issues and Perspectives , 1982 .

[63]  Ramesh Sharda,et al.  Decision support system effectiveness: a review and an empirical test , 1988 .

[64]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[65]  Jan Bell,et al.  THE EFFECT OF PRESENTATION FORM ON JUDGMENT CONFIDENCE IN PERFORMANCE EVALUATION , 1984 .

[66]  Fred D. Davis,et al.  User Perceptions of Decision Support Effectiveness: Two Production Planning Experiments * , 1994 .

[67]  Philip N. Johnson-Laird,et al.  Mental Models in Cognitive Science , 1980, Cogn. Sci..

[68]  Steven D. Penrod,et al.  Performance feedback improves the resolution of confidence judgments , 1988 .

[69]  John C. Henderson.  FINDING SYNERGY BETWEEN DECISION SUPPORT SYSTEMS AND EXPERT SYSTEMS RESEARCH , 1987 .

[70]  M. Björkman.  Internal Cue Theory: Calibration and Resolution of Confidence in General Knowledge , 1994 .

[71]  Willem A. Wagenaar,et al.  Calibration and the effects of knowledge and reconstruction in retrieval from memory , 1988, Cognition.

[72]  Janet A. Sniezek,et al.  The effect of choosing on confidence in choice , 1990 .

[73]  A. Bandura.  Self-efficacy: toward a unifying theory of behavioral change. , 1977, Psychological review.

[74]  Omar El Sawy,et al.  Building an Information System Design Theory for Vigilant EIS , 1992, Inf. Syst. Res..

[75]  Dave Mandelkern,et al.  GUIs: the next generation , 1993, CACM.

[76]  R. Hogarth,et al.  BEHAVIORAL DECISION THEORY: PROCESSES OF JUDGMENT AND CHOICE , 1981 .

[77]  Steven L. Alter,et al.  Why is Man-Computer Interaction Important for Decision Support Systems? , 1977 .

[78]  Gerardine DeSanctis,et al.  Computer-Based Support for Group Problem-Finding: An Experimental Investigation , 1988, MIS Q..

[79]  Peter G. W. Keen,et al.  Decision support systems: The next decade , 1987, Decis. Support Syst..

[80]  Richard A. Block,et al.  Overconfidence in estimation: Testing the anchoring-and-adjustment hypothesis , 1991 .

[81]  John Anthony Nelson.  Dialectic information systems: a methodology for planning and decision-making. , 1973 .

[82]  Ian I. Mitroff,et al.  A Program for Research on Management Information Systems , 1973 .

[83]  P. Juslin.  An explanation of the hard-easy effect in studies of realism of confidence in one's general knowledge , 1993 .

[84]  Deborah Hix,et al.  Human-computer interface development: concepts and systems for its management , 1989, CSUR.

[85]  M. Björkman,et al.  Knowledge, calibration, and resolution: A linear model , 1992 .

[86]  Bonnie M. Muir,et al.  Trust Between Humans and Machines, and the Design of Decision Aids , 1987, Int. J. Man Mach. Stud..

[87]  Richard Veryard,et al.  The role of visibility in systems , 1986 .

[88]  Thomas G. Moher,et al.  Response Assertiveness in Human-Computer Dialogues , 1990, Int. J. Man Mach. Stud..

[89]  Herbert A. Simon,et al.  The new science of management decision , 1960 .

[90]  T. Helstrup,et al.  One, two, or three memories? A problem-solving approach to memory for performed acts ☆ , 1987 .

[91]  A. Tversky,et al.  Prospect Theory : An Analysis of Decision under Risk Author ( s ) : , 2007 .

[92]  Dan Zakay,et al.  The influence of computerized feedback on overconfidence in knowledge , 1992 .