User Perceptions of Decision Support Effectiveness: Two Production Planning Experiments *

Decision support systems continue to be very popular in business, despite mixed research evidence as to their effectiveness. We hypothesize that what-if analysis, a prominent feature of most decision support systems, creates an “illusion of control” causing users to overestimate its effectiveness. Two experiments involving a production planning task are reported which examine decision makers' perceptions of the effectiveness of what-if analysis relative to the alternatives of unaided decision making, and quantitative decision rules. Experiment 1 found that almost all subjects believed what-if analysis was superior to unaided decision making, although using what-if analysis had no significant effect on performance. Experiment 2 found that decision makers were indifferent between what-if analysis and a quantitative decision rule which, if used, would have led to significant cost savings. Thus, what-if analysis did create an illusion of control: decision makers perceived performance differences where none existed, and did not detect large differences when they were present. In both experiments, decision makers exhibited difficulty realizing that their positive beliefs about what-if analysis were exaggerated. Such misjudgments could lead people to continue using what-if analysis even when it is not beneficial and to avoid potentially superior decision support technologies.

[1]  Paul E. Meehl,et al.  Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence , 1996 .

[2]  C. Holt,et al.  Derivation of a Linear Decision Rule for Production and Employment , 1956 .

[3]  C. C. Holt,et al.  Planning Production, Inventories, and Work Force. , 1962 .

[4]  E. H. Bowman.  Consistency and Optimality in Managerial Decision Making , 1963 .

[5]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[6]  E. Langer,et al.  Heads I win, tails it's chance: The illusion of control as a function of the sequence of outcomes in a purely chance task. , 1975 .

[7]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[8]  J. Frank Yates,et al.  Effort control and judgments , 1977 .

[9]  James A. Senn,et al.  Research in Management Information Systems: The Minnesota Experiments , 1977 .

[10]  Leonard M. Lodish,et al.  Evaluation of the Effectiveness of a Model Based Salesman's Planning System by Field Experimentation , 1977 .

[11]  Izak Benbasat,et al.  An Experimental Investigation of Some MIS Design Variables , 1977, MIS Q..

[12]  Peter G. W. Keen,et al.  Decision support systems : an organizational perspective , 1978 .

[13]  R. Hogarth,et al.  Confidence in judgment: Persistence of the illusion of validity. , 1978 .

[14]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[15]  Richard Staelin,et al.  Judgment Based Marketing Decision Models: An Experimental Investigation of the Decision Calculus Approach , 1979 .

[16]  Steven L. Alter,et al.  Decision support systems : current practice and continuing challenges , 1980 .

[17]  R. Hogarth,et al.  BEHAVIORAL DECISION THEORY: PROCESSES OF JUDGMENT AND CHOICE , 1981 .

[18]  Spyros Makridakis,et al.  The Value of Decision Making in a Complex Environment: An Experimental Approach , 1981 .

[19]  Spyros Makridakis,et al.  Forecasting and Planning: An Evaluation , 1981 .

[20]  R. Hogarth.  Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics. , 1981 .

[21]  John C. Henderson,et al.  An Evolutionary Strategy for Implementing a Decision Support System , 1981 .

[22]  Shelby H. McIntyre,et al.  An Experimental Study of the Impact of Judgment-Based Marketing Models , 1982 .

[23]  Hillel J. Einhorn,et al.  Judgment under uncertainty: Learning from experience and suboptimal rules in decision making , 1982 .

[24]  I. Benbasat,et al.  Individual-Differences In The Use Of Decision Support Aids , 1982 .

[25]  E. Langer.  The Psychology Of Control , 1983 .

[26]  John W. Payne,et al.  Effort and Accuracy in Choice , 1985 .

[27]  Don N. Kleinmuntz,et al.  Cognitive Heuristics and Feedback in a Dynamic Decision Environment , 1985 .

[28]  John Fripp.  How effective are models , 1985 .

[29]  Martin D. Goslar,et al.  Decision support systems: an empirical assessment for decision making , 1986 .

[30]  Daniel J. Power,et al.  AN EMPIRICAL ASSESSMENT OF COMPUTER‐ASSISTED DECISION ANALYSIS* , 1986 .

[31]  William Remus,et al.  Graduate students as surrogates for managers in experiments on business decision making , 1986 .

[32]  H. Arkes,et al.  Factors influencing the use of a decision rule in a probabilistic task , 1986 .

[33]  T. Pearson,et al.  Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[34]  James B. Thomas,et al.  The value of action and inference in dynamic decision making , 1987 .

[35]  W. Remus,et al.  Evidence and principles of functional and dysfunctional DSS , 1987 .

[36]  Brian L. Dos Santos,et al.  A Study of User Interface Aids for Model-Oriented Decision Support Systems , 1988 .

[37]  Ramesh Sharda,et al.  Decision support system effectiveness: a review and an empirical test , 1988 .

[38]  Brian T. Pentland,et al.  Use and productivity in personal computing , 1989, ICIS '89.

[39]  Niv Ahituv,et al.  Assessing the value of information , 1989, ICIS '89.

[40]  Niv Ahituv,et al.  Assessing the value of information , 1989, ICIS '89.

[41]  Brian T. Pentland,et al.  USE AND PRODUCTIVITY IN PERSONAL COMPUTING: AN EMPIRICAL TEST , 1989 .

[42]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[43]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[44]  Izak Benbasat,et al.  An evaluation of empirical research in managerial support systems , 1990, Decis. Support Syst..

[45]  B Kleinmuntz,et al.  Why we still use our heads instead of formulas: toward an integrative approach. , 1990, Psychological bulletin.

[46]  Janet A. Sniezek,et al.  Influences on the appropriateness of confidence in judgment: Practice, effort, information, and decision-making , 1991 .

[47]  Timothy D. Wilson,et al.  Thinking too much: introspection can reduce the quality of preferences and decisions. , 1991, Journal of personality and social psychology.

[48]  J. L. Powell,et al.  An attempt at increasing decision rule use in a judgment task , 1991 .