Bad numbers: coping with flawed decision support

Evidence is accumulating that many spreadsheet-based decision support systems contain errors. These errors can result in bad numbers, which in turn could lead to bad decisions. We review the literature on the origins and consequences of bad (erroneous) numbers produced by models and/or decision support systems built around them. Then, we present a case study in which an experiment in visual support for a sequential decision-making task was tainted by bad numbers. Both the literature review and the experiment indicate a robust human ability to overcome flawed decision support. We conclude with questions that need to be addressed in order to better understand the capabilities of humans to deal with erroneous results from decision support systems.

[1]  Hans Vangheluwe,et al.  An observer-based approach to modeling error identification , 1998 .

[2]  Aleksandr N. Anisimov.  Do Not Make a Mistake in Choosing the Model , 1992 .

[3]  William A. Wallace,et al.  Intelligent Interface Design: An Empirical Assessment of Knowledge Presentation in Expert Systems , 1990, MIS Q..

[4]  J. Schnoor.  Environmental Modeling: Fate and Transport of Pollutants in Water, Air, and Soil , 1996 .

[5]  James S. Hodges,et al.  Six (Or So) Things You Can Do with a Bad Model , 1991, Oper. Res..

[6]  George M. Kasper,et al.  A Theory of Decision Support System Design for User Calibration , 1996, Inf. Syst. Res..

[7]  Dag Haugland,et al.  Mathematical and computer modelling reports: Models for petroleum field exploitation , 1988 .

[8]  Raymond R. Panko,et al.  Spreadsheets on trial: a survey of research on spreadsheet risks , 1996, Proceedings of HICSS-29: 29th Hawaii International Conference on System Sciences.

[9]  Yiyuan Zhao,et al.  Effects of modeling errors on trajectory predictions in air traffic control automation , 1996 .

[10]  Samuel J. Raff.  The Navy passive ranging problem , 1991, Comput. Oper. Res..

[11]  Chris Chatfield,et al.  The Analysis of Time Series: An Introduction , 1981 .

[12]  Christopher Vyn Jones,et al.  Visualization and Optimization , 1997 .

[13]  T R Willemain,et al.  Influence of model management systems on decision making: empirical evidence and implications , 2000, J. Oper. Res. Soc..

[14]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[15]  Laurie B. Waisel,et al.  The cognitive role of visualization in modeling , 1998 .

[16]  Izak Benbasat,et al.  The Use of Explanations in Knowledge-Based Systems: Cognitive Perspective and a Process-Tracing Analysis , 2000, J. Manag. Inf. Syst..

[17]  Muhittin Oral,et al.  In search of a valid view of model validation for operations research , 1993 .

[18]  James J. Jiang,et al.  The Impact of Model Performance History Information on Users' Confidence in Decision Models: An Experimental Examination , 1996 .

[19]  George M. Kasper,et al.  Animation in User Interfaces Designed for Decision Support Systems: The Effects of Image Abstraction, Transition, and Interactivity on Decision Quality* , 1997 .

[20]  Sean B. Eom,et al.  A survey of decision support system applications (1988–1994) , 1998, J. Oper. Res. Soc..

[21]  Baruch Fischhoff,et al.  Decision making in online searching , 1988, J. Am. Soc. Inf. Sci..

[22]  Laurie B. Waisel,et al.  Using Diagrammatic Representations in Mathematical Modeling: The Sketches of Expert Modelers , 1997 .

[23]  William A. Wallace,et al.  An epistemological view of decision-aid technology with emphasis on expert systems , 1989, IEEE Trans. Syst. Man Cybern..

[24]  M. Rothkopf,et al.  Misapplications Reviews: Getting the Model Right—The Case of Competitive Bidding , 1993 .

[25]  Saul I. Gass.  OR in the real world: How things go wrong , 1991, Comput. Oper. Res..

[26]  K. Sengupta,et al.  Alternative conceptions of feedback in dynamic decision environments: an experimental investigation , 1993 .

[27]  William A. Wallace,et al.  An experimental investigation of the interaction between displays and models in multiple-criteria decision support , 1989 .

[28]  James A. Chisman.  Preparing students and new practitioners for the real-world of OR/MS , 1991, Comput. Oper. Res..

[29]  Saul I. Gass,et al.  Encyclopedia of Operations Research and Management Science , 1997 .

[30]  Gene Woolsey,et al.  Being wrong with Clarke & Wright , 1989, Computers & Operations Research.

[31]  Christopher Grassotti,et al.  Distortion Representation of Forecast Errors for Model Skill Assessment and Objective Analysis. Revision 1.12 , 1996 .

[32]  Efraim Turban,et al.  Decision support systems and intelligent systems , 1997 .

[33]  R. Fildes.  Database models and managerial intuition: 50% model +50% manager : Robert C. Blattberg and Stephen J. Hoch, Management Science 36 (1990) 887-899 , 1991 .

[34]  Raymond R. Panko,et al.  What we know about spreadsheet errors , 1998 .

[35]  William A. Wallace,et al.  Ethics in Modeling , 1994 .

[36]  C. M. Khoong.  Decision support systems: an extended research agenda , 1995 .

[37]  Robert M. O'Keefe,et al.  An Experimental Investigation into the Efficacy of Visual Interactive Simulation , 1995 .