An integrative evaluation framework for intelligent decision support systems

Evaluation of the overall effectiveness of decision support systems (DSS) has been a research topic since the early 1980s. As artificial intelligence methods have been incorporated into systems to create intelligent decision support systems (IDSS), researchers have attempted to quantify the value of the additional capabilities. Despite the useful and relevant insights generated by previous research, existing evaluation methodologies offer only a fragmented and incomplete view of IDSS value and the contribution of its technical infrastructure. This paper proposes an integrative, multiple criteria IDSS evaluation framework through a model that links the decision value of an IDSS to both the outcome from, and process of, decision making and down to specific components of the IDSS. The proposed methodology provides the designer and developer specific guidance on the intelligent tools most useful for a specific user with a particular decision problem. The proposed framework is illustrated by evaluating an actual IDSS that coordinates management of urban infrastructures.

[1]  Herbert A. Simon.  Two Heads Are Better than One: The Collaboration between AI and OR , 1987 .

[2]  T. L. Saaty.  A Scaling Method for Priorities in Hierarchical Structures , 1977 .

[3]  Guisseppi A. Forgionne,et al.  A multiple-criteria framework for evaluation of decision support systems , 2004 .

[4]  W. Hacker,et al.  Design Problem Solving , 2006 .

[5]  Alex Meystel.  Measuring Performance and Intelligence of Intelligent Systems , 2000 .

[6]  Patrick T. Harker,et al.  The Art and Science of Decision Making: The Analytic Hierarchy Process , 1989 .

[7]  Paul B. Kantor,et al.  Cross-Evaluation: A new model for information system evaluation , 2006, J. Assoc. Inf. Sci. Technol..

[8]  Paul R. Cohen,et al.  A Survey of the Eighth National Conference on Artificial Intelligence: Pulling Together or Pulling Apart? , 1991, AI Mag..

[9]  Tor Guimaraes,et al.  Assessing the quality of institutional DSS , 1995 .

[10]  Rajiv Kohli,et al.  Measuring Information Technology Payoff: A Meta - Analysis of Structural Variables in Firm - Level Empirical Research , 2003, Inf. Syst. Res..

[11]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[12]  R. French.  The Turing Test: the first 50 years , 2000, Trends in Cognitive Sciences.

[13]  Allen Newell,et al.  The Knowledge Level , 1989, Artif. Intell..

[14]  Thomas L. Saaty,et al.  How to Make a Decision: The Analytic Hierarchy Process , 1990 .

[15]  R. French,et al.  The Turing Test : The First Fifty Years , 2000 .

[16]  Alejandro Quintero,et al.  Prototyping an intelligent decision support system for improving urban infrastructures management , 2005, Eur. J. Oper. Res..

[17]  Jatinder N. D. Gupta,et al.  Toward a Comprehensive Framework for the Design and Evaluation of Intelligent Decision-making Support Systems (i-DMSS) , 2005, J. Decis. Syst..

[18]  Herbert A. Simon,et al.  Computer Science as Empirical Inquiry , 2011 .

[19]  B. Chandrasekaran,et al.  Generic Tasks in Knowledge-Based Reasoning: High-Level Building Blocks for Expert System Design , 1986, IEEE Expert.

[20]  Gloria Phillips-Wren,et al.  Evaluation of Decision-Making Support Systems , 2008 .

[21]  Herbert A. Simon.  The new science of management decision. , 1960 .

[22]  Frada Burstein,et al.  Intelligent decision support in the context of the modern organisation , 1997 .

[23]  Rajiv D. Banker,et al.  Value Implications of Investments in Information Technology , 2006, Manag. Sci..

[24]  Guisseppi A. Forgionne,et al.  A Multicriteria Model for the Evaluation of Intelligent Decision-making Support Systems (i-DMSS) , 2006 .

[25]  Rajiv Kohli,et al.  A multiple criteria assessment of decision technology system journal quality , 2001, Inf. Manag..

[26]  Peter Checkland,et al.  Systems Thinking, Systems Practice , 1981 .

[27]  Todd R. Johnson,et al.  Task-structure analysis for knowledge modeling , 1992, CACM.

[28]  Guisseppi A. Forgionne,et al.  Intelligent decision-making support systems : foundations, applications and challenges , 2006 .

[29]  Camille Rosenthal-Sabroux,et al.  Artificial intelligence tools for decision support systems , 1997 .

[30]  G. A. Forgionne.  An AHP model of DSS effectiveness , 1999 .

[31]  Peter G. W. Keen,et al.  Invited Article: Value Analysis: Justifying Decision Support Systems , 1981, MIS Q..

[32]  Jatinder N. D. Gupta,et al.  Decision-Making Support Systems , 2005, Encyclopedia of Information Science and Technology.

[33]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[34]  B. Chandrasekaran.  Design Problem Solving : A Task Analysis 1 , 1990 .

[35]  Guisseppi A. Forgionne,et al.  A decision-theoretic approach to the evaluation of information retrieval systems , 2006, Inf. Process. Manag..

[36]  Dennis M. Buede.  Structuring Value Attributes , 1986 .

[37]  John S. Chandler.  A Multiple Criteria Approach for Evaluating Information Systems , 1982, MIS Q..

[38]  Timothy A. Salthouse,et al.  Decision Making and Problem Solving , 1982 .

[39]  Frada Burstein,et al.  A Multi-faceted Decision Support System Evaluation Approach , 2001 .

[40]  Leonard Adelman,et al.  Evaluating decision support and expert systems , 1991 .

[41]  Pearl Pu,et al.  Performance Evaluation of Consumer Decision Support Systems , 2006, Int. J. E Bus. Res..

[42]  Ramesh Sharda,et al.  Decision support system effectiveness: a review and an empirical test , 1988 .

[43]  Jacky Akoka.  A framework for decision support systems evaluation , 1981, Inf. Manag..

[44]  Guisseppi A. Forgionne,et al.  Econometric Simulation for E-Business Strategy Evaluation , 2006, Int. J. E Bus. Res..

[45]  Peter F. Patel-Schneider,et al.  Term Subsumption Languages in Knowledge Representation , 1990, AI Mag..

[46]  Rajiv Kohli,et al.  Performance Impacts of Information Technology: Is Actual Usage the Missing Link? , 2003, Manag. Sci..

[47]  T. Saaty.  How to Make a Decision: The Analytic Hierarchy Process , 1990 .

[48]  Lorin M. Hitt,et al.  Beyond the productivity paradox , 1998, CACM.

[49]  Ralph H. Sprague,et al.  Invited Article: A Framework for the Development of Decisoin Support Systems , 1980, MIS Q..

[50]  Lorin M. Hitt,et al.  Beyond the Productivity Paradox: Computers are the Catalyst for Bigger Changes , 1998 .

[51]  P. Keen.  Value Analysis: Justifying Decision Support Systems , 1993 .

[52]  Ralph H. Sprague,et al.  A Framework for the Development of Decision Support Systems , 1993 .

[53]  Guisseppi A. Forgionne,et al.  Decision-Making Support System Effectiveness: the Process to Outcome Link , 2000, Inf. Knowl. Syst. Manag..

[54]  Lorin M. Hitt,et al.  Productivity, Business Profitability, and Consumer Surplus: Three Different Measures of Information Technology Value , 1996, MIS Q..

[55]  Robert J. Kauffman,et al.  50th Anniversary Article: The Evolution of Research on Information Systems: A Fiftieth-Year Survey of the Literature in Management Science , 2004, Manag. Sci..

[56]  N. Carr.  IT doesn't matter , 2003, IEEE Engineering Management Review.

[57]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[58]  Thomas L. Saaty,et al.  Marketing Applications of the Analytic Hierarchy Process , 1980 .

[59]  Rajiv Kohli,et al.  The IT Payoff: Measuring the Business Value of Information Technology Investments , 2002 .