Process Tracing Methods in Decision Support Systems Research: Exploring the Black Box

An overview is provided of the appficability to DSS research of process tracing methodologies in general and verbal protocol analysis in particular. Rationale is developed for why process tracing methods are an important addition to the inventory of methodologies available to researchers, stressing the need to explore the "black box" of decision processes. A variety of process tracing methods are presented, along with their relative strengths and weaknesses. Verbal protocol analysis is discussed as a particularly promising method for use in DSS research. The nature of verbal protocols is outlined and methods for analysis are reviewed. The criticisms of this method and the controversy surrounding its use are discussed, and an assessment of the validity of these criticisms is provided. Finally, areas of DSS research in which protocols may prove valuable are identified and relevant studies are examined.

[1]  Norman L. Chervany,et al.  An Experimental Evaluation of Information Overload in a Production Environment , 1974 .

[2]  Daniel G Bobrow,et al.  On data-limited and resource-limited processes , 1975, Cognitive Psychology.

[3]  Shelby H. McIntyre,et al.  An Experimental Study of the Impact of Judgment-Based Marketing Models , 1982 .

[4]  Andrew B. Whinston,et al.  Foundations of Decision Support Systems , 1981 .

[5]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[6]  D. Hofstadter,et al.  Gödel, Escher, Bach: An Eternal Golden Braid@@@Godel, Escher, Bach: An Eternal Golden Braid , 1980 .

[7]  Izak Benbasat,et al.  An experimental program investigating color-enhanced and graphical information presentation: an integration of the findings , 1986, CACM.

[8]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[9]  Gerardine DeSanctis,et al.  Understanding the effectiveness of computer graphics for decision support: a cumulative experimental approach , 1986, CACM.

[10]  J. Payne,et al.  Exploring predecisional behavior: An alternative approach to decision research , 1978 .

[11]  Eliot R. Smith,et al.  Limits on perception of cognitive processes: A reply to Nisbett and Wilson. , 1978 .

[12]  Don N. Kleinmuntz,et al.  Cognitive Heuristics and Feedback in a Dynamic Decision Environment , 1985 .

[13]  H. Simon,et al.  Information Processing Models of Cognition , 1979 .

[14]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[15]  James L. McKenney,et al.  Management decision systems : computer-based support for decision making , 1971 .

[16]  John R. Hayes,et al.  The Complete Problem Solver , 1981 .

[17]  Herbert A. Simon,et al.  The Structure of Ill Structured Problems , 1973, Artif. Intell..

[18]  Karl E. Weick,et al.  An Assessment Of Laboratory Experiments In Accounting , 1982 .

[19]  T. M. Amabile.  The social psychology of creativity , 1984 .

[20]  H. J. Einhorn,et al.  Linear regression and process-tracing models of judgment. , 1979 .

[21]  Gerardine DeSanctis,et al.  ICIS Paper: Methodological Issues in Experimental IS Research: Experiences and Recommendations , 1985, MIS Q..

[22]  Hillel J. Einhorn,et al.  A Synthesis: Accounting and Behavioral Science , 1976 .

[23]  Hugh J. Watson,et al.  Management's Role in the Approval and Administration of Decision Support Systems , 1983, MIS Q..

[24]  Izak Benbasat,et al.  An Experimental Evaluation of Graphical and Color-Enhanced Information Presentation , 1985 .

[25]  N. Pennington,et al.  Inside the Jury. , 1985 .

[26]  Ralph H. Sprague,et al.  Decision support systems: Putting theory into practice , 1986 .

[27]  K. Duncker,et al.  On problem-solving , 1945 .

[28]  I. Benbasat,et al.  Individual-Differences In The Use Of Decision Support Aids , 1982 .

[29]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[30]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[31]  Ralph H. Sprague,et al.  Building Effective Decision Support Systems , 1982 .

[32]  Jerrold H. May,et al.  An Intelligent Workstation for Electrocenter Design , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[33]  Izak Benbasat,et al.  An Investigation of the Effectiveness of Color and Graphical Information Presentation Under Varying Time Constraints , 1986, MIS Q..

[34]  J. Bettman.  An information processing theory of consumer choice , 1979 .

[35]  Izak Benbasat,et al.  Behavioral Aspects of Information Processing for the Design of Management Information Systems , 1982, IEEE Transactions on Systems, Man, and Cybernetics.

[36]  Henry C. Lucas,et al.  An Experimental Investigation of the Use of Computer-Based Graphics in Decision Making , 1981 .

[37]  Robert Libby,et al.  Accounting and human information processing : theory and applications , 1981 .

[38]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[39]  David M. Schweiger,et al.  Is the simultaneous verbal protocol a viable method for studying managerial problem solving and decision making , 1983 .

[40]  Larry D. Rosen,et al.  An eye fixation analysis of multialternative choice , 1975, Memory & cognition.

[41]  James Waldron Botkin,et al.  An intuitive computer system: a cognitive approach to the management learning process. , 1974 .

[42]  P HuberGeorge.  Cognitive Style as a Basis for MIS and DSS Designs , 1983 .

[43]  Dorothea P. Simon,et al.  Expert and Novice Performance in Solving Physics Problems , 1980, Science.

[44]  W ZmudRobert.  Individual Differences and MIS Success , 1979 .

[45]  John R. Anderson.  Cognitive Psychology and Its Implications , 1980 .

[46]  Peter A. White,et al.  Limitations on verbal reports of internal events: A refutation of Nisbett and Wilson and of Bem. , 1980 .

[47]  John W. Payne,et al.  Task complexity and contingent processing in decision making: An information search and protocol analysis☆ , 1976 .

[48]  Eric D. Carlson,et al.  An approach for designing decision support systems , 1978, DATB.

[49]  Paul Slovic,et al.  Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgment. , 1971 .

[50]  Nicholas P. Vitalari,et al.  Knowledge as a Basis for Expertise in Systems Analysis: An Empirical Study , 1985, MIS Q..

[51]  Iris Vessey,et al.  An investigation of the psychological processes underlying the debugging of computer programs , 1984 .

[52]  A. Newell.  ON THE ANALYSIS OF HUMAN PROBLEM SOLVING PROTOCOLS , 1966 .

[53]  Marinus J. Bouwman.  Human Diagnostic Reasoning by Computer: An Illustration from Financial Analysis , 1983 .

[54]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[55]  B. Dosher,et al.  Strategies for multiattribute binary choice. , 1983, Journal of experimental psychology. Learning, memory, and cognition.