An Empirical Evaluation

Both optimization techniques and expert systems technologies are popular approaches for developing tools to assist in complex problem-solving tasks. Because of the underlying complexity of many such tasks, however, the models of the world implicitly or explicitly embedded in such tools are often incomplete and the problem-solving methods fallible. The result can be "brittleness" in situations that were not anticipated by the system designers. To deal with this weakness, it has been suggested that "cooperative" rather than "automated" problem-solving systems be designed. Such cooperative systems are proposed to explicitly enhance the collaboration of the person (or a group of people) and the computer system. This study evaluates the impact of alternative design concepts on the performance of 30 airline pilots interacting with such a cooperative system designed to support enroute flight planning. The results clearly demonstrate that different system design concepts can strongly influence the cognitive processes and resultant performances of users. Based on think-aloud protocols, cognitive models are proposed to account for how features of the Computer system interacted with specific types of scenarios to influence exploration and decision making by the pilots. The results are then used to develop recommendations for guiding the design of cooperative systems.

[1]  D. Dorfman,et al.  Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data , 1969 .

[2]  W. J. Conover,et al.  Practical Nonparametric Statistics , 1972 .

[3]  A. Tversky.  Elimination by aspects: A theory of choice. , 1972 .

[4]  Earl D. Sacerdoti,et al.  Planning in a Hierarchy of Abstraction Spaces , 1974, IJCAI.

[5]  D. Middleton,et al.  An Experimental Evaluation of Four Face-to-Face Teaching Strategies , 1978 .

[6]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[7]  Barbara Hayes-Roth,et al.  A Cognitive Model of Planning , 1979, Cogn. Sci..

[8]  J. A. Acree.  On mutation , 1980 .

[9]  S. Harter.  A new self-report scale of intrinsic versus extrinsic orientation in the classroom: Motivational and informational components. , 1981 .

[10]  Mark Stefik,et al.  Planning with Constraints (MOLGEN: Part 1) , 1981, Artif. Intell..

[11]  T. Amemiya.  QUALITATIVE RESPONSE MODELS: A SURVEY , 1981 .

[12]  William B. Rouse,et al.  Studies of planning behavior of aircraft pilots in normal, abnormal, and emergency situations , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[13]  Larry Joe Morell.  A theory of error-based testing , 1983 .

[14]  Charles M. Reigeluth,et al.  Instructional Design Theories and Models : An Overview of Their Current Status , 1983 .

[15]  Martin R. Woodward,et al.  An integrated system for program testing using weak mutation and data flow analysis , 1985, ICSE '85.

[16]  Cheng Hsiao,et al.  Analysis of Panel Data , 1987 .

[17]  A. Cameron,et al.  Econometric models based on count data. Comparisons and applications of some estimators and tests , 1986 .

[18]  Paul E. Lehner,et al.  Cognitive Factors in User/Expert-System Interaction , 1987 .

[19]  G. Dionne,et al.  Automobile Insurance Ratemaking In The Presence Of Asymmetric Information , 1992 .

[20]  Vi Andrew Jefferson Offutt,et al.  Automatic test data generation , 1988 .

[21]  Aditya P. Mathur,et al.  High performance testing on SIMD machines , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[22]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[23]  Aditya P. Mathur,et al.  Modeling mutation on a vector processor , 1988, ICSE '88.

[24]  M. R. Woodward,et al.  From weak to strong, dead or alive? an analysis of some mutation testing issues , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[25]  G. Dionne,et al.  A Generalization of Automobile Insurance Rating Models: The Negative Binomial Distribution with a Regression Component , 1988, ASTIN Bulletin.

[26]  B. Weiner.  History of motivational research in education , 1990 .

[27]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[28]  A. Graesser,et al.  Question answering in the context of stories , 1991 .

[29]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[30]  R. Hilgers.  Distribution-Free Confidence Bounds for ROC Curves , 1991, Methods of Information in Medicine.

[31]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[32]  Brian Marick,et al.  The weak mutation hypothesis , 1991, TAV4.

[33]  R. Arnott.  Moral Hazard and Competitive Insurance Markets , 1992 .

[34]  Roy P. Pargas,et al.  Mutation Testing of Software Using MIMD Computer , 1992, ICPP.

[35]  Byoungju Choi,et al.  High-performance mutation testing , 1993, J. Syst. Softw..

[36]  D. Guilkey,et al.  Estimation and testing in the random effects probit model , 1993 .

[37]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[38]  W. Hall,et al.  Confidence Bands for Receiver Operating Characteristic Curves , 1993, Medical decision making : an international journal of the Society for Medical Decision Making.

[39]  Philip J. Smith,et al.  Knowledge-Based Search Tactics , 1993, Inf. Process. Manag..

[40]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[41]  G. Campbell,et al.  Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. , 1994, Statistics in medicine.

[42]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[43]  G. Dionne,et al.  Inferring technological parameters from incomplete panel data , 1998 .

[44]  Benedict du Boulay,et al.  Implementation of motivational tactics in tutoring systems , 1995 .

[45]  M. Lechner.  Some Specification Tests for Probit Models Estimated on Panel Data , 1995 .

[46]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.