A Comparison of Machine Learning with Human Judgment

This paper compares human judgment with machine learning in a check processing context. An experiment was conducted comparing teams of subjects with three commercially available machine learning algorithms: recursive partitioning, ID3, and a back-propagation neural network. Also, the statistical technique discriminant analysis was compared. The subjects were allowed to induce rules from historical data under ideal human conditions, such as adequate time and opportunity to sort the data as desired. The results on multiple holdout samples indicate that human judgment, recursive partitioning, and the ID3 algorithm were equally accurate and more accurate than a back-propagation neural network. Subjects who chose mixed strategies of judgment were more accurate than those using noncompensatory strategies, while no subjects chose compensatory strategies. Large decision trees were not more accurate than smaller ones. There appeared to be a time threshold for humans to form accurate decision rules. Holdout sample accuracy tended to increase with primary sample accuracy. ID3 built larger trees than did either humans or recursive partitioning. The conclusion of this research is that the knowledge engineer faced with available historical data concerning a classification problem should not waste his time discerning rules, since he will only take longer and be no more accurate than a good learning tool. Knowledge of these tools will be the requisite skill for the knowledge engineer of the 1990s. Implications for IS design and further research are discussed.

[1]  Note on subjects' hypotheses in multiple-cue probability learning , 1987 .

[2]  J. A. Sniezek,et al.  Learning of Negative Cue Validities as a Function of Statistical Sophistication and Instruction , 1978 .

[3]  Kevin Knight,et al.  Connectionist ideas and algorithms , 1990, CACM.

[4]  H. J. Einhorn,et al.  Accepting error to make less error. , 1986, Journal of personality assessment.

[5]  D. Bunn,et al.  Interaction of judgemental and statistical forecasting methods: issues & , 1991 .

[6]  James V. Hansen,et al.  Inducing rules for expert system development: an example using default and bankruptcy data , 1988 .

[7]  Håkan Alm.  Effects of pretraining on the construction of complex rules in probabilistic inference tasks , 1982 .

[8]  Nigel Ford.  From information- to knowledge-management: the role of rule induction and neural net machine learning techniques in knowledge generation , 1989, J. Inf. Sci..

[9]  Berndt Brehmer,et al.  Does having to justify one's judgments change the nature of the judgment process? , 1983 .

[10]  James V. Hansen,et al.  Artificial Intelligence and Generalized Qualitative‐Response Models: An Empirical Test on Two Audit Decision‐Making Domains , 1992 .

[11]  John S. Chandler,et al.  PREDICTING STOCK MARKET BEHAVIOR THROUGH RULE INDUCTION: AN APPLICATION OF THE LEARNING‐FROM‐EXAMPLE APPROACH* , 1987 .

[12]  J. Ross Quinlan,et al.  Learning Efficient Classification Procedures and Their Application to Chess End Games , 1983 .

[13]  Cliff T. Ragsdale,et al.  Introducing Discriminant Analysis to the Business Statistics Curriculum , 1992 .

[14]  Laurence R. Paquette,et al.  The effect of decision strategy and task complexity on decision performance in an accounting context , 1988 .

[15]  Donald H. Deane,et al.  Acquisition and application of knowledge in complex inference tasks. , 1972 .

[16]  Lewis A. Taylor,et al.  Decision quality and commitment within a probabilistic environment , 1987 .

[17]  Sirkka L. Jarvenpaa,et al.  The effect of task demands and graphical format on information processing strategies , 1989 .

[18]  J. Klayman.  Cue discovery in probabilistic environments: Uncertainty and experimentation. , 1988 .

[19]  Ting-Peng Liang,et al.  A composite approach to inducing knowledge for expert systems design , 1992 .

[20]  Sholom M. Weiss,et al.  An Empirical Comparison of Pattern Recognition, Neural Nets, and Machine Learning Classification Methods , 1989, IJCAI.

[21]  Hal R. Arkes,et al.  Estimates of contingency between two dichotomous variables. , 1983 .

[22]  B. Dosher,et al.  Strategies for multiattribute binary choice. , 1983, Journal of experimental psychology. Learning, memory, and cognition.

[23]  Michelene T. H. Chi,et al.  Expertise in Problem Solving. , 1981 .

[24]  Janice D. Callahan,et al.  Rule Induction for Group Decisions with Statistical Data — An Example , 1991 .

[25]  Hugh J. Watson,et al.  A Generalized Model for Automating Judgmental Decisions , 1976 .

[26]  Ken T. Trotman,et al.  Member Variation, Recognition of Expertise, and Group Performance , 1987 .

[27]  P. Muchinsky,et al.  Human inference behavior in abstract and meaningful environments , 1975 .

[28]  Ignizio.  Introduction to expert systems , 1985 .

[29]  Fred Collopy,et al.  Management science: D. Bunn and G. Wright, “Interaction of Judgmental and Statistical Forecasting Methods: Issues and Analysis”, 37 (1991) 501–518 , 1992 .

[30]  John W. Payne,et al.  Task complexity and contingent processing in decision making: An information search and protocol analysis☆ , 1976 .

[31]  Berndt Brehmer,et al.  Content and consistency in probabilistic inference tasks , 1980 .

[32]  A. Abdel-khalik,et al.  Information Choice and Utilization in an Experiment on Default Prediction , 1980 .

[33]  Douglas H. Fisher,et al.  An Empirical Comparison of ID3 and Back-propagation , 1989, IJCAI.

[34]  Halbert White,et al.  Neural-network learning and statistics , 1989 .

[35]  T. Wallsten,et al.  Processing probabilistic multidimensional information for decisions. , 1982 .

[36]  Colin F. Camerer,et al.  General conditions for the success of bootstrapping models , 1981 .

[37]  Richard A. Cosier,et al.  Effects of cognitive styles and decision setting on performance , 1990 .

[38]  J. A. Sniezek.  The role of variable labels in cue probability learning tasks , 1986 .

[39]  T. R. Stewart.  Chapter 2 Judgment Analysis: Procedures , 1988 .

[40]  Peter Wright.  The harassed decision maker: Time pressures, distractions, and the use of evidence. , 1974 .

[41]  B. Brehmer.  Effect of practice on utilization of nonlinear rules in inference tasks , 1979 .

[42]  Effects of information‐presentation mode and task complexity on the learning of probabilistic inference tasks , 1980 .

[43]  Gabriel J. Biehal,et al.  Information-Presentation Format and Learning Goals as Determinants of Consumers' Memory Retrieval and Choice Processes , 1982 .

[44]  Daniel G. Bobrow,et al.  Expert systems: perils and promise , 1986, CACM.

[45]  Richard W. Olshavsky,et al.  Task complexity and contingent processing in decision making: A replication and extension , 1979 .

[46]  B. Brehmer,et al.  Note on the policies acquired in interpersonal learning , 1979 .

[47]  H. Rothstein.  The effects of time pressure on judgment in multiple cue probability learning , 1986 .