A Methodology for simplification and interpretation of backpropagation-based neural network models

Abstract A new methodology for building inductive expert systems known as neural networks has emerged as one of the most promising applications of artificial intelligence in the 1990s. The primary advantages of a neural network approach for modeling expert decision processes are: (1) the ability of the network to learn from examples of experts' decisions that avoids the costly, time consuming, and error prone task of trying to directly extract knowledge of a problem domain from an expert and (2) the ability of the network to handle noisy, incomplete, and distorted data that are typically found in decision making under conditions of uncertainty. Unfortunately, a major limitation of neural network-based models has been the opacity of the inference process. Unlike conventional expert system decision support tools, decision makers are generally unable to understand the basis of neural network decisions. This problem often makes such systems undesirable for decision support applications. A new methodology is presented that allows the development of highly simplified backpropagation neural network models. This methodology simplifies netw variables that are not contributing to the networks ability to produce accurate predictions. Elimination of unnecessary input variables directly reduces the number of network parameters that must be estimated and consequently the complexity of the network structure. A primary benefit of this development methodology is that it is based on a variable importance measure that addresses the problem of producing an interpretation of a neural network's functioning. Decision makers may easily understand the resulting networks in terms of the proportional contribution each input variable is making in the production of accurate predictions. Furthermore, in actual application the accuracy of these simplified models should be comparable to or better than the more complex models developed with the standard approach. This new methodology is demonstrated by two classification problems based on sets of actual data.

[1]  Ming S. Hung,et al.  A neural network approach to the classification problem , 1990 .

[2]  G. David Garson,et al.  Interpreting neural-network connection weights , 1991 .

[3]  Maureen Caudill,et al.  Neural networks primer, part III , 1988 .

[4]  Lashon B. Booker,et al.  Proceedings of the fourth international conference on Genetic algorithms , 1991 .

[5]  Philip D. Wasserman,et al.  Neural computing - theory and practice , 1989 .

[6]  S. T. Buckland,et al.  Computer Intensive Statistical Methods: Validation, Model Selection, and Bootstrap , 1993 .

[7]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[8]  Ivan Bratko,et al.  Some comments on rule induction , 1987, The Knowledge Engineering Review.

[9]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[10]  S. Chatterjee,et al.  Regression Analysis by Example , 1979 .

[11]  Pi-Sheng Deng,et al.  Automating Knowledge Acquisition and Refinement for Decision Support: A Connectionist Inductive Inference Model* , 1993 .

[12]  Padhraic Smyth,et al.  Rule-Based Neural Networks for Classification and Probability Estimation , 1992, Neural Computation.

[13]  James V. Hansen,et al.  Artificial Intelligence and Generalized Qualitative‐Response Models: An Empirical Test on Two Audit Decision‐Making Domains , 1992 .

[14]  H. White.  Some Asymptotic Results for Learning in Single Hidden-Layer Feedforward Network Models , 1989 .

[15]  B. Efron,et al.  The Jackknife: The Bootstrap and Other Resampling Plans. , 1983 .

[16]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[17]  Melody Y. Kiang,et al.  Managerial Applications of Neural Networks: The Case of Bank Failure Predictions , 1992 .

[18]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[19]  Paul E. Meehl,et al.  Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence , 1996 .

[20]  Kevin Knight,et al.  Connectionist ideas and algorithms , 1990, CACM.

[21]  Mike James,et al.  Classification Algorithms , 1986, Encyclopedia of Machine Learning and Data Mining.

[22]  Avron Barr,et al.  The Handbook of Artificial Intelligence , 1982 .

[23]  E. Mine Cinar,et al.  Neural Networks: A New Tool for Predicting Thrift Failures , 1992 .

[24]  L. Darrell Whitley,et al.  Genetic Reinforcement Learning with Multilayer Neural Networks , 1991, ICGA.

[25]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[26]  Maureen Caudill,et al.  Neural network training tips and techniques , 1991 .

[27]  Robert J. Marks,et al.  Performance Comparisons Between Backpropagation Networks and Classification Trees on Three Real-World Applications , 1989, NIPS.

[28]  Stephen I. Gallant,et al.  Neural network learning and expert systems , 1993 .

[29]  Victoria Y. Yoon,et al.  Artificial Neural Networks: An Emerging New Technique , 1992, Data Base.