DistAl: an inter-pattern distance-based constructive learning algorithm

Multilayer networks of threshold logic units offer an attractive framework for the design of pattern classification systems. A new constructive neural network learning algorithm (DistAl) based on inter-pattern distance is introduced. DistAl constructs a single hidden layer of spherical threshold neurons. Each neuron is designed to exclude a cluster of training patterns belonging to the same class. The weights and thresholds of the hidden neurons are determined directly by comparing the inter-pattern distances of the training patterns. This offers a significant advantage over other constructive learning algorithms that use an iterative weight modification strategy to train individual neurons. The individual clusters are combined by a single output layer of threshold neurons. The speed of DistAl makes it a good candidate for data mining and knowledge acquisition from very large data sets. Results of experiments show that DistAl compares favorably with other neural network learning algorithms for pattern classification.

[1]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Pattern Classification , 1995 .

[2]  Ron Kohavi.  Feature Subset Selection as Search with Probabilistic Estimates , 1994 .

[3]  Rajesh Parekh,et al.  Constructive theory refinement in knowledge based neural networks , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[4]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[5]  Rajesh Parekh,et al.  Constructive learning: inducing grammars and neural networks , 1998 .

[6]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[7]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[8]  Sebastian Thrun,et al.  Lifelong Learning: A Case Study. , 1995 .

[9]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[10]  Nils J. Nilsson,et al.  The Mathematical Foundations of Learning Machines , 1990 .

[11]  S. Grossberg,et al.  Pattern Recognition by Self-Organizing Neural Networks , 1991 .

[12]  Tony R. Martinez,et al.  Improved Heterogeneous Distance Functions , 1996, J. Artif. Intell. Res..

[13]  Jihoon Yang,et al.  Pruning strategies for the MTiling constructive learning algorithm , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[14]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[15]  Vasant Honavar,et al.  Mobile Intelligent Agents for Document Classification and Retrieval: A Machine Learning Approach , 1998 .

[16]  Rajesh Parekh,et al.  Analysis of Decision Boundaries Generated by Constructive Neural Network Learning Algorithms , 1995 .

[17]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Real-Valued Pattern Classification , 1997 .

[18]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[19]  Marcus R. Frean,et al.  A "Thermal" Perceptron Learning Rule , 1992, Neural Computation.

[20]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (3rd ed.) , 1996 .

[21]  Neil Burgess,et al.  A Constructive Algorithm that Converges for Real-Valued Input Patterns , 1994, Int. J. Neural Syst..

[22]  Janet L. Kolodner,et al.  Case-Based Reasoning , 1989, IJCAI 1989.

[23]  B. Natarajan.  Machine Learning: A Theoretical Approach , 1992 .

[24]  Vasant Honavar,et al.  Toward learning systems that integrate multiple strategies and representations , 1994 .

[25]  Pat Langley,et al.  Elements of Machine Learning , 1995 .

[26]  Vasant Honavar,et al.  Intelligent Diagnosis Systems , 1998 .

[27]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[28]  R. Nosofsky.  Attention, similarity, and the identification-categorization relationship. , 1986, Journal of experimental psychology. General.

[29]  Daphne Koller,et al.  Hierarchically Classifying Documents Using Very Few Words , 1997, ICML.

[30]  David G. Lowe,et al.  Similarity Metric Learning for a Variable-Kernel Classifier , 1995, Neural Computation.

[31]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[32]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[33]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[34]  Jude Shavlik,et al.  A Framework for Combining Symbolic and Neural Learning , 1992 .

[35]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[36]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[37]  Rajesh Parekh,et al.  Pruning strategies for constructive neural network learning algorithms , 1997 .

[38]  Michael Biehl,et al.  Properties of an adaptive perceptron algorithm , 1990 .

[39]  Jean-Pierre Nadal,et al.  Study of a Growth Algorithm for a Feedforward Network , 1989, Int. J. Neural Syst..

[40]  G Salton,et al.  Developments in Automatic Text Retrieval , 1991, Science.

[41]  Peter D. Turney.  Theoretical analyses of cross-validation error and voting in instance-based learning , 1994, J. Exp. Theor. Artif. Intell..

[42]  Vasant Honavar,et al.  Generative learning structures for generalized connectionist networks , 1990 .

[43]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[44]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[45]  Marco Richeldi,et al.  Performing Effective Feature Selection by Investigating the Deep Structure of the Data , 1996, KDD.

[46]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[47]  David W. Aha,et al.  Incremental Constructive Induction: An Instance-Based Approach , 1991, ML.

[48]  Jeffrey M. Bradshaw,et al.  Software agents , 1997 .

[49]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[50]  Nada Lavrac,et al.  The Multi-Purpose Incremental Learning System AQ15 and Its Testing Application to Three Medical Domains , 1986, AAAI.

[51]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[52]  Ah Chung Tsoi,et al.  A Constructive Algorithm for the Training of a Multilayer Perceptron Based on the Genetic Algorithm , 1993, Complex Syst..

[53]  A. Tversky.  Features of Similarity , 1977 .

[54]  Bruce G. Batchelor,et al.  Pattern Recognition: Ideas in Practice , 1978 .

[55]  Jihoon Yang,et al.  Feature Subset Selection Using a Genetic Algorithm , 1998, IEEE Intell. Syst..

[56]  Bruno Raffin,et al.  Learning and Generalization with Minimerror, A Temperature-Dependent Learning Algorithm , 1995, Neural Computation.

[57]  Sholom M. Weiss,et al.  An Empirical Comparison of Pattern Recognition, Neural Nets, and Machine Learning Classification Methods , 1989, IJCAI.

[58]  DistAl: An inter-pattern distance-based constructive learning algorithm , 1999, Intell. Data Anal..

[59]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[60]  Vasant Honavar,et al.  Experiments with the cascade-correlation algorithm , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[61]  Stephen I. Gallant,et al.  Neural network learning and expert systems , 1993 .

[62]  Vasant Honavar.  Machine learning: Principles and applications , 1999 .

[63]  David W. Aha,et al.  Noise-Tolerant Instance-Based Learning Algorithms , 1989, IJCAI.

[64]  M. Golea,et al.  A Growth Algorithm for Neural Network Decision Trees , 1990 .

[65]  M. Golea,et al.  A Convergence Theorem for Sequential Learning in Two-Layer Perceptrons , 1990 .

[66]  Vasant Honavar,et al.  Generative learning structures and processes for generalized connectionist networks , 1993, Inf. Sci..

[67]  M. J. D. Powell,et al.  Radial basis functions for multivariable interpolation: a review , 1987 .

[68]  Jihoon Yang,et al.  MUpstart-a constructive neural network learning algorithm for multi-category pattern classification , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[69]  Pedro M. Domingos.  Rule Induction and Instance-Based Learning: A Unified Approach , 1995, IJCAI.

[70]  Stephen I. Gallant,et al.  Perceptron-based learning algorithms , 1990, IEEE Trans. Neural Networks.