On the Behavior of Artificial Neural Network Classifiers in High-Dimensional Spaces

It is widely believed in the pattern recognition field that when a fixed number of training samples is used to design a classifier, the generalization error of the classifier tends to increase as the number of features gets larger. In this paper, we discuss the generalization error of the artificial neural network (ANN) classifiers in high-dimensional spaces, under a practical condition that the ratio of the training sample size to the dimensionality is small. Experimental results show that the generalization error of ANN classifiers seems much less sensitive to the feature size than 1-NN, Parzen and quadratic classifiers.

[1]  R. Lippmann.  Pattern classification using neural networks , 1989, IEEE Communications Magazine.

[2]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[3]  A. Khotanzad,et al.  Distortion invariant character recognition by a multi-layer perceptron and back-propagation learning , 1988, IEEE 1988 International Conference on Neural Networks.

[4]  Gerald Tesauro,et al.  How Tight Are the Vapnik-Chervonenkis Bounds? , 1992, Neural Computation.

[5]  Keinosuke Fukunaga,et al.  Bayes Error Estimation Using Parzen and k-NN Procedures , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Shun-ichi Amari,et al.  Four Types of Learning Curves , 1992, Neural Computation.

[7]  Anil K. Jain,et al.  39 Dimensionality and sample size considerations in pattern recognition practice , 1982, Classification, Pattern Recognition and Reduction of Dimensionality.

[8]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[9]  Anil K. Jain,et al.  Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Keinosuke Fukunaga,et al.  Bias of Nearest Neighbor Error Estimates , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Eric B. Baum,et al.  When Are k-Nearest Neighbor and Back Propagation Accurate for Feasible Sized Sets of Examples? , 1990, EURASIP Workshop.

[12]  Sariinas Ra Ud Ys.  ON THE EFFECTIVENESS OF PARZEN WINDOW CLASSIFIER , 1991 .

[13]  John Van Ness,et al.  On the dominance of non-parametric Bayes rule discriminant algorithms in high dimensions , 1980, Pattern Recognit..

[14]  Anil K. Jain,et al.  Classifier design with Parzen Windows , 1988 .

[15]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[16]  Keinosuke Fukunaga,et al.  Effects of Sample Size in Classifier Design , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[18]  Anil K. Jain,et al.  Small sample size problems in designing artificial neural networks , 1991 .

[19]  Alireza Khotanzad,et al.  Shape and Texture Recognition by a Neural Network , 1991 .

[20]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[21]  S. Raudys,et al.  Why do multilayer perceptrons have favorable small sample properties , 1994 .

[22]  G. F. Hughes,et al.  On the mean accuracy of statistical pattern recognizers , 1968, IEEE Trans. Inf. Theory.

[23]  Šarūnas Raudys.  On the effectiveness of Parzen window classifier , 1991 .

[24]  Sariinas Raudys,et al.  ON SHAPE OF PATTERN ERROR FUNCTION, INITIALIZATIONS AND INTRINSIC DIMENSIONALITY IN ANN CLASSIFIER DESIGN , 1993 .