Evaluation of prototype learning algorithms for nearest-neighbor classifier in application to handwritten character recognition

Abstract Prototype learning is effective in improving the classification performance of nearest-neighbor (NN) classifier and in reducing the storage and computation requirements. This paper reviews some prototype learning algorithms for NN classifier design and evaluates their performance in application to handwritten character recognition. The algorithms include the well-known LVQ and some parameter optimization approaches that aim to minimize an objective function by gradient search. We also propose some new algorithms based on parameter optimization and evaluate their performance together with the existing ones. Eleven prototype learning algorithms are tested in handwritten numeral recognition on the CENPARMI database and in handwritten Chinese character recognition on the ETL8B2 database. The experimental results show that the algorithms based on parameter optimization generally outperform the LVQ. Particularly, the minimum classification error (MCE) approach of Juang and Katagiri (IEEE Trans. Signal Process. 40 (12) (1992) 3043), the generalized LVQ (GLVQ) of Sato and Yamada (Proceedings of the 14th ICPR, Vol. I, Brisbane, 1998, p. 322) and a new algorithm MAXP1 yield best results.

[1]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[2]  Yoshihiko Hamamoto,et al.  A Bootstrap Technique for Nearest Neighbor Classifier Design , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Seong-Whan Lee,et al.  Nonlinear shape normalization methods for the recognition of large-set handwritten characters , 1994, Pattern Recognit..

[4]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[5]  C. G. Hilborn,et al.  The Condensed Nearest Neighbor Rule , 1967 .

[6]  Peter E. Hart,et al.  The condensed nearest neighbor rule (Corresp.) , 1968, IEEE Trans. Inf. Theory.

[7]  Cheng-Lin Liu,et al.  Preprocessing and statistical/structural feature extraction for handwritten numeral recognition , 1997 .

[8]  Chin-Liang Chang,et al.  Finding Prototypes For Nearest Neighbor Classifiers , 1974, IEEE Transactions on Computers.

[9]  Seong-Whan Lee,et al.  Optimal design of reference models for large-set handwritten character recognition , 1994, Pattern Recognit..

[10]  Elias N. Houstis,et al.  On neurobiological, neuro-fuzzy, machine learning, and statistical pattern recognition techniques , 1997, IEEE Trans. Neural Networks.

[11]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[12]  Uri Lipowezky.  Selection of the optimal prototype subset for 1-NN classification , 1998, Pattern Recognit. Lett..

[13]  Horst Bunke,et al.  Handbook of Character Recognition and Document Image Analysis , 1997 .

[14]  Shlomo Geva,et al.  Adaptive nearest neighbor pattern classification , 1991, IEEE Trans. Neural Networks.

[15]  Kazuhiko Yamamoto,et al.  Research on Machine Recognition of Handprinted Characters , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Keinosuke Fukunaga,et al.  A Branch and Bound Algorithm for Computing k-Nearest Neighbors , 1975, IEEE Transactions on Computers.

[17]  Biing-Hwang Juang,et al.  Discriminative learning for minimum error classification [pattern recognition] , 1992, IEEE Trans. Signal Process..

[18]  Kiichi Urahama,et al.  Gradient descent learning of nearest neighbor classifiers with outlier rejection , 1995, Pattern Recognit..

[19]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[20]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[21]  Atsushi Sato,et al.  A formulation of learning vector quantization using a new misclassification measure , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[22]  Hiromitsu Yamada,et al.  A nonlinear normalization method for handprinted kanji character recognition - line density equalization , 1990, Pattern Recognit..

[23]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[24]  Jin Hyung Kim,et al.  High accuracy handwritten Chinese character recognition by improved feature matching method , 1997, Proceedings of the Fourth International Conference on Document Analysis and Recognition.

[25]  Christine Decaestecker,et al.  Finding prototypes for nearest neighbour classification by means of gradient descent and deterministic annealing , 1997, Pattern Recognit..

[26]  G. Gates.  The Reduced Nearest Neighbor Rule , 1998 .

[27]  Hong Yan,et al.  Prototype optimization for nearest neighbor classifiers using a two-layer perceptron , 1993, Pattern Recognit..

[28]  Seong-Whan Lee,et al.  Multilayer cluster neural network for totally unconstrained handwritten numeral recognition , 1995, Neural Networks.

[29]  Thierry Denoeux,et al.  A k-nearest neighbor classification rule based on Dempster-Shafer theory , 1995, IEEE Trans. Syst. Man Cybern..

[30]  Ching Y. Suen,et al.  A simulated annealing approach to construct optimized prototypes for nearest-neighbor classification , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[31]  Rabab Kreidieh Ward,et al.  Vector Quantization Technique for Nonparametric Classifier Design , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  T. Kohonen,et al.  Statistical pattern recognition with neural networks: benchmarking studies , 1988, IEEE 1988 International Conference on Neural Networks.

[33]  James C. Bezdek,et al.  Multiple-prototype classifier design , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[34]  G. Gates,et al.  The reduced nearest neighbor rule (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[35]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Rama Chellappa,et al.  Evaluation of pattern classifiers for fingerprint and OCR applications , 1994, Pattern Recognit..

[37]  Hermann Ney,et al.  On the Probabilistic Interpretation of Neural Network Classifiers and Discriminative Training Criteria , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[39]  Keinosuke Fukunaga,et al.  An Optimal Global Nearest Neighbor Metric , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Seong-Whan Lee,et al.  LVQ combined with simulated annealing for optimal design of large-set reference models , 1996, Neural Networks.

[41]  Francesco Ricci,et al.  Data Compression and Local Metrics for Nearest Neighbor Classification , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Francesc J. Ferri,et al.  Considerations about sample-size sensitivity of a family of edited nearest-neighbor rules , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[43]  Patrick Grother,et al.  Fast implementations of nearest neighbor classifiers , 1997, Pattern Recognit..

[44]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[45]  Kenneth Rose,et al.  A global optimization technique for statistical classifier design , 1996, IEEE Trans. Signal Process..

[46]  Belur V. Dasarathy,et al.  Nearest neighbor (NN) norms: NN pattern classification techniques , 1991 .

[47]  Frans C. A. Groen,et al.  The box-cox metric for nearest neighbour classification improvement , 1997, Pattern Recognit..

[48]  Jürgen Franke.  ISOLATED HANDPRINTED DIGIT RECOGNITION , 1997 .

[49]  Atsushi Sato,et al.  Generalized Learning Vector Quantization , 1995, NIPS.

[50]  Wentai Liu,et al.  Optical recognition of handwritten Chinese characters: Advances since 1980 , 1993, Pattern Recognit..

[51]  Essaid Bouktache,et al.  A Fast Algorithm for the Nearest-Neighbor Classifier , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Keinosuke Fukunaga,et al.  The optimal distance measure for nearest neighbor classification , 1981, IEEE Trans. Inf. Theory.

[53]  Erkki Oja,et al.  Neural and statistical classifiers-taxonomy and two case studies , 1997, IEEE Trans. Neural Networks.

[54]  Ching Y. Suen,et al.  Computer recognition of unconstrained handwritten numerals , 1992, Proc. IEEE.

[55]  James C. Bezdek,et al.  Nearest prototype classification: clustering, genetic algorithms, or random search? , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[56]  Teuvo Kohonen,et al.  Improved versions of learning vector quantization , 1990, 1990 IJCNN International Joint Conference on Neural Networks.