Application of the cross entropy method to the GLVQ algorithm

This paper discusses an alternative approach to parameter optimization of well-known prototype-based learning algorithms (minimizing an objective function via gradient search). The proposed approach considers a stochastic optimization called the cross entropy method (CE method). The CE method is used to tackle efficiently the initialization sensitiveness problem associated with the original generalized learning vector quantization (GLVQ) algorithm and its variants. Results presented in this paper indicate that the CE method can be successfully applied to this kind of problem on real-world data sets. As far as known by the authors, it is the first use of the CE method in prototype-based learning.

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  Bjarne E. Helvik,et al.  Cross entropy guided ant-like agents finding dependable primary/backup path patterns in networks , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[3]  Koby Crammer,et al.  Margin Analysis of the LVQ Algorithm , 2002, NIPS.

[4]  A. Kai Qin,et al.  Initialization insensitive LVQ algorithm based on cost-function adaptation , 2005, Pattern Recognit..

[5]  Arnaud Doucet,et al.  The cross-entropy method for blind multiuser detection , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[6]  Thomas Villmann,et al.  Relevance LVQ versus SVM , 2004, ICAISC.

[7]  Venkatesan Guruswami,et al.  2006 IEEE International Symposium on Information Theory , 2006 .

[8]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[9]  S. Katagiri,et al.  Discriminative Learning for Minimum Error Classification , 2009 .

[10]  Atsushi Sato,et al.  A formulation of learning vector quantization using a new misclassification measure , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[11]  Dirk P. Kroese,et al.  The Cross-Entropy Method for Continuous Multi-Extremal Optimization , 2006 .

[12]  Thomas Villmann,et al.  Supervised Neural Gas with General Similarity Measure , 2005, Neural Processing Letters.

[13]  Dirk P. Kroese,et al.  Global likelihood optimization via the cross-entropy method with an application to mixture models , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[14]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[15]  Dirk P. Kroese,et al.  The Cross Entropy Method: A Unified Approach To Combinatorial Optimization, Monte-carlo Simulation (Information Science and Statistics) , 2004 .

[16]  Arnaldo Spalvieri,et al.  Certain facts about Kohonen's LVQ1 algorithm , 1996 .

[17]  Reuven Y. Rubinstein,et al.  Optimization of computer simulation models with rare events , 1997 .

[18]  Thomas Villmann,et al.  Generalized relevance learning vector quantization , 2002, Neural Networks.

[19]  B. Hammer,et al.  Monitoring technical systems with prototype based clustering , 2003 .

[20]  Dirk P. Kroese,et al.  Application of the cross-entropy method to clustering and vector quantization , 2007, J. Glob. Optim..

[21]  Fred W. Glover,et al.  Simulation optimization: a review, new developments, and applications , 2005, Proceedings of the Winter Simulation Conference, 2005..

[22]  Jue Wu,et al.  Cross Entropy: A New Solver for Markov Random Field Modeling and Applications to Medical Image Segmentation , 2005, MICCAI.

[23]  Panu Somervuo,et al.  Self-Organizing Maps and Learning Vector Quantization for Feature Sequences , 1999, Neural Processing Letters.

[24]  Dirk P. Kroese,et al.  The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning , 2004 .

[25]  Dirk P. Kroese,et al.  Cross‐Entropy Method , 2011 .

[26]  Biing-Hwang Juang,et al.  Discriminative learning for minimum error classification [pattern recognition] , 1992, IEEE Trans. Signal Process..

[27]  Barbara Hammer,et al.  Relevance determination in Learning Vector Quantization , 2001, ESANN.

[28]  Teuvo Kohonen,et al.  Improved versions of learning vector quantization , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[29]  Tito Homem-de-Mello,et al.  Solving the Vehicle Routing Problem with Stochastic Demands using the Cross-Entropy Method , 2005, Ann. Oper. Res..

[30]  Tony R. Martinez,et al.  Distribution-balanced stratified cross-validation for accuracy estimation , 2000, J. Exp. Theor. Artif. Intell..

[31]  Gert Pfurtscheller,et al.  Distinction Sensitive Learning Vector Quantisation-a new noise-insensitive classification method , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[32]  Shie Mannor,et al.  A Tutorial on the Cross-Entropy Method , 2005, Ann. Oper. Res..