An eigenvector based center selection for fast training scheme of RBFNN

Abstract The Radial Basis Function Neural Network (RBFNN) model is one of the most popular Feedforward Neural Network architectures. Calculating the proper RBF centers efficiently is one of the key problems in the configuration of an RBFNN model. In previous studies, clustering approaches, especially the k -means clustering, are most frequently employed to obtain the RBF centers. However, these approaches are usually time-consuming, particularly for the data sets with a relatively large scale. Meanwhile, some approaches have been proposed to save the training time by sacrificing the accuracy. This paper introduces an approach to quickly determine the RBF centers for an RBFNN model. An eigenvector based clustering method is employed to calculate the RBF centers in the input feature space. RBF centers for the RBFNN model thus can be determined very quickly by calculating the principal components of the data matrix instead of the iterative calculation process of k -means clustering. After that, the connecting weights of the network can be easily obtained via either pseudo-inverse solution or the gradient descent algorithm. To evaluate the proposed approach, the performance of RBFNNs trained via different training schemes is compared in the experiments. It shows that the proposed method greatly reduces the training time of an RBFNN while allowing the RBFNN to attain a comparable accuracy result.

[1]  K. Fan.  On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations: II. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[2]  Chris H. Q. Ding,et al.  Spectral Relaxation for K-means Clustering , 2001, NIPS.

[3]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[4]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[5]  C. Micchelli.  Interpolation of scattered data: Distance matrices and conditionally positive definite functions , 1986 .

[6]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[7]  David Haussler,et al.  Probably Approximately Correct Learning , 2010, Encyclopedia of Machine Learning.

[8]  Marc'Aurelio Ranzato,et al.  Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[10]  M. J. D. Powell,et al.  Radial basis functions for multivariable interpolation: a review , 1987 .

[11]  Kezhi Mao,et al.  RBF neural network center selection based on Fisher ratio class separability measure , 2002, IEEE Trans. Neural Networks.

[12]  Nikos A. Vlassis,et al.  The global k-means clustering algorithm , 2003, Pattern Recognit..

[13]  Richard M. Leahy,et al.  An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Qiang Du,et al.  Centroidal Voronoi Tessellations: Applications and Algorithms , 1999, SIAM Rev..

[15]  Dianhui Wang,et al.  Fast decorrelated neural network ensembles with random weights , 2014, Inf. Sci..

[16]  Yoh-Han Pao,et al.  Stochastic choice of basis functions in adaptive function approximation and the functional-link net , 1995, IEEE Trans. Neural Networks.

[17]  Panu Somervuo,et al.  Self-organizing maps of symbol strings , 1998, Neurocomputing.

[18]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[19]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[20]  M. Hoher,et al.  Similarities of LVQ and RBF learning-a survey of learning rules and the application to the classification of signals from high-resolution electrocardiography , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[21]  Yi Li,et al.  Principal Component Analysis for Clustering Temporomandibular Joint Data , 2015 .

[22]  Maria Cristina Felippetto de Castro,et al.  RBF neural networks with centers assignment via Karhunen-Loeve transform , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[23]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[24]  Jin Young Choi,et al.  Pre-clustered principal component analysis for fast training of new face databases , 2007, 2007 International Conference on Control, Automation and Systems.

[25]  R. Weber,et al.  A Rough-Fuzzy approach for Support Vector Clustering , 2016, Inf. Sci..

[26]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[27]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[28]  Chiung-Chou Liao,et al.  Genetic k-means algorithm based RBF network for photovoltaic MPP prediction , 2010 .

[29]  Teuvo Kohonen,et al.  Improved versions of learning vector quantization , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[30]  A. D. Gordon,et al.  An Algorithm for Euclidean Sum of Squares Classification , 1977 .

[31]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[32]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[33]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[34]  Hailin Li,et al.  Accurate and efficient classification based on common principal components analysis for multivariate time series , 2016, Neurocomputing.

[35]  Friedhelm Schwenker,et al.  Three learning phases for radial-basis-function networks , 2001, Neural Networks.

[36]  Sung-Kwun Oh,et al.  Design of K-means clustering-based polynomial radial basis function neural networks (pRBF NNs) realized with the aid of particle swarm optimization and differential evolution , 2012, Neurocomputing.

[37]  C. L. Philip Chen,et al.  A rapid learning and dynamic stepwise updating algorithm for flat neural networks and the application to time-series prediction , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[38]  Richard M. Watanabe,et al.  A Principal Components-Based Clustering Method to Identify Variants Associated with Complex Traits , 2011, Human Heredity.

[39]  Chris H. Q. Ding,et al.  K-means clustering via principal component analysis , 2004, ICML.

[40]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .