Sensitivity Analysis for Probabilistic Neural Network Structure Reduction

In this paper, we propose the use of local sensitivity analysis (LSA) for the structure simplification of the probabilistic neural network (PNN). Three algorithms are introduced. The first algorithm applies LSA to the PNN input layer reduction by selecting significant features of input patterns. The second algorithm utilizes LSA to remove redundant pattern neurons of the network. The third algorithm combines the proposed two and constitutes the solution of how they can work together. PNN with a product kernel estimator is used, where each multiplicand computes a one-dimensional Cauchy function. Therefore, the smoothing parameter is separately calculated for each dimension by means of the plug-in method. The classification qualities of the reduced and full structure PNN are compared. Furthermore, we evaluate the performance of PNN, for which global sensitivity analysis (GSA) and the common reduction methods are applied, both in the input layer and the pattern layer. The models are tested on the classification problems of eight repository data sets. A 10-fold cross validation procedure is used to determine the prediction ability of the networks. Based on the obtained results, it is shown that the LSA can be used as an alternative PNN reduction approach.

[1]  Wee Ser,et al.  Probabilistic neural-network structure determination for pattern classification , 2000, IEEE Trans. Neural Networks Learn. Syst..

[2]  Piotr A. Kowalski,et al.  Data Sample Reduction for Classification of Interval Information Using Neural Network Sensitivity Analysis , 2010, AIMSA.

[3]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[4]  Piotr A. Kowalski,et al.  A Complete Algorithm for the Reduction of Pattern Data in the Classification of Interval Information , 2016 .

[5]  Michael R. Berthold,et al.  Constructive training of probabilistic neural networks , 1998, Neurocomputing.

[6]  Marko Robnik-Sikonja,et al.  Theoretical and Empirical Analysis of ReliefF and RReliefF , 2003, Machine Learning.

[7]  Piotr Kulczycki,et al.  Statistical kernel estimators for data analysis and exploration tasks: theory and applications , 2009 .

[8]  Hojjat Adeli,et al.  A probabilistic neural network for earthquake magnitude prediction , 2009, Neural Networks.

[9]  Leszek Rutkowski,et al.  Adaptive probabilistic neural networks for pattern classification in time-varying environment , 2004, IEEE Transactions on Neural Networks.

[10]  Evor L. Hines,et al.  Comparison of neural network predictors in the classification of tracheal-bronchial breath sounds by respiratory auscultation , 2004, Artif. Intell. Medicine.

[11]  Jacek M. Zurada,et al.  Sensitivity analysis for minimization of input data dimension for feedforward neural network , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[12]  Jacek M. Zurada,et al.  Perturbation method for deleting redundant inputs of perceptron networks , 1997, Neurocomputing.

[13]  Jeng-Shyang Pan,et al.  Kernel class-wise locality preserving projection , 2008, Inf. Sci..

[14]  Yong Wang,et al.  Complete neighborhood preserving embedding for face recognition , 2010, Pattern Recognit..

[15]  Ronald Marsh,et al.  Conjugate gradient and approximate Newton methods for an optimal probabilistic neural network for food color classification , 1998 .

[16]  A. Zaknich,et al.  A vector quantisation reduction method for the probabilistic neural network , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[17]  Roy L. Streit,et al.  Maximum likelihood training of probabilistic neural networks , 1994, IEEE Trans. Neural Networks.

[18]  Maciej Kusy,et al.  Assessment of prediction ability for reduced probabilistic neural network in data classification problems , 2017, Soft Comput..

[19]  Daniel S. Yeung,et al.  Diversified Sensitivity-Based Undersampling for Imbalance Classification Problems , 2015, IEEE Transactions on Cybernetics.

[20]  Q. Ruan,et al.  Orthogonal Laplacianfaces for 3D face recognition , 2010, 2010 2nd International Conference on Industrial and Information Systems.

[21]  Kjell Elenius,et al.  Multi-layer perceptrons and probabilistic neural networks for phoneme recognition , 1993, EUROSPEECH.

[22]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[23]  Dominique Bertrand,et al.  Reduction of the size of the learning data in a probabilistic neural network by hierarchical clustering. Application to the discrimination of seeds by artificial vision , 1996 .

[24]  Alfred Jean Philippe Lauret,et al.  A node pruning algorithm based on a Fourier amplitude sensitivity test method , 2006, IEEE Transactions on Neural Networks.

[25]  Patrick P. K. Chan,et al.  MLPNN Training via a Multiobjective Optimization of Training Error and Stochastic Sensitivity , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Daniel S. Yeung,et al.  Feature selection using localized generalization error for supervised classification problems using RBFNN , 2008, Pattern Recognit..

[27]  Srinivasan Ramakrishnan,et al.  Image texture classification using wavelet based curve fitting and probabilistic neural network , 2007, Int. J. Imaging Syst. Technol..

[28]  Hua Zhang,et al.  A new watermarking approach based on probabilistic neural network in wavelet domain , 2008, Soft Comput..

[29]  Maciej Kusy,et al.  Probabilistic Neural Network Structure Reduction for Medical Data Classification , 2013, ICAISC.

[30]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[31]  Andries Petrus Engelbrecht,et al.  A new pruning heuristic based on variance analysis of sensitivity information , 2001, IEEE Trans. Neural Networks.

[32]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[33]  George C. Anastassopoulos,et al.  Genetic algorithm pruning of probabilistic neural networks in medical disease estimation , 2011, Neural Networks.

[34]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[35]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[36]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[37]  Eric Fock,et al.  Global Sensitivity Analysis Approach for Input Selection and System Identification Purposes—A New Framework for Feedforward Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[38]  Daniel S. Yeung,et al.  Localized Generalization Error Model and Its Application to Architecture Selection for Radial Basis Function Neural Network , 2007, IEEE Transactions on Neural Networks.

[39]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[40]  Stefano Tarantola,et al.  Variance‐Based Methods , 2008 .

[41]  S. Gopal,et al.  Orthogonal least square center selection technique - A robust scheme for multiple source Partial Discharge pattern recognition using Radial Basis Probabilistic Neural Network , 2011, Expert Syst. Appl..

[42]  Donald F. Specht,et al.  Probabilistic neural networks and the polynomial Adaline as complementary techniques for classification , 1990, IEEE Trans. Neural Networks.

[43]  Patrick P. K. Chan,et al.  Steganalysis classifier training via minimizing sensitivity for different imaging sources , 2014, Inf. Sci..

[44]  Pietro Burrascano,et al.  Learning vector quantization for the probabilistic neural network , 1991, IEEE Trans. Neural Networks.

[45]  Chu Kiong Loo,et al.  A Global k-means Approach for Autonomous Cluster Initialization of Probabilistic Neural Network , 2008, Informatica.

[46]  Maciej Kusy,et al.  Application of Reinforcement Learning Algorithms for the Adaptive Computation of the Smoothing Parameter for Probabilistic Neural Network , 2015, IEEE Transactions on Neural Networks and Learning Systems.