Determining significance of input neurons for probabilistic neural network by sensitivity analysis procedure

In classical feedforward neural networks such as multilayer perceptron, radial basis function network, or counter‐propagation network, the neurons in the input layer correspond to features of the training patterns. The number of these features may be large, and their meaningfulness can be various. Therefore, the selection of appropriate input neurons should be regarded. The aim of this paper is to present a complete step‐by‐step algorithm for determining the significance of particular input neurons of the probabilistic neural network (PNN). It is based on the sensitivity analysis procedure applied to a trained PNN. The proposed algorithm is utilized in the task of reduction of the input layer of the considered network, which is achieved by removing appropriately indicated features from the data set. For comparison purposes, the PNN's input neuron significance is established by using the ReliefF and variable importance procedures that provide the relevance of the input features in the data set. The performance of the reduced PNN is verified against a full structure network in classification problems using real benchmark data sets from an available machine learning repository. The achieved results are also referred to the ones attained by entropy‐based algorithms. The prediction ability expressed in terms of misclassifications is obtained by means of a 10‐fold cross‐validation procedure. Received outcomes point out interesting properties of the proposed algorithm. It is shown that the efficiency determined by all tested reduction methods is comparable.

[1]  Dominique Bertrand,et al.  Reduction of the size of the learning data in a probabilistic neural network by hierarchical clustering. Application to the discrimination of seeds by artificial vision , 1996 .

[2]  Jianchuan Yin,et al.  Time-delay wavelet network predictor based on sensitivity analysis with application to predictive ship course control , 2016, J. Comput. Methods Sci. Eng..

[3]  Claudio De Stefano,et al.  A GA-based feature selection approach with an application to handwritten character recognition , 2014, Pattern Recognit. Lett..

[4]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[5]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[6]  Roy L. Streit,et al.  Maximum likelihood training of probabilistic neural networks , 1994, IEEE Trans. Neural Networks.

[7]  Maciej Kusy,et al.  Application of Reinforcement Learning Algorithms for the Adaptive Computation of the Smoothing Parameter for Probabilistic Neural Network , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[9]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[10]  Jacek M. Zurada,et al.  Sensitivity analysis for minimization of input data dimension for feedforward neural network , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[11]  Jacek M. Zurada,et al.  Perturbation method for deleting redundant inputs of perceptron networks , 1997, Neurocomputing.

[12]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[13]  Piotr A. Kowalski,et al.  Interval probabilistic neural network , 2015, Neural Computing and Applications.

[14]  Eric Fock,et al.  Global Sensitivity Analysis Approach for Input Selection and System Identification Purposes—A New Framework for Feedforward Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Kjell Elenius,et al.  Multi-layer perceptrons and probabilistic neural networks for phoneme recognition , 1993, EUROSPEECH.

[16]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[17]  Andries Petrus Engelbrecht,et al.  A new pruning heuristic based on variance analysis of sensitivity information , 2001, IEEE Trans. Neural Networks.

[18]  Piotr A. Kowalski,et al.  Sensitivity Analysis for Probabilistic Neural Network Structure Reduction , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Donald F. Specht,et al.  Probabilistic neural networks and the polynomial Adaline as complementary techniques for classification , 1990, IEEE Trans. Neural Networks.

[20]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[21]  Maciej Kusy.  Dimensionality Reduction for Probabilistic Neural Network in Medical Data Classification Problems , 2015 .

[22]  Ivan Tomov Dimov,et al.  A neural network sensitivity analysis in the presence of random fluctuations , 2017, Neurocomputing.

[23]  O. Mangasarian,et al.  Multisurface method of pattern separation for medical diagnosis applied to breast cytology. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[24]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[25]  Martin Krzywinski,et al.  Points of significance: P values and the search for significance , 2016, Nature Methods.

[26]  Piotr A. Kowalski,et al.  A Complete Algorithm for the Reduction of Pattern Data in the Classification of Interval Information , 2016 .

[27]  Marko Robnik-Sikonja,et al.  Theoretical and Empirical Analysis of ReliefF and RReliefF , 2003, Machine Learning.

[28]  Hojjat Adeli,et al.  A probabilistic neural network for earthquake magnitude prediction , 2009, Neural Networks.

[29]  D.S. Yeung,et al.  Input dimensionality reduction for radial basis neural network classification problems using sensitivity measure , 2002, Proceedings. International Conference on Machine Learning and Cybernetics.

[30]  M. Krzywinski,et al.  Points of Significance: Interpreting P values , 2017, Nature Methods.

[31]  Daming Shi,et al.  Sensitivity analysis applied to the construction of radial basis function networks , 2005, Neural Networks.

[32]  Zhi-Hong Mao,et al.  Dimensionality Reduction in Control and Coordination of the Human Hand , 2008, IEEE Transactions on Biomedical Engineering.

[33]  Pietro Burrascano,et al.  Learning vector quantization for the probabilistic neural network , 1991, IEEE Trans. Neural Networks.

[34]  Piotr A. Kowalski,et al.  Complete Gradient Clustering Algorithm for Features Analysis of X-Ray Images , 2010 .

[35]  Michael R. Berthold,et al.  Constructive training of probabilistic neural networks , 1998, Neurocomputing.

[36]  Piotr Duda,et al.  New Splitting Criteria for Decision Trees in Stationary Data Streams , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Chong-Ho Choi,et al.  Input feature selection for classification problems , 2002, IEEE Trans. Neural Networks.