A study on the use of statistical tests for experimentation with neural networks: Analysis of parametric test conditions and non-parametric tests

In this paper, we focus on the experimental analysis on the performance in artificial neural networks with the use of statistical tests on the classification task. Particularly, we have studied whether the sample of results from multiple trials obtained by conventional artificial neural networks and support vector machines checks the necessary conditions for being analyzed through parametrical tests. The study is conducted by considering three possibilities on classification experiments: random variation in the selection of test data, the selection of training data and internal randomness in the learning algorithm.The results obtained state that the fulfillment of these conditions are problem-dependent and indefinite, which justifies the need of using non-parametric statistics in the experimental analysis.

[1]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[2]  Ignacio Rojas,et al.  Statistical analysis of the parameters of a neuro-genetic algorithm , 2002, IEEE Trans. Neural Networks.

[3]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[4]  Nicolás García-Pedrajas,et al.  Immune Network based Ensembles , 2007, ESANN.

[5]  Ethem Alpaydın,et al.  Combined 5 x 2 cv F Test for Comparing Supervised Classification Learning Algorithms , 1999, Neural Comput..

[6]  Handbook of Parametric and Nonparametric Statistical Procedures , 2004 .

[7]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[8]  Xinxing Wu,et al.  Constructing of the risk classification model of cervical cancer by artificial neural network , 2007, Expert Syst. Appl..

[9]  James C. Bezdek,et al.  Nearest prototype classifier designs: An experimental study , 2001, Int. J. Intell. Syst..

[10]  Shian-Chang Huang,et al.  Evaluation of ANN and SVM classifiers as predictors to the diagnosis of students with learning disabilities , 2008, Expert Syst. Appl..

[11]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[12]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[13]  I. Song,et al.  Working Set Selection Using Second Order Information for Training Svm, " Complexity-reduced Scheme for Feature Extraction with Linear Discriminant Analysis , 2022 .

[14]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[15]  M. Zekic-Susac,et al.  Modeling computer and web attitudes using neural networks , 2005, 27th International Conference on Information Technology Interfaces, 2005..

[16]  Ah Chung Tsoi,et al.  On the distribution of performance from multiple neural-network trials , 1997, IEEE Trans. Neural Networks.

[17]  Steve W. Smye,et al.  A Comparison of Cox Regression and Neural Networks for Risk Stratification in Cases of Acute Lymphoblastic Leukaemia in Children , 1999, Neural Computing & Applications.

[18]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[19]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[20]  Sheng-Tun Li,et al.  The evaluation of consumer loans using support vector machines , 2006, Expert Syst. Appl..

[21]  Yong-Soo Kim,et al.  Comparison of the decision tree, artificial neural network, and linear regression methods based on the number and types of independent variables and sample size , 2008, Expert Syst. Appl..

[22]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[23]  Elisa Guerrero Vázquez,et al.  Multiple comparison procedures applied to model selection , 2002, Neurocomputing.

[24]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[25]  S. P. Wright,et al.  Adjusted P-values for simultaneous inference , 1992 .

[26]  Monica Lam,et al.  Neural network techniques for financial performance prediction: integrating fundamental and technical analysis , 2004, Decis. Support Syst..

[27]  Bernhard Schölkopf,et al.  New Support Vector Algorithms , 2000, Neural Computation.

[28]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[29]  Chih-Jen Lin,et al.  Working Set Selection Using Second Order Information for Training Support Vector Machines , 2005, J. Mach. Learn. Res..

[30]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[31]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[32]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[33]  Y Lu,et al.  A Sequential Learning Scheme for Function Approximation Using Minimal Radial Basis Function Neural Networks , 1997, Neural Computation.

[34]  G. Lyons,et al.  Bayesian ANN classifier for ECG arrhythmia diagnostic system: a comparison study , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[35]  R. Iman,et al.  Approximations of the critical region of the fbietkan statistic , 1980 .