A Modified Sigma-Pi-Sigma Neural Network with Adaptive Choice of Multinomials

Sigma-Pi-Sigma neural networks (SPSNNs) as a kind of high-order neural networks can provide more powerful mapping capability than the traditional feedforward neural networks (Sigma-Sigma neural networks). In the existing literature, in order to reduce the number of the Pi nodes in the Pi layer, a special multinomial P_s is used in SPSNNs. Each monomial in P_s is linear with respect to each particular variable sigma_i when the other variables are taken as constants. Therefore, the monomials like sigma_i^n or sigma_i^n sigma_j with n>1 are not included. This choice may be somehow intuitive, but is not necessarily the best. We propose in this paper a modified Sigma-Pi-Sigma neural network (MSPSNN) with an adaptive approach to find a better multinomial for a given problem. To elaborate, we start from a complete multinomial with a given order. Then we employ a regularization technique in the learning process for the given problem to reduce the number of monomials used in the multinomial, and end up with a new SPSNN involving the same number of monomials (= the number of nodes in the Pi-layer) as in P_s. Numerical experiments on some benchmark problems show that our MSPSNN behaves better than the traditional SPSNN with P_s.

[1]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[2]  Wei Wu,et al.  Convergence of Batch BP Algorithm with Penalty for FNN Training , 2006, ICONIP.

[3]  D. Rumelhart,et al.  Generalization by weight-elimination applied to currency exchange rate prediction , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[4]  Zongben Xu,et al.  $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast Solver , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[6]  Xiaodong Liu,et al.  Batch gradient training method with smoothing $$\boldsymbol{\ell}_{\bf 0}$$ℓ0 regularization for feedforward neural networks , 2015, Neural Computing and Applications.

[7]  Huan Liu,et al.  Feature Selection for Classification: A Review , 2014, Data Classification: Algorithms and Applications.

[8]  Jacek M. Zurada,et al.  Batch gradient method with smoothing L1/2 regularization for training of feedforward neural networks , 2014, Neural Networks.

[9]  Mohammad Hassan Moradi,et al.  A Hybrid Higher Order Neural Classifier for handling classification problems , 2011, Expert Syst. Appl..

[10]  G. P. Liu,et al.  Nonlinear Identification and Control: A Neural Network Approach [Book Review] , 2002, IEEE Control Systems.

[11]  Jacek M. Zurada,et al.  Computational properties and convergence analysis of BPNN for cyclic and almost cyclic learning with penalty , 2012, Neural Networks.

[12]  Tong Zhang,et al.  Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization , 2013, Mathematical Programming.

[13]  Jing Wang,et al.  Convergence of batch gradient learning algorithm with smoothing L1/2 regularization for Sigma-Pi-Sigma neural networks , 2015, Neurocomputing.

[14]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[15]  Zongben Xu,et al.  L1/2 regularization , 2010, Science China Information Sciences.

[16]  Chien-Kuo Li.  A Sigma-Pi-Sigma Neural Network (SPSNN) , 2004, Neural Processing Letters.

[17]  Danilo Comminiello,et al.  Group sparse regularization for deep neural networks , 2016, Neurocomputing.

[18]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[19]  Bernardo A. Huberman,et al.  AN IMPROVED THREE LAYER, BACK PROPAGATION ALGORITHM , 1987 .

[20]  Madan M. Gupta,et al.  Higher Order Neural Networks: Fundamental Theory and Applications , 2010 .