A Polynomial Neural Network with Controllable Precision and Human-Readable Topology for Prediction and System Identification

Although artificial neural networks (ANNs) are successful, there is still a concern among many over their "black box" nature. Why do they work? Could we design a "transparent" network? This paper presents a controllable and readable polynomial neural network (CR-PNN) for approximation, prediction, and system identification. CR-PNN is simple enough to be described as one "small" formula, so that we can control the approximation precision and explain the internal structure of the network. CR-PNN, in fact, essentially is the fascinating Taylor expansion in the form of network. The number of layers represents precision. Derivatives in Taylor expansion are exactly imitated by error back-propagation algorithm. Firstly, we demonstrated that CR-PNN shows excellent analysis performance to the "black box" system through ten synthetic data with noise. Also, the results were compared with synthetic data to substantiate its search easily towards the global optimum. Secondly, it was verified, by ten real-world applications, that CR-PNN brought better generalization capability relative to the typical ANNs that approximate depended on the nonlinear activation function. Finally, 200,000 repeated experiments, with 4898 samples, demonstrated that CR-PNN is five times more efficient than typical ANN for one epoch and ten times more efficient than typical ANN for one forward-propagation. In short, compared with the traditional neural networks, the novelties and advantages of CR-PNN include readability of the internal structure, easy to find global optimal solution, lower computational complexity, and likely better robustness to real-world approximation. (We're strong believers in Open Source, and provide CR-PNN code for others. GitHub: this https URL)

[1]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[2]  Xi Cheng,et al.  Polynomial Regression As an Alternative to Neural Nets , 2018, ArXiv.

[3]  J. Ross Quinlan,et al.  Combining Instance-Based and Model-Based Learning , 1993, ICML.

[4]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[5]  Yi-Cheng Liu,et al.  Using mixture design and neural networks to build stock selection decision support systems , 2017, Neural Computing and Applications.

[6]  I-Cheng Yeh,et al.  Building real estate valuation models with comparative approach through case-based reasoning , 2018, Appl. Soft Comput..

[7]  José Antonio Lozano,et al.  Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Paulo Cortez,et al.  Modeling wine preferences by data mining from physicochemical properties , 2009, Decis. Support Syst..

[9]  Jiujun Cheng,et al.  Dendritic Neuron Model With Effective Learning Algorithms for Classification, Approximation, and Prediction , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[11]  Shuai Li,et al.  A Novel Recurrent Neural Network for Manipulator Control With Improved Noise Tolerance , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Feng Ding,et al.  Modified Gram–Schmidt Method-Based Variable Projection Algorithm for Separable Nonlinear Models , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Haoyong Yu,et al.  Biomimetic Hybrid Feedback Feedforward Neural-Network Learning Control , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Rich Caruana,et al.  Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping , 2000, NIPS.

[15]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[16]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[17]  Feng Yu,et al.  A short-term load forecasting model of natural gas based on optimized genetic algorithm and improved BP neural network , 2014 .

[18]  R Todeschini,et al.  A similarity-based QSAR model for predicting acute toxicity towards the fathead minnow (Pimephales promelas). , 2015, SAR and QSAR in environmental research (Print).

[19]  Renato José Sassi,et al.  Traffic flow breakdown prediction using feature reduction through Rough-Neuro Fuzzy Networks , 2011, The 2011 International Joint Conference on Neural Networks.

[20]  Sung-Kwun Oh,et al.  Polynomial neural networks architecture: analysis and design , 2003, Comput. Electr. Eng..

[21]  I-Cheng Yeh,et al.  Modeling of strength of high-performance concrete using artificial neural networks , 1998 .

[22]  R. J. Sassi,et al.  Study on Daily Demand Forecasting Orders using Artificial Neural Network , 2016, IEEE Latin America Transactions.

[23]  P. Cortez,et al.  A data mining approach to predict forest fires using meteorological data , 2007 .

[24]  T. Poggio,et al.  On optimal nonlinear associative recall , 1975, Biological Cybernetics.

[25]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.