Echo-State Restricted Boltzmann Machines: A Perspective on Information Compensation

Feature learning has been introduced in the modeling of echo state networks (ESNs). However, the procedure of feature learning is generally accompanied by certain information loss. In this regard, this paper proposes a hybrid neural network model from the information compensation view, named echo-state restricted Boltzmann machine (ERBM). It is deemed as a unified and coherent architecture with the successive functionalities of feature learning, information compensation, input superposition, and supervised nonlinear approximation. This is the first systematic model seeking to improve the ESN representative by that a direct weighted compensation channel is built to enhance feature learning. On the widely used benchmarks, we demonstrate that ERBM is fully competent to handle the nonlinear approximation tasks and superior to the state of the art in nonlinear approximation, robustness, and memory capacity.

[1]  Guan Gui,et al.  Co-Robust-ADMM-Net: Joint ADMM Framework and DNN for Robust Sparse Composite Regularization , 2018, IEEE Access.

[2]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[3]  Walid Saad,et al.  Echo State Networks for Self-Organizing Resource Allocation in LTE-U With Uplink–Downlink Decoupling , 2016, IEEE Transactions on Wireless Communications.

[4]  Alfredo De Santis,et al.  Network anomaly detection with the restricted Boltzmann machine , 2013, Neurocomputing.

[5]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[6]  Simone Scardapane,et al.  Semi-supervised Echo State Networks for Audio Classification , 2017, Cognitive Computation.

[7]  Junfei Qiao,et al.  Growing Echo-State Network With Multiple Subreservoirs , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Xiaohua Wang,et al.  Echo State Networks With Orthogonal Pigeon-Inspired Optimization for Image Restoration , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Yiannis Demiris,et al.  Echo State Gaussian Process , 2011, IEEE Transactions on Neural Networks.

[10]  Jang Myung Lee,et al.  Fuzzy Echo State Neural Networks and Funnel Dynamic Surface Control for Prescribed Performance of a Nonlinear Dynamic System , 2014, IEEE Transactions on Industrial Electronics.

[11]  Geoffrey E. Hinton,et al.  Factored conditional restricted Boltzmann Machines for modeling motion style , 2009, ICML '09.

[12]  Guan Gui,et al.  Deep Learning for an Effective Nonorthogonal Multiple Access Scheme , 2018, IEEE Transactions on Vehicular Technology.

[13]  Guan Gui,et al.  Deep Learning for Super-Resolution Channel Estimation and DOA Estimation Based Massive MIMO System , 2018, IEEE Transactions on Vehicular Technology.

[14]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[15]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[16]  Christian Igel,et al.  An Introduction to Restricted Boltzmann Machines , 2012, CIARP.

[17]  Ali Deihimi,et al.  Short-term electric load and temperature forecasting using wavelet echo state networks with neural reconstruction , 2013 .

[18]  Tao Li,et al.  Deep belief echo-state network and its application to time series prediction , 2017, Knowl. Based Syst..

[19]  Ren Ping Liu,et al.  ResInNet: A Novel Deep Neural Network With Feature Reuse for Internet of Things , 2019, IEEE Internet of Things Journal.

[20]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[21]  Haizhou Li,et al.  Real-Time Keypoint Recognition Using Restricted Boltzmann Machine , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[23]  Pascal Vincent,et al.  Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives , 2012, ArXiv.

[24]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[25]  Antonello Rizzi,et al.  Short-Term Electric Load Forecasting Using Echo State Networks and PCA Decomposition , 2015, IEEE Access.

[26]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Ren Ping Liu,et al.  Effect of hybrid circle reservoir injected with wavelet-neurons on performance of echo state network , 2014, Neural Networks.