A Novel Fractional Gradient-Based Learning Algorithm for Recurrent Neural Networks

In this research, we propose a novel algorithm for learning of the recurrent neural networks called as the fractional back-propagation through time (FBPTT). Considering the potential of the fractional calculus, we propose to use the fractional calculus-based gradient descent method to derive the FBPTT algorithm. The proposed FBPTT method is shown to outperform the conventional back-propagation through time algorithm on three major problems of estimation namely nonlinear system identification, pattern classification and Mackey–Glass chaotic time series prediction.

[1]  Thomas J. Osler,et al.  A Child's Garden of Fractional Derivatives , 2000 .

[2]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[3]  Gholamreza Haffari,et al.  A Latent Variable Recurrent Neural Network for Discourse Relation Language Models , 2016, ArXiv.

[4]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[5]  Torbjörn Wigren,et al.  Recursive prediction error identification and scaling of non-linear state space models using a restricted black box parameterization , 2006, Autom..

[6]  Ngoc Thang Vu,et al.  Bi-directional recurrent neural network with ranking loss for spoken language understanding , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[7]  B. T. Krishna,et al.  Active and Passive Realization of Fractance Device of Order 1/2 , 2008 .

[8]  J. Mesirov,et al.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. , 1999, Science.

[9]  G. Jumarie,et al.  Modified Riemann-Liouville derivative and fractional Taylor series of nondifferentiable functions further results , 2006, Comput. Math. Appl..

[10]  Jing Peng,et al.  An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories , 1990, Neural Computation.

[11]  Huijun Gao,et al.  Active Suspension Control With Frequency Band Constraints and Actuator Input Delay , 2012, IEEE Transactions on Industrial Electronics.

[12]  Michael Fairbank,et al.  An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Ahmad Jawwad,et al.  Design of Efficient Adaptive Beamforming Algorithms for Novel MIMO Architectures , 2014 .

[14]  Andrew W. Senior,et al.  Fast and accurate recurrent neural network acoustic models for speech recognition , 2015, INTERSPEECH.

[15]  Guy Jumarie,et al.  On the derivative chain-rules in fractional calculus via fractional difference and their application to systems modelling , 2013 .

[16]  Hermann Ney,et al.  Comparison of feedforward and recurrent neural network language models , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[17]  Yi-Fei Pu,et al.  A recursive two-circuits series analog fractance circuit for any order fractional calculus , 2006, International Commission for Optics.

[18]  Huijun Gao,et al.  Finite Frequency $H_{\infty }$ Control for Vehicle Active Suspension Systems , 2011, IEEE Transactions on Control Systems Technology.

[19]  Andreas Stolcke,et al.  Recurrent neural network and LSTM models for lexical utterance classification , 2015, INTERSPEECH.

[20]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Sungzoon Cho,et al.  Hand motion identification of grasp-and-lift task from electroencephalography recordings using recurrent neural networks , 2016, 2016 International Conference on Big Data and Smart Computing (BigComp).

[22]  Quoc V. Le.  A Tutorial on Deep Learning Part 2: Autoencoders, Convolutional Neural Networks and Recurrent Neural Networks , 2015 .

[23]  Heikki Huttunen,et al.  Recurrent neural networks for polyphonic sound event detection in real life recordings , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  Ijaz Mansoor Qureshi,et al.  Adaptive step-size modified fractional least mean square algorithm for chaotic time series prediction , 2014 .

[25]  Mark J. F. Gales,et al.  Recurrent neural network language model adaptation for multi-genre broadcast speech recognition , 2015, INTERSPEECH.

[26]  Sergey Levine,et al.  Learning deep neural network policies with continuous memory states , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Auke Jan Ijspeert,et al.  Fractional Multi-models of the Frog Gastrocnemius Muscle , 2008 .

[28]  Raja Muhammad Asif Zahoor,et al.  Novel generalization of Volterra LMS algorithm to fractional order with application to system identification , 2018, Neural Computing and Applications.

[29]  Zhigang Zeng,et al.  Global asymptotical stability analysis for a kind of discrete-time recurrent neural network with discontinuous activation functions , 2016, Neurocomputing.

[30]  Alan W. Black,et al.  Recurrent Neural Network Postfilters for Statistical Parametric Speech Synthesis , 2016, ArXiv.

[31]  Isis Bonet,et al.  Backpropagation through Time Algorithm for Training Recurrent Neural Networks using Variable Length Instances , 2013 .

[32]  R. Magin,et al.  Modeling the Cardiac Tissue Electrode Interface Using Fractional Calculus , 2008 .

[33]  R. Feynman,et al.  RECENT APPLICATIONS OF FRACTIONAL CALCULUS TO SCIENCE AND ENGINEERING , 2003 .

[34]  D. N. Tibarewala,et al.  A back-propagation through time based recurrent neural network approach for classification of cognitive EEG states , 2015, 2015 IEEE International Conference on Engineering and Technology (ICETECH).

[35]  Yasser Roudi,et al.  Learning with hidden variables , 2015, Current Opinion in Neurobiology.

[36]  Quoc V. Le,et al.  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition , 2015, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[37]  Huijun Gao,et al.  Transient-Performance-Guaranteed Robust Adaptive Control and Its Application to Precision Motion Control Systems , 2016, IEEE Transactions on Industrial Electronics.

[38]  Johan Schoukens,et al.  Three free data sets for development and benchmarking in nonlinear system identification , 2013, 2013 European Control Conference (ECC).

[39]  Marc'Aurelio Ranzato,et al.  Learning Longer Memory in Recurrent Neural Networks , 2014, ICLR.

[40]  Guy Jumarie,et al.  An approach via fractional analysis to non-linearity induced by coarse-graining in space , 2010 .

[41]  Koshy George,et al.  Improving Transient Response in Adaptive Control of Nonlinear Systems , 2016 .

[42]  Raja Muhammad Asif Zahoor,et al.  Two-stage fractional least mean square identification algorithm for parameter estimation of CARMA systems , 2015, Signal Process..

[43]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[44]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Geoffrey E. Hinton,et al.  A Simple Way to Initialize Recurrent Networks of Rectified Linear Units , 2015, ArXiv.

[46]  Peter,et al.  Prediction of Temperature Daily Profile by Stochastic Update of Backpropagation through Time Algorithm , 2012 .

[47]  Michiel Hermans,et al.  Optoelectronic Systems Trained With Backpropagation Through Time , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Sudipto Saha,et al.  Prediction of continuous B‐cell epitopes in an antigen using recurrent neural network , 2006, Proteins.

[49]  Thomas J. Osler,et al.  Fractional Derivatives and Special Functions , 1976 .

[50]  Rutuparna Panda,et al.  Fractional generalized splines and signal processing , 2006, Signal Process..

[51]  Kun Li,et al.  Voice conversion using deep Bidirectional Long Short-Term Memory based Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[52]  Ronald J. Williams,et al.  Gradient-based learning algorithms for recurrent networks and their computational complexity , 1995 .

[53]  G. Bohannan.  Analog Fractional Order Controller in Temperature and Motor Control Applications , 2008 .

[54]  José António Tenreiro Machado,et al.  Experimental Signal Analysis of Robot Impacts in a Fractional Calculus Perspective , 2007, J. Adv. Comput. Intell. Intell. Informatics.

[55]  Tomas Mikolov,et al.  Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets , 2015, NIPS.

[56]  Kenji Doya,et al.  Adaptive neural oscillator using continuous-time back-propagation learning , 1989, Neural Networks.

[57]  Alfonso Baños,et al.  Automatic Loop Shaping in QFT Using CRONE Structures , 2008 .

[58]  O. Agrawal,et al.  Advances in Fractional Calculus , 2007 .

[59]  Guangren Shi,et al.  A Novel Time-series Artificial Neural Network: A Case Study for Forecasting Oil Production , 2016 .

[60]  Biing-Hwang Juang,et al.  Recurrent deep neural networks for robust speech recognition , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[61]  Andreas Stolcke,et al.  A comparative study of recurrent neural network models for lexical domain classification , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[62]  Mohammed Bennamoun,et al.  A Novel Adaptive Kernel for the RBF Neural Networks , 2016, Circuits, Systems, and Signal Processing.

[63]  Eduardo Camponogara,et al.  System Identification of a Vertical Riser Model with Echo State Networks , 2015 .

[64]  Guy Jumarie,et al.  Table of some basic fractional calculus formulae derived from a modified Riemann-Liouville derivative for non-differentiable functions , 2009, Appl. Math. Lett..

[65]  Imran Naseem,et al.  RVP-FLMS: A robust variable power fractional LMS algorithm , 2016, 2016 6th IEEE International Conference on Control System, Computing and Engineering (ICCSCE).

[66]  Marc Weilbeer,et al.  Efficient Numerical Methods for Fractional Differential Equations and their Analytical Background , 2005 .

[67]  Zhe Gan,et al.  Deep Temporal Sigmoid Belief Networks for Sequence Modeling , 2015, NIPS.

[68]  Zhiqiang He,et al.  A Novel Generalization of Modified LMS Algorithm to Fractional Order , 2015, IEEE Signal Processing Letters.