Noise Benefits in Feedback Machine Learning: Bidirectional Backpropagation

The new bidirectional backpropagation algorithm converts an ordinary feedforward neural network into a simple feedback dynamical system. The algorithm minimizes a joint performance measure so that training in one direction does not overwrite training in the reverse direction. This involves little extra computation. The forward direction gives the usual classification or regression network. The new backward pass approximates the centroids of the input pattern classes in a neural classifier. The bidirectional algorithm can also approximate inverse point mappings in the rare cases where such mappings exist. Carefully injected noise can speed the convergence of the bidirectional backpropagation. This holds because backpropagation is a special case of the expectation-maximization algorithm for maximum likelihood and because such noise can always boost its convergence. The noise also tends to improve accuracy in classification and regression.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Osonde Osoba,et al.  Noisy hidden Markov models for speech recognition , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[3]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[4]  Bart Kosko,et al.  Neural networks and fuzzy systems: a dynamical systems approach to machine intelligence , 1991 .

[5]  Osonde Osoba,et al.  Noise can speed backpropagation learning and deep bidirectional pretraining , 2020, Neural Networks.

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  Bart Kosko,et al.  Using noise to speed up video classification with recurrent backpropagation , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[8]  Osonde A. Osoba,et al.  The Noisy Expectation-Maximization Algorithm for Multiplicative Noise Injection , 2016 .

[9]  Osonde A. Osoba,et al.  The Noisy Expectation Maximization Algorithm , 2013 .

[10]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[11]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[12]  Bart Kosko,et al.  Bidirectional Backpropagation , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[13]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[14]  Yoshua Bengio,et al.  Gated Feedback Recurrent Neural Networks , 2015, ICML.

[15]  Bart Kosko,et al.  Using Noise to Speed up Markov Chain Monte Carlo Estimation , 2015, INNS Conference on Big Data.

[16]  Osonde Osoba,et al.  Noise-enhanced convolutional neural networks , 2016, Neural Networks.

[17]  Michael I. Jordan,et al.  Machine learning: Trends, perspectives, and prospects , 2015, Science.