Effect of recurrent infomax on the information processing capability of input-driven recurrent neural networks

Reservoir computing is a framework for exploiting the inherent transient dynamics of recurrent neural networks (RNNs) as a computational resource. On the basis of this framework, much research has been conducted to evaluate the relationship between the dynamics of RNNs and the RNNs' information processing capability. In this study, we present a detailed analysis of the information processing capability of an RNN optimized by recurrent infomax (RI), an unsupervised learning method that maximizes the mutual information of RNNs by adjusting the connection weights of the network. The results indicate that RI leads to the emergence of a delay-line structure and that the network optimized by the RI possesses a superior short-term memory, which is the ability to store the temporal information of the input stream in its transient dynamics.

[1]  L Pesquera,et al.  Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. , 2012, Optics express.

[2]  Helmut Hauser,et al.  A soft body as a reservoir: case studies in a dynamic model of octopus-inspired soft robotic arm , 2013, Front. Comput. Neurosci..

[3]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[4]  Nils Bertschinger,et al.  Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks , 2004, Neural Computation.

[5]  José Carlos Príncipe,et al.  Liquid state machines and cultured cortical networks: The separation property , 2009, Biosyst..

[6]  Leon O. Chua,et al.  A Nonlinear Dynamics Perspective of Wolfram's New Kind of Science Part I: Threshold of Complexity , 2002, Int. J. Bifurc. Chaos.

[7]  W. Maass,et al.  State-dependent computations: spatiotemporal processing in cortical networks , 2009, Nature Reviews Neuroscience.

[8]  L. Appeltant,et al.  Information processing using a single dynamical node as complex system , 2011, Nature communications.

[9]  Audrius V. Avizienis,et al.  Emergent Criticality in Complex Turing B‐Type Atomic Switch Networks , 2012, Advanced materials.

[10]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[11]  Paul-Gerhard Plöger,et al.  Echo State Networks used for Motor Control , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[12]  Tao Li,et al.  Information processing via physical soft body , 2015, Scientific Reports.

[13]  Surya Ganguli,et al.  Memory traces in dynamical systems , 2008, Proceedings of the National Academy of Sciences.

[14]  John G. Harris,et al.  Automatic speech recognition using a predictive echo state network classifier , 2007, Neural Networks.

[15]  Gilles Laurent,et al.  Transient Dynamics for Neural Processing , 2008, Science.

[16]  Rik Van de Walle,et al.  Real-Time Reservoir Computing Network-Based Systems for Detection Tasks on Visual Contents , 2015, 2015 7th International Conference on Computational Intelligence, Communication Systems and Networks.

[17]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[18]  Chrisantha Fernando,et al.  Pattern Recognition in a Bucket , 2003, ECAL.

[19]  Herbert Jaeger,et al.  Echo State Property Linked to an Input: Exploring a Fundamental Characteristic of Recurrent Neural Networks , 2013, Neural Computation.

[20]  Keisuke Fujii,et al.  Harnessing disordered quantum dynamics for machine learning , 2016, 1602.08159.

[21]  Dean V. Buonomano,et al.  ROBUST TIMING AND MOTOR PATTERNS BY TAMING CHAOS IN RECURRENT NEURAL NETWORKS , 2012, Nature Neuroscience.

[22]  Benjamin Schrauwen,et al.  Event detection and localization for small mobile robots using reservoir computing , 2008, Neural Networks.

[23]  Han Ju,et al.  Spatiotemporal Memory Is an Intrinsic Property of Networks of Dissociated Cortical Neurons , 2015, The Journal of Neuroscience.

[24]  D. Buonomano,et al.  Neural dynamics of in vitro cortical networks reflects experienced temporal patterns , 2010, Nature Neuroscience.

[25]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[26]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[27]  Dean V. Buonomano,et al.  Temporal Interval Learning in Cortical Cultures Is Encoded in Intrinsic Network Dynamics , 2016, Neuron.

[28]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[29]  Benjamin Schrauwen,et al.  Information Processing Capacity of Dynamical Systems , 2012, Scientific Reports.

[30]  Takeshi Kaneko,et al.  Recurrent Infomax Generates Cell Assemblies, Neuronal Avalanches, and Simple Cell-Like Selectivity , 2009, Neural Computation.

[31]  L. F. Abbott,et al.  Generating Coherent Patterns of Activity from Chaotic Neural Networks , 2009, Neuron.

[32]  Thomas J. Naughton,et al.  Photonic neural networks , 2012, Nature Physics.

[33]  L. Abbott,et al.  Beyond the edge of chaos: amplification and temporal integration by recurrent networks in the chaotic regime. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[34]  Damien Querlioz,et al.  Neuromorphic computing with nanoscale spintronic oscillators , 2017, Nature.

[35]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[36]  Tao Li,et al.  Exploiting the Dynamics of Soft Materials for Machine Learning , 2018, Soft robotics.

[37]  Hitoshi Kubota,et al.  Evaluation of memory capacity of spin torque oscillator for recurrent neural networks , 2018, Japanese Journal of Applied Physics.

[38]  Helmut Hauser,et al.  Exploiting short-term memory in soft body dynamics as a computational resource , 2014, Journal of The Royal Society Interface.

[39]  Antonius M J VanDongen,et al.  Short-Term Memory in Networks of Dissociated Cortical Neurons , 2013, The Journal of Neuroscience.

[40]  Hitoshi Kubota,et al.  Macromagnetic Simulation for Reservoir Computing Utilizing Spin Dynamics in Magnetic Tunnel Junctions , 2018, Physical Review Applied.