Deep Learning with a Single Neuron: Folding a Deep Neural Network in Time using Feedback-Modulated Delay Loops

Deep neural networks are among the most widely applied machine learning tools showing outstanding performance in a broad range of tasks. We present a method for folding a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops. This single-neuron deep neural network comprises only a single nonlinearity and appropriately adjusted modulations of the feedback signals. The network states emerge in time as a temporal unfolding of the neuron's dynamics. By adjusting the feedback-modulation within the loops, we adapt the network's connection weights. These connection weights are determined via a modified back-propagation algorithm that we designed for such types of networks. Our approach fully recovers standard Deep Neural Networks (DNN), encompasses sparse DNNs, and extends the DNN concept toward dynamical systems implementations. The new method, which we call Folded-in-time DNN (Fit-DNN), exhibits promising performance in a set of benchmark tasks.

[1]  Dimitri Breda,et al.  Pseudospectral Discretization of Nonlinear Delay Equations: New Prospects for Numerical Bifurcation Analysis , 2016, SIAM J. Appl. Dyn. Syst..

[2]  Laurent Larger,et al.  High-Speed Photonic Reservoir Computing Using a Time-Delay-Based Architecture: Million Words per Second Classification , 2017 .

[3]  Indranil Saha,et al.  journal homepage: www.elsevier.com/locate/neucom , 2022 .

[4]  André Röhm,et al.  Performance boost of time-delay reservoir computing by non-resonant clock cycle , 2019, Neural Networks.

[5]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[6]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[7]  Serge Massar,et al.  High performance photonic reservoir computer based on a coherently driven passive cavity , 2015, ArXiv.

[8]  Jianhong Wu,et al.  Introduction to Neural Dynamics and Signal Transmission Delay , 2001 .

[9]  Damien Querlioz,et al.  Neuromorphic computing with nanoscale spintronic oscillators , 2017, Nature.

[10]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[11]  Joseph D. Hart,et al.  Experiments with arbitrary networks in time-multiplexed delay systems. , 2017, Chaos.

[12]  Catherine D. Schuman,et al.  A Survey of Neuromorphic Computing and Neural Networks in Hardware , 2017, ArXiv.

[13]  Daniel Brunner,et al.  Parallel photonic information processing at gigabyte per second data rates using transient states , 2013, Nature Communications.

[14]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[15]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[16]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[17]  Toshiyuki Yamane,et al.  Recent Advances in Physical Reservoir Computing: A Review , 2018, Neural Networks.

[18]  B. Averbeck,et al.  Reinforcement learning in artificial and biological systems , 2019, Nature Machine Intelligence.

[19]  Laurent Larger,et al.  Delayed dynamical systems: networks, chimeras and reservoir computing , 2018, Philosophical Transactions of the Royal Society A.

[20]  L. Appeltant,et al.  Information processing using a single dynamical node as complex system , 2011, Nature communications.

[21]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[22]  Peter Stone,et al.  Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science , 2017, Nature Communications.

[23]  Gordon Pipa,et al.  An Analytical Approach to Single Node Delay-Coupled Reservoir Computing , 2013, ICANN.

[24]  Ingo Fischer,et al.  PAM-4 Transmission at 1550 nm Using Photonic Reservoir Computing Post-Processing , 2019, IEEE Access.

[25]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[26]  Miguel C. Soriano,et al.  Reservoir computing with a single time-delay autonomous Boolean node , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[27]  Guo-Wei Wei,et al.  Protein structure prediction beyond AlphaFold , 2019, Nature Machine Intelligence.

[28]  Giovanni Giacomelli,et al.  Spatio-temporal phenomena in complex systems with time delays , 2017, 2206.03120.

[29]  Salim Mejaouri,et al.  Reservoir computing with a single delay-coupled non-linear mechanical oscillator , 2018, Journal of Applied Physics.

[30]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[31]  Giovanni Samaey,et al.  DDE-BIFTOOL Manual - Bifurcation analysis of delay differential equations , 2014, 1406.7144.

[32]  Paul J. Werbos,et al.  Applications of advances in nonlinear sensitivity analysis , 1982 .

[33]  Thomas Erneux,et al.  Introduction to Focus Issue: Time-delay dynamics. , 2017, Chaos.

[34]  Daniel Brunner,et al.  Conditions for reservoir computing performance using semiconductor lasers with delayed optical feedback. , 2017, Optics express.

[35]  Serge Massar,et al.  All-optical Reservoir Computing , 2012, Optics express.

[36]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[37]  M. C. Soriano,et al.  Advances in photonic reservoir computing , 2017 .

[38]  Marco Cococcioni,et al.  Photonic Neural Networks: A Survey , 2019, IEEE Access.

[39]  Bin Dong,et al.  Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations , 2017, ICML.

[40]  L Pesquera,et al.  Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. , 2012, Optics express.

[41]  Giacomelli,et al.  Relationship between delayed and spatially extended dynamical systems. , 1996, Physical review letters.

[42]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[43]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[44]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[45]  Guy Lever,et al.  Human-level performance in 3D multiplayer games with population-based reinforcement learning , 2018, Science.

[46]  Andrew J. Davison,et al.  Learning One-Shot Imitation From Humans Without Humans , 2019, IEEE Robotics and Automation Letters.

[47]  Rose,et al.  Conjecture on the dimensions of chaotic attractors of delayed-feedback dynamical systems. , 1987, Physical review. A, General physics.

[48]  J..  CHAOTIC ATTRACTORS OF AN INFINITE-DIMENSIONAL DYNAMICAL SYSTEM , 2002 .

[49]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .