Infinite-dimensional Folded-in-time Deep Neural Networks

The method recently introduced in [1] realizes a deep neural network with just a single nonlinear element and delayed feedback. It is applicable for the description of physically implemented neural networks. In this work, we present an infinite-dimensional generalization, which allows for a more rigorous mathematical analysis and a higher flexibility in choosing the weight functions. Precisely speaking, the weights are described by Lebesgue integrable functions instead of step functions. We also provide a functional backpropagation algorithm, which enables gradient descent training of the weights. In addition, with a slight modification, our concept realizes recurrent neural networks.

[1]  Germany,et al.  Deep Learning with a Single Neuron: Folding a Deep Neural Network in Time using Feedback-Modulated Delay Loops , 2020, ArXiv.

[2]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[3]  Laurent Larger,et al.  High-Speed Photonic Reservoir Computing Using a Time-Delay-Based Architecture: Million Words per Second Classification , 2017 .

[4]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[5]  M. C. Soriano,et al.  Consistency properties of chaotic systems driven by time-delayed feedback. , 2018, Physical review. E.

[6]  Laurent Larger,et al.  Optimal nonlinear information processing capacity in delay-based reservoir computers , 2014, Scientific Reports.

[7]  Satoshi Sunada,et al.  Compact reservoir computing with a photonic integrated circuit. , 2018, Optics express.

[8]  Giacomelli,et al.  Relationship between delayed and spatially extended dynamical systems. , 1996, Physical review letters.

[9]  Pierdomenico Pepe,et al.  Stability, Control, and Computation for Time-Delay Systems, An Eigenvalue-Based Approach, Michiels Wim, Niculescu Silviu-Iulian, in: Advances in Design and Control. SIAM (2014) , 2021, Autom..

[10]  Indranil Saha,et al.  journal homepage: www.elsevier.com/locate/neucom , 2022 .

[11]  Michael C. Mozer,et al.  A Focused Backpropagation Algorithm for Temporal Pattern Recognition , 1989, Complex Syst..

[12]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[13]  Salim Mejaouri,et al.  Reservoir computing with a single delay-coupled non-linear mechanical oscillator , 2018, Journal of Applied Physics.

[14]  Serge Massar,et al.  All-optical Reservoir Computing , 2012, Optics express.

[15]  M. Orio,et al.  Density functional theory , 2009, Photosynthesis Research.

[16]  Gordon Pipa,et al.  An Analytical Approach to Single Node Delay-Coupled Reservoir Computing , 2013, ICANN.

[17]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[18]  Marco Cococcioni,et al.  Photonic Neural Networks: A Survey , 2019, IEEE Access.

[19]  Laurent Larger,et al.  Delayed dynamical systems: networks, chimeras and reservoir computing , 2018, Philosophical Transactions of the Royal Society A.

[20]  André Röhm,et al.  Performance boost of time-delay reservoir computing by non-resonant clock cycle , 2019, Neural Networks.

[21]  Gordon Wetzstein,et al.  Inference in artificial intelligence with deep optics and photonics , 2020, Nature.

[22]  Giovanni Giacomelli,et al.  Pattern formation in systems with multiple delayed feedbacks. , 2014, Physical review letters.

[23]  Daniel Brunner,et al.  Parallel photonic information processing at gigabyte per second data rates using transient states , 2013, Nature Communications.

[24]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[25]  Gordon Pipa,et al.  Optimized Temporal Multiplexing for Reservoir Computing with a Single Delay-Coupled Node , 2014 .

[26]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[27]  Sergey A. Kaschenko,et al.  Normal and quasinormal forms for systems of difference and differential-difference equations , 2016, Commun. Nonlinear Sci. Numer. Simul..

[28]  William H. Guss.  Deep Function Machines: Generalized Neural Networks for Topological Layer Expression , 2016, ArXiv.

[29]  B. Averbeck,et al.  Reinforcement learning in artificial and biological systems , 2019, Nature Machine Intelligence.

[30]  Guo-Wei Wei,et al.  Protein structure prediction beyond AlphaFold , 2019, Nature Machine Intelligence.

[31]  Damien Querlioz,et al.  Neuromorphic computing with nanoscale spintronic oscillators , 2017, Nature.

[32]  R. Parr.  Density-functional theory of atoms and molecules , 1989 .

[33]  Laurent Larger,et al.  Tutorial: Photonic Neural Networks in Delay Systems , 2018, Journal of Applied Physics.

[34]  J. Andrew Bagnell,et al.  Kernel Conjugate Gradient for Fast Kernel Machines , 2007, IJCAI.

[35]  Germany,et al.  Emulating complex networks with a single delay differential equation , 2021 .

[36]  Giovanni Giacomelli,et al.  Dynamical systems with multiple long-delayed feedbacks: Multiscale analysis and spatiotemporal equivalence. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[37]  Felix Köster,et al.  Limitations of the recall capabilities in delay based reservoir computing systems , 2020, Cognitive Computation.

[38]  Alexander J. Smola,et al.  Online learning with kernels , 2001, IEEE Transactions on Signal Processing.

[39]  Miguel C. Soriano,et al.  Comparison of Photonic Reservoir Computing Systems for Fiber Transmission Equalization , 2020, IEEE Journal of Selected Topics in Quantum Electronics.

[40]  Giovanni Giacomelli,et al.  Spatio-temporal phenomena in complex systems with time delays , 2017, 2206.03120.

[41]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[42]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[43]  Andrew J. Davison,et al.  Learning One-Shot Imitation From Humans Without Humans , 2019, IEEE Robotics and Automation Letters.

[44]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[45]  Peter L. Bartlett,et al.  Boosting Algorithms as Gradient Descent , 1999, NIPS.

[46]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[47]  S. Brendle,et al.  Calculus of Variations , 1927, Nature.

[48]  L. Appeltant,et al.  Information processing using a single dynamical node as complex system , 2011, Nature communications.

[49]  J. Andrew Bagnell,et al.  Boosted Backpropagation Learning for Training Deep Modular Networks , 2010, ICML.

[50]  Miguel C. Soriano,et al.  A Unifying Analysis of Chaos Synchronization and Consistency in Delay-Coupled Semiconductor Lasers , 2019, IEEE Journal of Selected Topics in Quantum Electronics.

[51]  Peter L. Bartlett,et al.  Functional Gradient Techniques for Combining Hypotheses , 2000 .

[52]  Guy Lever,et al.  Human-level performance in 3D multiplayer games with population-based reinforcement learning , 2018, Science.

[53]  Catherine D. Schuman,et al.  A Survey of Neuromorphic Computing and Neural Networks in Hardware , 2017, ArXiv.

[54]  Joseph D. Hart,et al.  Experiments with arbitrary networks in time-multiplexed delay systems. , 2017, Chaos.

[55]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[56]  Paul J. Werbos,et al.  Applications of advances in nonlinear sensitivity analysis , 1982 .

[57]  Nicolas Le Roux,et al.  Continuous Neural Networks , 2007, AISTATS.

[58]  André Röhm,et al.  Multiplexed networks: reservoir computing with virtual and real nodes , 2018, Journal of Physics Communications.

[59]  Jan Danckaert,et al.  Real-time Audio Processing with a Cascade of Discrete-Time Delay Line-Based Reservoir Computers , 2017, Cognitive Computation.

[60]  M. C. Soriano,et al.  Advances in photonic reservoir computing , 2017 .

[61]  Jianhong Wu,et al.  Introduction to Functional Differential Equations , 2013 .

[62]  Guy Van der Sande,et al.  Task-Independent Computational Abilities of Semiconductor Lasers with Delayed Optical Feedback for Reservoir Computing , 2019, Photonics.

[63]  Ingo Fischer,et al.  Consistency properties of a chaotic semiconductor laser driven by optical feedback. , 2015, Physical review letters.

[64]  L Pesquera,et al.  Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. , 2012, Optics express.

[65]  S. Barnett,et al.  Philosophical Transactions of the Royal Society A : Mathematical , 2017 .

[66]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[67]  Miguel C. Soriano,et al.  Reservoir computing with a single time-delay autonomous Boolean node , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.