Resonant Machine Learning Based on Complex Growth Transform Dynamical Systems

Traditional energy-based learning models associate a single energy metric to each configuration of variables involved in the underlying optimization process. Such models associate the lowest energy state with the optimal configuration of variables under consideration and are thus inherently dissipative. In this article, we propose an energy-efficient learning framework that exploits structural and functional similarities between a machine-learning network and a general electrical network satisfying Tellegen's theorem. In contrast to the standard energy-based models, the proposed formulation associates two energy components, namely, active and reactive energy with the network. The formulation ensures that the network's active power is dissipated only during the process of learning, whereas the reactive power is maintained to be zero at all times. As a result, in steady state, the learned parameters are stored and self-sustained by electrical resonance determined by the network's nodal inductances and capacitances. Based on this approach, this article introduces three novel concepts: 1) a learning framework where the network's active-power dissipation is used as a regularization for a learning objective function that is subjected to zero total reactive-power constraint; 2) a dynamical system based on complex-domain, continuous-time growth transforms that optimizes the learning objective function and drives the network toward electrical resonance under steady-state operation; and 3) an annealing procedure that controls the tradeoff between active-power dissipation and the speed of convergence. As a representative example, we show how the proposed framework can be used for designing resonant support vector machines (SVMs), where the support vectors correspond to an LC network with self-sustained oscillations. We also show that this resonant network dissipates less active power compared with its non-resonant counterpart.

[1]  Sandeep Subramanian,et al.  Deep Complex Networks , 2017, ICLR.

[2]  Pierre Baldi,et al.  Computing with Arrays of Coupled Oscillators: An Application to Preattentive Texture Discrimination , 1990, Neural Computation.

[3]  Akira Hirose,et al.  Complex-Valued Neural Networks: Theories and Applications , 2003 .

[4]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[5]  Tohru Nitta,et al.  Orthogonality of Decision Boundaries in Complex-Valued Neural Networks , 2004, Neural Computation.

[6]  Robert P. W. Duin,et al.  Support vector domain description , 1999, Pattern Recognit. Lett..

[7]  Geoffrey E. Hinton,et al.  Restricted Boltzmann machines for collaborative filtering , 2007, ICML '07.

[8]  Marc'Aurelio Ranzato,et al.  A Unified Energy-Based Framework for Unsupervised Learning , 2007, AISTATS.

[9]  Yu Cheng,et al.  Deep Structured Energy Based Models for Anomaly Detection , 2016, ICML.

[10]  Mark Tygert,et al.  A theoretical argument for complex-valued convolutional networks , 2015, ArXiv.

[11]  Shantanu Chakrabartty,et al.  Decentralized Global Optimization Based on a Growth Transform Dynamical System Model , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Damien Querlioz,et al.  Neuromorphic computing with nanoscale spintronic oscillators , 2017, Nature.

[13]  Mark Tygert,et al.  A Mathematical Motivation for Complex-Valued Convolutional Networks , 2015, Neural Computation.

[14]  George Tucker,et al.  Energy-Inspired Models: Learning with Sampler-Induced Distributions , 2019, NeurIPS.

[15]  Wolfgang Just,et al.  Reactive Power Compensation: A Practical Guide , 2012 .

[16]  L. Baum,et al.  Growth transformations for functions on manifolds. , 1968 .

[17]  Sergios Theodoridis,et al.  Complex Support Vector Machines for Regression and Quaternary Classification , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Bernhard Schölkopf,et al.  Support Vector Method for Novelty Detection , 1999, NIPS.

[19]  A. Nadas,et al.  A generalization of the Baum algorithm to rational objective functions , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[20]  Sergey Levine,et al.  Reinforcement Learning with Deep Energy-Based Policies , 2017, ICML.

[21]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[22]  Hualiang Li,et al.  Algorithms for Complex ML ICA and Their Stability Analysis Using Wirtinger Calculus , 2010, IEEE Transactions on Signal Processing.

[23]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[24]  Junichi Yamagishi,et al.  Complex-Valued Restricted Boltzmann Machine for Direct Learning of Frequency Spectra , 2017, INTERSPEECH.

[25]  Kazuyuki Murase,et al.  Wirtinger Calculus Based Gradient Descent and Levenberg-Marquardt Learning Algorithms in Complex-Valued Neural Networks , 2011, ICONIP.

[26]  Thierry Paul,et al.  Quantum computation and quantum information , 2007, Mathematical Structures in Computer Science.

[27]  Joseph A. Edminister Schaum's outline of theory and problems of electric circuits , 1993 .

[28]  Gert Cauwenberghs,et al.  Gini Support Vector Machine: Quadratic Entropy Based Robust Multi-Class Probability Regression , 2007, J. Mach. Learn. Res..

[29]  S. Strogatz Exploring complex networks , 2001, Nature.

[30]  J. Dixon,et al.  Reactive Power Compensation Technologies: State-of-the-Art Review , 2005, Proceedings of the IEEE.

[31]  Anthony S. Maida,et al.  Deep Quaternion Networks , 2017, 2018 International Joint Conference on Neural Networks (IJCNN).

[32]  N. Mukunda,et al.  The Character of Physical Law , 2018 .

[33]  Shantanu Chakrabartty,et al.  Continuous-time Optimization using Sub-threshold Current-mode Growth Transform Circuits , 2018, 2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS).

[34]  D. Ruppert The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[35]  R. Karl Rethemeyer,et al.  Network analysis , 2011 .

[36]  Yoshua Bengio,et al.  Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation , 2016, Front. Comput. Neurosci..

[37]  S. Strogatz From Kuramoto to Crawford: exploring the onset of synchronization in populations of coupled oscillators , 2000 .

[38]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[39]  Ahana Gangopadhyay,et al.  Extended Polynomial Growth Transforms for Design and Training of Generalized Support Vector Machines , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[40]  Fu Jie Huang,et al.  A Tutorial on Energy-Based Learning , 2006 .

[41]  W. Wirtinger Zur formalen Theorie der Funktionen von mehr komplexen Veränderlichen , 1927 .

[42]  Wolfgang Porod,et al.  Computing With Networks of Oscillatory Dynamical Systems , 2019, Proceedings of the IEEE.

[43]  Nitzan Guberman,et al.  On Complex Valued Convolutional Neural Networks , 2016, ArXiv.

[44]  Jiquan Ngiam,et al.  Learning Deep Energy Models , 2011, ICML.