Hardware Design for Autonomous Bayesian Networks

Directed acyclic graphs or Bayesian networks that are popular in many AI-related sectors for probabilistic inference and causal reasoning can be mapped to probabilistic circuits built out of probabilistic bits (p-bits), analogous to binary stochastic neurons of stochastic artificial neural networks. In order to satisfy standard statistical results, individual p-bits not only need to be updated sequentially but also in order from the parent to the child nodes, necessitating the use of sequencers in software implementations. In this article, we first use SPICE simulations to show that an autonomous hardware Bayesian network can operate correctly without any clocks or sequencers, but only if the individual p-bits are appropriately designed. We then present a simple behavioral model of the autonomous hardware illustrating the essential characteristics needed for correct sequencer-free operation. This model is also benchmarked against SPICE simulations and can be used to simulate large-scale networks. Our results could be useful in the design of hardware accelerators that use energy-efficient building blocks suited for low-level implementations of Bayesian networks. The autonomous massively parallel operation of our proposed stochastic hardware has biological relevance since neural dynamics in brain is also stochastic and autonomous by nature.

[1]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[2]  H. Ohno,et al.  Spintronics based random access memory: a review , 2017 .

[3]  Kaushik Roy,et al.  Stochastic Spin-Orbit Torque Devices as Elements for Bayesian Inference , 2017, Scientific Reports.

[4]  Supriyo Bandyopadhyay,et al.  Low Energy Barrier Nanomagnet Design for Binary Stochastic Neurons: Design Challenges for Real Nanomagnets With Fabrication Defects , 2019, IEEE Magnetics Letters.

[5]  Ramtin Zand,et al.  A Novel Compound Synapse Using Probabilistic Spin–Orbit-Torque Switching for MTJ-Based Deep Neural Networks , 2019, IEEE Journal on Exploratory Solid-State Computational Devices and Circuits.

[6]  Jean-Philippe Diguet,et al.  FPGA implementation of Bayesian network inference for an embedded diagnosis , 2015, 2015 IEEE Conference on Prognostics and Health Management (PHM).

[7]  Eric Jonas,et al.  Stochastic architectures for probabilistic computation , 2014 .

[8]  Stefano Fusi,et al.  Building brain-inspired computing , 2019, Nature Communications.

[9]  Dong-Chul Park.  Image Classification Using Naïve Bayes Classifier , 2016 .

[10]  Joerg Appenzeller,et al.  Spin–Orbit Torque-Controlled Magnetic Tunnel Junction With Low Thermal Stability for Tunable Random Number Generation , 2019, IEEE Magnetics Letters.

[11]  Supriyo Datta,et al.  Probabilistic Circuits for Autonomous Learning: A Simulation Study , 2020, Frontiers in Computational Neuroscience.

[12]  E. Moro,et al.  Transition from ferromagnetism to superparamagnetism on the nanosecond time scale , 2002 .

[13]  Fabien Alibart,et al.  Pattern classification by memristive crossbar circuits using ex situ and in situ training , 2013, Nature Communications.

[14]  Krishna V. Palem,et al.  Probabilistic system-on-a-chip architectures , 2007, TODE.

[15]  R. Feynman.  Simulating physics with computers , 1999 .

[16]  Jian-Ping Wang,et al.  Telegraphic switching signals by magnet tunnel junctions for neural spiking signals with high information capacity , 2018, Journal of Applied Physics.

[17]  S. Majetich,et al.  Superparamagnetic perpendicular magnetic tunnel junctions for true random number generators , 2018 .

[18]  Brian M. Sutton,et al.  Stochastic p-bits for Invertible Logic , 2016, 1610.00377.

[19]  Hitoshi Kubota,et al.  Noise-enhanced synchronization of stochastic magnetic oscillators , 2014, 1405.4360.

[20]  Punyashloka Debashis.  Spintronic Devices as P-bits for Probabilistic Computing , 2020 .

[21]  Marek Kaminski,et al.  Real-time prediction of acute cardiovascular events using hardware-implemented Bayesian networks , 2016, Comput. Biol. Medicine.

[22]  Jacques Droulez,et al.  Bayesian Inference With Muller C-Elements , 2016, IEEE Transactions on Circuits and Systems I: Regular Papers.

[23]  Boulder,et al.  Large-angle, gigahertz-rate random telegraph switching induced by spin-momentum transfer , 2004, cond-mat/0404109.

[24]  Supriyo Datta,et al.  Implementing p-bits With Embedded MTJ , 2017, IEEE Electron Device Letters.

[25]  Supriyo Datta,et al.  Experimental demonstration of nanomagnet networks as hardware for Ising computing , 2016, 2016 IEEE International Electron Devices Meeting (IEDM).

[26]  Cristiano Premebida,et al.  Dynamic Bayesian network for semantic place classification in mobile robotics , 2017, Auton. Robots.

[27]  B. Diény,et al.  Realizing an Isotropically Coercive Magnetic Layer for Memristive Applications by Analogy to Dry Friction , 2020, Physical Review Applied.

[28]  William H. Hsu,et al.  A Survey of Algorithms for Real-Time Bayesian Network Inference , 2002 .

[29]  Deepanjan Datta,et al.  Modeling of spin transport in MTJ devices , 2012 .

[30]  Jonathan L. Ticknor.  A Bayesian regularized artificial neural network for stock market forecasting , 2013, Expert Syst. Appl..

[31]  Geoffrey E. Hinton.  Boltzmann machine , 2007, Scholarpedia.

[32]  Chetan Singh Thakur,et al.  Bayesian Estimation and Inference Using Stochastic Electronics , 2016, Front. Neurosci..

[33]  Min Zou,et al.  A new dynamic Bayesian network (DBN) approach for identifying gene regulatory networks from time course microarray data , 2005, Bioinform..

[34]  Supriyo Datta,et al.  R-DBN: A Resistive Deep Belief Network Architecture Leveraging the Intrinsic Behavior of Probabilistic Devices , 2017, ArXiv.

[35]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[36]  S. Datta,et al.  Low-Barrier Nanomagnets as p-Bits for Spin Logic , 2016, IEEE Magnetics Letters.

[37]  Pritish Narayanan,et al.  Equivalent-accuracy accelerated neural-network training using analogue memory , 2018, Nature.

[38]  Supriyo Datta,et al.  Hardware emulation of stochastic p-bits for invertible logic , 2017, Scientific Reports.

[39]  Concha Bielza,et al.  Bayesian networks in neuroscience: a survey , 2014, Front. Comput. Neurosci..

[40]  Supriyo Datta,et al.  A building block for hardware belief networks , 2016, Scientific Reports.

[41]  M. Gerstein,et al.  A Bayesian Networks Approach for Predicting Protein-Protein Interactions from Genomic Data , 2003, Science.

[42]  Damien Querlioz,et al.  Bioinspired Programming of Memory Devices for Implementing an Inference Engine , 2015, Proceedings of the IEEE.

[43]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[44]  Supriyo Datta,et al.  Autonomous Probabilistic Coprocessing With Petaflips per Second , 2019, IEEE Access.

[45]  Radford M. Neal.  Connectionist Learning of Belief Networks , 1992, Artif. Intell..

[46]  Adnan Darwiche,et al.  Modeling and Reasoning with Bayesian Networks , 2009 .

[47]  Joseph S. Friedman,et al.  Low-Energy Truly Random Number Generation with Superparamagnetic Tunnel Junctions for Unconventional Computing , 2017, 1706.05262.

[48]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[49]  Supriyo Datta,et al.  Intrinsic optimization using stochastic nanomagnets , 2016, Scientific Reports.

[50]  Supriyo Datta,et al.  Integer factorization using stochastic magnetic tunnel junctions , 2019, Nature.

[51]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[52]  Kang L. Wang,et al.  VOLTAGE-CONTROLLED MAGNETIC ANISOTROPY IN SPINTRONIC DEVICES , 2012 .

[53]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[54]  Concha Bielza,et al.  Comparison of Bayesian networks and artificial neural networks for quality detection in a machining process , 2009, Expert Syst. Appl..

[55]  D. Ralph,et al.  Spin-Torque Switching with the Giant Spin Hall Effect of Tantalum , 2012, Science.

[56]  Daniel Nikovski,et al.  Constructing Bayesian Networks for Medical Diagnosis from Incomplete and Partially Correct Statistics , 2000, IEEE Trans. Knowl. Data Eng..

[57]  Kerem Yunus Camsari,et al.  Weighted $p$ -Bits for FPGA Implementation of Probabilistic Circuits , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[58]  M. R. Mahmoodi,et al.  Versatile stochastic dot product circuits based on nonvolatile memories for high performance neurocomputing and neurooptimization , 2019, Nature Communications.

[59]  Supriyo Datta,et al.  From Charge to Spin and Spin to Charge: Stochastic Magnets for Probabilistic Switching , 2020, Proceedings of the IEEE.

[60]  José A. Gámez,et al.  Medical images modality classification using discrete Bayesian Networks , 2016 .

[61]  Supriyo Datta,et al.  Subnanosecond Fluctuations in Low-Barrier Nanomagnets , 2019 .

[62]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[63]  J. Appenzeller,et al.  Spin-torque devices with hard axis initialization as Stochastic Binary Neurons , 2018, Scientific Reports.

[64]  Supriyo Datta,et al.  Correlated fluctuations in spin orbit torque coupled perpendicular nanomagnets , 2019, Physical Review B.

[65]  Supriyo Datta,et al.  Low-Barrier Magnet Design for Efficient Hardware Binary Stochastic Neurons , 2019, IEEE Magnetics Letters.

[66]  Supriyo Datta,et al.  Scalable Emulation of Stoquastic Hamiltonians with Room Temperature p-bits , 2019 .

[67]  Goh Wang Ling,et al.  PCMOS-based Hardware Implementation of Bayesian Network , 2007, 2007 IEEE Conference on Electron Devices and Solid-State Circuits.

[68]  Shiliang Sun,et al.  A bayesian network approach to traffic flow forecasting , 2006, IEEE Transactions on Intelligent Transportation Systems.

[69]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[70]  G. Roberts,et al.  Updating Schemes, Correlation Structure, Blocking and Parameterization for the Gibbs Sampler , 1997 .

[71]  Michal Linial,et al.  Using Bayesian Networks to Analyze Expression Data , 2000, J. Comput. Biol..

[72]  J. W. Brown.  Thermal fluctuation of fine ferromagnetic particles , 1979 .

[73]  Chris H. Kim,et al.  Low-Energy Deep Belief Networks Using Intrinsic Sigmoidal Spintronic-based Probabilistic Neurons , 2017, ACM Great Lakes Symposium on VLSI.

[74]  Joseph S. Friedman,et al.  Circuit-Level Evaluation of the Generation of Truly Random Bits with Superparamagnetic Tunnel Junctions , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[75]  Sheng Ma,et al.  Adaptive diagnosis in distributed systems , 2005, IEEE Transactions on Neural Networks.

[76]  Supriyo Datta,et al.  Implementing Bayesian Networks with Embedded Stochastic MRAM , 2018, ArXiv.

[77]  P. Upadhyaya,et al.  Modular Compact Modeling of Magnetic Tunnel Junction Devices , 2018, 1805.00066.

[78]  David Heckerman,et al.  Causal independence for probability assessment and inference using Bayesian networks , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[79]  Hitoshi Kubota,et al.  Neural-like computing with populations of superparamagnetic basis functions , 2016, Nature Communications.

[80]  Supriyo Datta,et al.  Scaled Quantum Circuits Emulated with Room Temperature p-Bits , 2018, 1810.07144.