Exploiting Inherent Error Resiliency of Deep Neural Networks to Achieve Extreme Energy Efficiency Through Mixed-Signal Neurons

Neuromorphic computing, inspired by the brain, promises extreme efficiency for certain classes of learning tasks, such as classification and pattern recognition. The performance and power consumption of neuromorphic computing depend heavily on the choice of the neuron architecture. Digital neurons (Dig-N) are conventionally known to be accurate and efficient at high speed while suffering from high leakage currents from a large number of transistors in a large design. On the other hand, analog/mixed-signal neurons (MS-Ns) are prone to noise, variability, and mismatch but can lead to extremely low-power designs. In this paper, we will analyze, compare, and contrast existing neuron architectures with a proposed MS-N in terms of performance, power, and noise, thereby demonstrating the applicability of the proposed MS-N for achieving extreme energy efficiency (femtojoule/multiply and accumulate or less). The proposed MS-N is implemented in 65-nm CMOS technology and exhibits <inline-formula> <tex-math notation="LaTeX">$> 100\times $ </tex-math></inline-formula> better energy efficiency across all frequencies over two traditional Dig-Ns synthesized in the same technology node. We also demonstrate that the inherent error resiliency of a fully connected or even convolutional neural network can handle the noise as well as the manufacturing nonidealities of the MS-N up to certain degrees. Notably, a system-level implementation on CIFAR-10 data set exhibits a worst case increase in classification error by 2.1% when the integrated noise power in the bandwidth is <inline-formula> <tex-math notation="LaTeX">$\sim 0.1 ~\mu $ </tex-math></inline-formula> V2, along with <inline-formula> <tex-math notation="LaTeX">$\pm 3\sigma $ </tex-math></inline-formula> amount of variation and mismatch introduced in the transistor parameters for the proposed neuron with 8-bit precision.

[1]  Marian Verhelst,et al.  An Always-On 3.8 $\mu$ J/86% CIFAR-10 Mixed-Signal Binary CNN Processor With All Memory on Chip in 28-nm CMOS , 2019, IEEE Journal of Solid-State Circuits.

[2]  Jim D. Garside,et al.  SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation , 2013, IEEE Journal of Solid-State Circuits.

[3]  Kwabena Boahen,et al.  A Neuromorph's Prospectus , 2017, Computing in Science & Engineering.

[4]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[5]  Robert J. Wood,et al.  Spiking neural network (SNN) control of a flapping insect-scale robot , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[6]  Franziska Hoffmann,et al.  Design Of Analog Cmos Integrated Circuits , 2016 .

[7]  Arijit Raychowdhury,et al.  In-sensor analytics and energy-aware self-optimization in a wireless sensor node , 2017, 2017 IEEE MTT-S International Microwave Symposium (IMS).

[8]  Willy Sansen,et al.  analog design essentials , 2011 .

[9]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[10]  Shreyas Sen,et al.  Invited: Context-aware energy-efficient communication for IoT sensor nodes , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[11]  Rahul Sarpeshkar,et al.  Analog Versus Digital: Extrapolating from Electronics to Neurobiology , 1998, Neural Computation.

[12]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[13]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[14]  Debayan Das,et al.  RF-PUF: Enhancing IoT Security Through Authentication of Wireless Nodes Using In-Situ Machine Learning , 2018, IEEE Internet of Things Journal.

[15]  Kwabena Boahen,et al.  Design and validation of a real-time spiking-neural-network decoder for brain–machine interfaces , 2013, Journal of neural engineering.

[16]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[17]  K. Yelick,et al.  Intelligent RAM (IRAM): chips that remember and compute , 1997, 1997 IEEE International Solids-State Circuits Conference. Digest of Technical Papers.

[18]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[19]  Anand Raghunathan,et al.  Computing in Memory With Spin-Transfer Torque Magnetic RAM , 2017, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[20]  Kaushik Roy,et al.  An Energy-Efficient Mixed-Signal Neuron for Inherently Error-Resilient Neuromorphic Systems , 2017, 2017 IEEE International Conference on Rebooting Computing (ICRC).

[21]  Shreyas Sen,et al.  Self-Optimizing IoT Wireless Video Sensor Node With In-Situ Data Analytics and Context-Driven Energy-Aware Real-Time Adaptation , 2017, IEEE Transactions on Circuits and Systems I: Regular Papers.

[22]  Venkatesh Srinivasan,et al.  A 531 nW/MHz, 128/spl times/32 current-mode programmable analog vector-matrix multiplier with over two decades of linearity , 2004, Proceedings of the IEEE 2004 Custom Integrated Circuits Conference (IEEE Cat. No.04CH37571).

[23]  Wolfgang Nebel,et al.  Low power design in deep submicron electronics , 1997 .

[24]  Jongkil Park,et al.  A 65k-neuron 73-Mevents/s 22-pJ/event asynchronous micro-pipelined integrate-and-fire array transceiver , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[25]  Pinaki Mazumder,et al.  CMOS and Memristor-Based Neural Network Design for Position Detection , 2012, Proceedings of the IEEE.

[26]  Chintan Thakkar.  Design of Multi-Gb/s Multi-Coefficient Mixed-Signal Equalizers , 2012 .

[27]  Henry Markram,et al.  The human brain project. , 2012, Scientific American.

[28]  Abhijit Chatterjee,et al.  A Power-Scalable Channel-Adaptive Wireless Receiver Based on Built-In Orthogonally Tunable LNA , 2012, IEEE Transactions on Circuits and Systems I: Regular Papers.

[29]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[30]  C. Mead,et al.  White noise in MOS transistors and resistors , 1993, IEEE Circuits and Devices Magazine.

[31]  Abhijit Chatterjee,et al.  Process-Variation Tolerant Channel-Adaptive Virtually Zero-Margin Low-Power Wireless Receiver Systems , 2014, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[32]  Byungsub Kim,et al.  A 10-Gb/s Compact Low-Power Serial I/O With DFE-IIR Equalization in 65-nm CMOS , 2009, IEEE Journal of Solid-State Circuits.

[33]  Abhijit Chatterjee,et al.  Self-Learning RF Receiver Systems: Process Aware Real-Time Adaptation to Channel Conditions for Low Power Operation , 2017, IEEE Transactions on Circuits and Systems I: Regular Papers.