Photonic tensor core machine learning accelerator

Photonic Tensor Cores (PTCs) have been raised as one of the major candidates to accelerate Neural Network hardware, taking advantage of the high bandwidth, low latency, and energy efficiency that propagating light has over the electrical counterpart. However, actual solutions still rely on external bulk components, such as lasers, modulators, and photodetectors. In this work, we show the first fully integrated hybrid Silicon Photonics PTC for computing Matrix-Vector Multiplication (MVM), one of the main steps for any Neural Network layer. The PTC is formed by a WDM 3-fold InP laser array connected to an active Silicon Photonic PIC through Photonic Wire Bondings. The CW lasers are modulated by high-speed Mach-Zehnder modulators to generate the input vector. The mix of all the signals is sent into a 3x3 matrix, formed by high-speed add-drop coupled microring resonators, whose tuning signals are the weights of the matrix. Outputs are collected by a bank of high-speed integrated photodetectors. The whole photonics IC has a footprint of 4.1×1.7 mm2, including lasers, allowing to have just electrical I/O. The full integration of input modulators, weights, and photodetectors can allow the PIC to work at over 20 GHz bandwidth with extremely low latency. This integration is a key step toward the actual deployment of photonics as an NN-accelerator for future AI systems.

[1]  L. Chrostowski,et al.  Towards a high-density photonic tensor core enabled by intensity-modulated microrings and photonic wire bonding , 2023, Scientific Reports.

[2]  V. Sorger,et al.  Bistable All‐Optical Devices Based on Nonlinear Epsilon‐Near‐Zero (ENZ) Materials , 2022, Laser & Photonics Reviews.

[3]  S. Rafatirad,et al.  Towards Generalized ML Model in Automated Physiological Arousal Computing: A Transfer Learning-Based Domain Generalization Approach , 2022, 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[4]  S. Semancik,et al.  Deep Learning Image Analysis of Nanoplasmonic Sensors: Toward Medical Breath Monitoring. , 2022, ACS applied materials & interfaces.

[5]  V. Sorger,et al.  Reducing Latency in Sensing for Optical Convolutional Neural Network , 2022, 2022 IEEE Photonics Conference (IPC).

[6]  A. Hanbicki,et al.  Extraordinary tunnel electroresistance in layer-by-layer engineered van der Waals ferroelectric tunnel junctions , 2022, Matter.

[7]  V. Sorger,et al.  High-density Integrated Photonic Tensor Processing Unit with a Matrix Multiply Compile , 2022, 2022 IEEE Photonics Conference (IPC).

[8]  Bicky A. Marquez,et al.  Multi-Level Encoding and Decoding in a Scalable Photonic Tensor Processor With a Photonic General Matrix Multiply (GeMM) Compiler , 2022, IEEE Journal of Selected Topics in Quantum Electronics.

[9]  V. Sorger,et al.  High-performance opto-electronics with emerging materials , 2022, NanoScience + Engineering.

[10]  V. Sorger,et al.  Photonic tensor core for machine learning: a review , 2022, NanoScience + Engineering.

[11]  Mario Miscuglio,et al.  Batch processing and data streaming Fourier-based convolutional neural network accelerator , 2022, NanoScience + Engineering.

[12]  V. Sorger,et al.  Reconfigurable application-specific photonic integrated circuit for solving partial differential equations , 2022, Nanophotonics.

[13]  V. Sorger,et al.  A Deep Neural Network Accelerator using Residue Arithmetic in a Hybrid Optoelectronic System , 2022, ACM J. Emerg. Technol. Comput. Syst..

[14]  V. Sorger,et al.  Self-powered broadband photodetector based on MoS2/Sb2Te3 heterojunctions: a promising approach for highly sensitive detection , 2022, Nanophotonics.

[15]  C. Chuah,et al.  Convolution Neural Network for Pain Intensity Assessment from Facial Expression , 2022, 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[16]  Joel K. W. Yang,et al.  Programmable chalcogenide-based all-optical deep neural networks , 2022, Nanophotonics.

[17]  V. Sorger,et al.  Waveguide Integrated MoTe2 Photodetector with High speed and energy efficient , 2022, 2022 Conference on Lasers and Electro-Optics (CLEO).

[18]  V. Sorger,et al.  4-bit Ultra Low-loss Cascaded Nonvolatile Photonic Memories Using Broadband Transparent Phase change materials , 2022, 2022 Conference on Lasers and Electro-Optics (CLEO).

[19]  C. Wright,et al.  Phase-change materials for energy-efficient photonic memory and computing , 2022, MRS Bulletin.

[20]  V. Sorger,et al.  Electrical programmable multilevel nonvolatile photonic random-access memory , 2022, Light, science & applications.

[21]  V. Sorger Photonic TPU ASIC for AI acceleration , 2022, AI and Optical Data Sciences III.

[22]  A. Babakhani,et al.  PIC-based joint transform correlator CNN , 2022, AI and Optical Data Sciences III.

[23]  V. Sorger,et al.  20Gbps high-gain BW-product TMD slot-detector on PIC , 2022, 2D Photonic Materials and Devices V.

[24]  V. Sorger,et al.  Photonic Tensor Core with Photonic Compute-in-Memory , 2022, 2022 Optical Fiber Communications Conference and Exhibition (OFC).

[25]  C. Wright,et al.  Broadband photonic tensor core with integrated ultra-low crosstalk wavelength multiplexers , 2022, Nanophotonics.

[26]  S. Cheong,et al.  Strong laser polarization control of coherent phonon excitation in van der Waals material Fe3GeTe2 , 2022, npj 2D Materials and Applications.

[27]  Volker J. Sorger,et al.  Emerging Devices and Packaging Strategies for Electronic-Photonic AI Accelerators , 2021, ArXiv.

[28]  V. Sorger,et al.  100 GHz micrometer-compact broadband monolithic ITO Mach–Zehnder interferometer modulator enabling 3500 times higher packing density , 2021, 2112.10926.

[29]  Ray T. Chen,et al.  ADEPT: automatic differentiable DEsign of photonic tensor cores , 2021, DAC.

[30]  V. Sorger,et al.  Self-driven highly responsive PN junction InSe heterostructure near-infrared light detector , 2021, 2D Photonic Materials and Devices V.

[31]  V. Sorger,et al.  Training on System for Opto-Electrical Neural Network , 2022, Imaging and Applied Optics Congress 2022 (3D, AOA, COSI, ISA, pcAOP).

[32]  V. Sorger,et al.  Design and Testing of Integrated Photonic Chip for Convolution Neural Network , 2022, Imaging and Applied Optics Congress 2022 (3D, AOA, COSI, ISA, pcAOP).

[33]  V. Sorger,et al.  Electrical Programmable Low-loss high cyclable Nonvolatile Photonic Random-Access Memory , 2022 .

[34]  V. Sorger,et al.  Electrical Pulse Driven Multi-Level Nonvolatile Photonic Memories Using Broadband Transparent Phase Change Materials , 2022 .

[35]  V. Sorger,et al.  An ITO - Graphene heterojunction integrated absorption modulator on Si-photonics for neuromorphic nonlinear activation , 2021, APL Photonics.

[36]  F. Yaman,et al.  A silicon photonic–electronic neural network for fibre nonlinearity compensation , 2021, Nature Electronics.

[37]  V. Sorger,et al.  Compact Plasmonic Enhanced MoTe2 Photodetector based on Engineering Gain-Bandwidth-Product Scaling Laws , 2021, 2021 26th Microoptics Conference (MOC).

[38]  V. Sorger,et al.  1fJ/bit Coupling-based ITO Monolithic Modulator in Integrated Photonics , 2021, 2021 26th Microoptics Conference (MOC).

[39]  V. Sorger,et al.  Highly Accurate, Reliable and Non-Contaminating Two-Dimensional Material Transfer System , 2021, 2021 26th Microoptics Conference (MOC).

[40]  V. Reddi,et al.  An Electro-Photonic System for Accelerating Deep Neural Networks , 2021, ACM J. Emerg. Technol. Comput. Syst..

[41]  V. Sorger,et al.  Strain-Induced Spatially Resolved Charge Transport in 2H-MoTe2 , 2021, ACS Applied Electronic Materials.

[42]  T. Itoh,et al.  Approximate analog computing with metatronic circuits , 2021, Communications Physics.

[43]  R. Wallace,et al.  Understanding and optimization of graphene gas sensors , 2021, Applied Physics Letters.

[44]  Yangyang Zhao,et al.  Simulation of a Nano Plasmonic Pillar-Based Optical Sensor with AI-Assisted Signal Processing , 2021, ECS Meeting Abstracts.

[45]  V. Sorger,et al.  Effect of Strain in WS2 Monolayer Integrated Excitonic Photodetector , 2021, 2021 Conference on Lasers and Electro-Optics (CLEO).

[46]  V. Sorger,et al.  Performance Analysis of Integrated Electro-Optic Phase Modulators Based on Emerging Materials , 2021, IEEE Journal of Selected Topics in Quantum Electronics.

[47]  Ray T. Chen,et al.  Towards lab-on-chip ultrasensitive ethanol detection using photonic crystal waveguide operating in the mid-infrared , 2021, Nanophotonics.

[48]  V. Sorger,et al.  Symmetry perception with spiking neural networks , 2021, Scientific Reports.

[49]  Christopher J. Anders,et al.  Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications , 2021, Proceedings of the IEEE.

[50]  Ray T. Chen,et al.  Heterogeneously integrated ITO plasmonic Mach–Zehnder interferometric modulator on SOI , 2020, Scientific reports.

[51]  Xuan Li,et al.  Parallel convolutional processing using an integrated photonic tensor core , 2021, Nature.

[52]  A. Babakhani,et al.  Electronic Bottleneck Suppression in Next‐Generation Networks with Integrated Photonic Digital‐to‐Analog Converters , 2019, Advanced Photonics Research.

[53]  V. Sorger,et al.  Complex Exponential Neural Network for Optical System , 2021, Photonics in Switching and Computing 2021.

[54]  Bhavin J. Shastri,et al.  Photonics for artificial intelligence and neuromorphic computing , 2020, Nature Photonics.

[55]  V. Sorger,et al.  GHz-Fast Multiplexed OAM Generation and Detection , 2020 .

[56]  Mario Miscuglio,et al.  Electro-Optical Hybrid Fourier Neural Network with Amplitude-Only Modulation , 2020 .

[57]  V. Sorger,et al.  Efficient MoTe2 Slot-enhanced Photodetector based on Engineering Gain-Bandwidth-Product Scaling Laws , 2020 .

[58]  V. Sorger,et al.  Massively parallel amplitude-only Fourier neural network , 2020, AI and Optical Data Sciences II.

[59]  Ray T. Chen,et al.  Broadband Sub-λ GHz ITO Plasmonic Mach-Zehnder Modulator in Silicon Photonics , 2020 .

[60]  Tarek El-Ghazawi,et al.  Analog Computing with Metatronic Circuits , 2020, ArXiv.

[61]  Tarek El-Ghazawi,et al.  A Design Methodology for Post-Moore’s Law Accelerators: The Case of a Photonic Neuromorphic Processor , 2020, 2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP).

[62]  V. Sorger,et al.  2D Material Based Electro-Absorption Modulator in Si Photonics , 2020, 2020 Conference on Lasers and Electro-Optics (CLEO).

[63]  V. Sorger,et al.  WS2 Monolayer Integrated Photodetector , 2020, 2020 Conference on Lasers and Electro-Optics (CLEO).

[64]  Mario Miscuglio,et al.  GSST-Based Photonic Memory Multilevel Perceptron , 2020, 2020 Conference on Lasers and Electro-Optics (CLEO).

[65]  Puneet Gupta,et al.  Million-Channel Parallelism Fourier-Optic Convolutional Filter and Neural Network Processor , 2020, 2020 Conference on Lasers and Electro-Optics (CLEO).

[66]  Mario Miscuglio,et al.  Sub-wavelength GHz-fast broadband ITO Mach–Zehnder modulator on silicon photonics , 2020 .

[67]  Mario Miscuglio,et al.  Photonic tensor cores for machine learning , 2020, Applied Physics Reviews.

[68]  Paul R. Prucnal,et al.  Photonic Multiply-Accumulate Operations for Neural Networks , 2020, IEEE Journal of Selected Topics in Quantum Electronics.

[69]  S. Solares,et al.  Strain-engineered high-responsivity MoTe2 photodetector for silicon photonic integrated circuits , 2019, Nature Photonics.

[70]  Mario Miscuglio,et al.  A Lateral MOS-Capacitor-Enabled ITO Mach–Zehnder Modulator for Beam Steering , 2019, Journal of Lightwave Technology.

[71]  Yifei Zhang,et al.  Intelligent edge processing with photonic multilevel memory , 2020 .

[72]  Tarek El-Ghazawi,et al.  Integrated Photonics Architectures for Residue Number System Computations , 2019, 2019 IEEE International Conference on Rebooting Computing (ICRC).

[73]  V. Sorger,et al.  Ultrasensitive Phototransistor Based on Multi-layered MoTe2 , 2019, Frontiers in Optics + Laser Science APS/DLS.

[74]  Tarek El-Ghazawi,et al.  Silicon Photonic Enabled Residue Number System Adder and Multiplier , 2019, 2019 IEEE Research and Applications of Photonics in Defense Conference (RAPID).

[75]  M. Zaghloul,et al.  Miniaturized nanohole array based plasmonic sensor for the detection of acetone and ethanol with insights into the kinetics of adsorptive plasmonic sensing. , 2019, Nanoscale.

[76]  Tarek El-Ghazawi,et al.  ITO-based electro-absorption modulator for photonic neural activation function , 2019, APL Materials.

[77]  Volker J. Sorger,et al.  Roadmap on material-function mapping for photonic-electronic hybrid neural networks , 2019, APL Materials.

[78]  Paul R. Prucnal,et al.  Silicon Photonic Modulator Neuron , 2018, Physical Review Applied.

[79]  N. Santhanam,et al.  Artificial-intelligence hardware: New opportunities for semiconductor companies , 2019 .

[80]  Mona E. Zaghloul,et al.  Design and Fabrication of a Plasmonic Gas Sensor , 2018, 2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS).

[81]  Aleksandar Nesic,et al.  Hybrid integration of silicon photonics circuits and InP lasers by photonic wire bonding , 2018, Optica.

[82]  Ke Li,et al.  Multipurpose silicon photonics signal processor core , 2017, Nature Communications.

[83]  David A. Patterson,et al.  In-datacenter performance analysis of a tensor processing unit , 2017, 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

[84]  Dirk Englund,et al.  Deep learning with coherent nanophotonic circuits , 2017, 2017 Fifth Berkeley Symposium on Energy Efficient Electronic Systems & Steep Transistors Workshop (E3S).

[85]  D. Miller,et al.  Attojoule Optoelectronics for Low-Energy Information Processing and Communications , 2016, Journal of Lightwave Technology.

[86]  Peng-Shuai Wang,et al.  O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis , 2017, ArXiv.

[87]  Michael I. Jordan,et al.  Machine learning: Trends, perspectives, and prospects , 2015, Science.

[88]  C. Koos,et al.  Multi-chip integration of lasers and silicon photonics by photonic wire bonding , 2015, 2015 Conference on Lasers and Electro-Optics (CLEO).

[89]  Jason Cong,et al.  Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks , 2015, FPGA.

[90]  W. Freude,et al.  Connecting Silicon Photonic Circuits to Multicore Fibers by Photonic Wire Bonding , 2015, Journal of Lightwave Technology.

[91]  Jason Cong,et al.  Minimizing Computation in Convolutional Neural Networks , 2014, ICANN.

[92]  R Schmogrow,et al.  Photonic wire bonding: a novel concept for chip-scale interconnects. , 2012, Optics express.