Optical Convolutional Neural Networks: Methodology and Advances (Invited)

As a leading branch of deep learning, the convolutional neural network (CNN) is inspired by the natural visual perceptron mechanism of living things, showing great application in image recognition, language processing, and other fields. Photonics technology provides a new route for intelligent signal processing with the dramatic potential of its ultralarge bandwidth and ultralow power consumption, which automatically completes the computing process after the signal propagates through the processor with an analog computing architecture. In this paper, we focus on the key enabling technology of optical CNN, including reviewing the recent advances in the research hotspots, overviewing the current challenges and limitations that need to be further overcome, and discussing its potential application.

[1]  J. Capmany,et al.  Compact optical convolution processing unit based on multimode interference , 2023, Nature communications.

[2]  Wei Ma,et al.  Parallel photonic acceleration processor for matrix-matrix multiplication. , 2023, Optics letters.

[3]  N. Zhu,et al.  On-Demand Reconfigurable Incoherent Optical Matrix Operator for Real-Time Video Image Display , 2023, Journal of Lightwave Technology.

[4]  G. Gu,et al.  Zero-power optical convolutional neural network using incoherent light , 2023, Optics and Lasers in Engineering.

[5]  Xingyuan Xu,et al.  Photonic multiplexing techniques for neuromorphic computing , 2023, Nanophotonics.

[6]  Weiwei Hu,et al.  Microcomb-based integrated photonic processing unit , 2023, Nature Communications.

[7]  Thomas M. Chen,et al.  FatNet: High Resolution Kernels for Classification Using Fully Convolutional Optical Neural Networks , 2022, AI.

[8]  Ziming Hong,et al.  Experimental demonstration of a photonic convolutional accelerator based on a monolithically integrated multi-wavelength distributed feedback laser. , 2022, Optics letters.

[9]  V. Sorger,et al.  High‐Throughput Multichannel Parallelized Diffraction Convolutional Neural Network Accelerator , 2022, Laser & Photonics Reviews.

[10]  Long Huang,et al.  Optical processor for a binarized neural network. , 2022, Optics letters.

[11]  Y. Mao,et al.  MXene‐Based Broadband Ultrafast Nonlinear Activator for Optical Computing , 2022, Advanced Optical Materials.

[12]  Sigang Yang,et al.  LOEN: Lensless opto-electronic neural network empowered machine vision , 2022, Light: Science & Applications.

[13]  Xingzhao Liu,et al.  Training optronic convolutional neural networks on an optical system through backpropagation algorithms. , 2022, Optics express.

[14]  Xinliang Zhang,et al.  A small microring array that performs large complex-valued matrix-vector multiplication , 2022, Frontiers of Optoelectronics.

[15]  Ming Li,et al.  Photonics-enabled spiking timing-dependent convolutional neural network for real-time image classification. , 2022, Optics express.

[16]  N. Pleros,et al.  Programmable photonic neural networks combining WDM with coherent linear optics , 2022, Scientific Reports.

[17]  D. Bunandar,et al.  Delocalized photonic deep learning on the internet’s edge , 2022, Science.

[18]  S. Fan,et al.  Design of Compact Meta-Crystal Slab for General Optical Convolution , 2022, ACS Photonics.

[19]  Shurui Li,et al.  4F optical neural network acceleration: an architecture perspective , 2022, OPTO.

[20]  W. Zhen,et al.  Ultrasensitive, Ultrafast, and Gate-Tunable Two-Dimensional Photodetectors in Ternary Rhombohedral ZnIn2S4 for Optical Neural Networks. , 2022, ACS applied materials & interfaces.

[21]  H. Cai,et al.  Space-efficient optical computing with an integrated chip diffractive neural network , 2022, Nature Communications.

[22]  Yichen Shen,et al.  Photonic matrix multiplication lights up photonic accelerator and beyond , 2022, Light: Science & Applications.

[23]  Wei Zhuang,et al.  A Convolution Neural Network Implemented by Three 3 × 3 Photonic Integrated Reconfigurable Linear Processors , 2022, Photonics.

[24]  Yuankai Huo,et al.  Meta-optic accelerators for object classifiers , 2022, Science advances.

[25]  Shaofu Xu,et al.  High-order tensor flow processing using integrated photonic circuits , 2021, Nature Communications.

[26]  P. Prucnal,et al.  Monolithic Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment , 2021, Optica.

[27]  Ray T. Chen,et al.  A Compact Butterfly-Style Silicon Photonic–Electronic Neural Chip for Hardware-Efficient Deep Learning , 2021, ACS Photonics.

[28]  Junjie Yu,et al.  Optical Multi-Imaging-Casting Accelerator for Fully Parallel Universal Convolution Computing , 2021, Photonics Research.

[29]  Xingzhao Liu,et al.  Position-robust optronic convolutional neural networks dealing with images position variation , 2021, Optics Communications.

[30]  Piero Castoldi,et al.  Photonic Integrated Reconfigurable Linear Processors as Neural Network Accelerators , 2021, Applied Sciences.

[31]  Jianji Dong,et al.  Photonic Matrix Computing: From Fundamentals to Applications , 2021, Nanomaterials.

[32]  Firooz Aflatouni,et al.  An on-chip photonic deep neural network for image classification , 2021, Nature.

[33]  Shaofu Xu,et al.  Optical coherent dot-product chip for sophisticated deep learning regression , 2021, Light: Science & Applications.

[34]  Mario Miscuglio,et al.  Prospects and applications of photonic neural networks , 2021, Advances in Physics: X.

[35]  Logan G. Wright,et al.  Deep physical neural networks trained with backpropagation , 2021, Nature.

[36]  Jia Liu,et al.  Research progress in optical neural networks: theory, applications and developments , 2021, PhotoniX.

[37]  Yuhao Zhu,et al.  Design framework for metasurface optics-based convolutional neural networks. , 2021, Applied optics.

[38]  Zuyuan He,et al.  Photonic Convolution Neural Network Based on Interleaved Time-Wavelength Modulation , 2021, Journal of Lightwave Technology.

[39]  Hongwei Chen,et al.  Optoelectronic convolutional neural networks based on time-stretch method , 2021, Science China Information Sciences.

[40]  Shaofu Xu,et al.  Optical Convolutional Neural Network With WDM-Based Optical Patching and Microring Weighting Banks , 2021, IEEE Photonics Technology Letters.

[41]  A. Boes,et al.  11 TOPS photonic convolutional accelerator for optical neural networks , 2021, Nature.

[42]  Gordon Wetzstein,et al.  Inference in artificial intelligence with deep optics and photonics , 2020, Nature.

[43]  Bhavin J. Shastri,et al.  Photonics for artificial intelligence and neuromorphic computing , 2020, Nature Photonics.

[44]  Dirk Englund,et al.  Programmable photonic circuits , 2020, Nature.

[45]  Ivana Gasulla,et al.  Principles, fundamentals, and applications of programmable integrated photonics , 2020 .

[46]  Qionghai Dai,et al.  Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit , 2020, Nature Photonics.

[47]  V. Sorger,et al.  Massively parallel amplitude-only Fourier neural network , 2020, AI and Optical Data Sciences II.

[48]  Sajjad Abdollahramezani,et al.  Meta-optics for spatial optical analog computing , 2020, ArXiv.

[49]  Weiwen Zou,et al.  Optical patching scheme for optical convolutional neural networks based on wavelength-division multiplexing and optical delay lines. , 2020, Optics letters.

[50]  Butler W. Lampson,et al.  There’s plenty of room at the Top: What will drive computer performance after Moore’s law? , 2020, Science.

[51]  Tao Yan,et al.  In situ optical backpropagation training of diffractive optical neural networks , 2020 .

[52]  Ichiro Takeuchi,et al.  Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network , 2020, Nature communications.

[53]  Long Chen,et al.  Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review , 2020, IEEE Transactions on Intelligent Transportation Systems.

[54]  Xuan Li,et al.  Parallel convolutional processing using an integrated photonic tensor core , 2021, Nature.

[55]  A. Lvovsky,et al.  Backpropagation through nonlinear units for the all-optical training of neural networks , 2019, Photonics Research.

[56]  Tiberiu T. Cocias,et al.  A survey of deep learning techniques for autonomous driving , 2019, J. Field Robotics.

[57]  Masaya Notomi,et al.  Novel frontier of photonics for data processing—Photonic accelerator , 2019, APL Photonics.

[58]  Zuyuan He,et al.  Programmable matrix operation with reconfigurable time-wavelength plane manipulation and dispersed time delay. , 2019, Optics express.

[59]  Weiwen Zou,et al.  High-accuracy optical convolution unit architecture for convolutional neural networks by cascaded acousto-optical modulator arrays. , 2019, Optics express.

[60]  Qiming Zhang,et al.  Artificial neural networks enabled by nanophotonics , 2019, Light: Science & Applications.

[61]  Yue Jiang,et al.  All-optical neural network with nonlinear activation functions , 2019, Optica.

[62]  Shanhui Fan,et al.  Reprogrammable Electro-Optic Nonlinear Activation Functions for Optical Neural Networks , 2019, IEEE Journal of Selected Topics in Quantum Electronics.

[63]  Paul R. Prucnal,et al.  Machine Learning With Neuromorphic Photonics , 2019, Journal of Lightwave Technology.

[64]  Eli Shlizerman,et al.  An Optical Frontend for a Convolutional Neural Network , 2018, Applied optics.

[65]  Gordon Wetzstein,et al.  Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification , 2018, Scientific Reports.

[66]  Ivana Gasulla,et al.  Programmable multifunctional integrated nanophotonics , 2018, Nanophotonics.

[67]  Richard K. G. Do,et al.  Convolutional neural networks: an overview and application in radiology , 2018, Insights into Imaging.

[68]  Shanhui Fan,et al.  Training of Photonic Neural Networks through In Situ Backpropagation , 2018, 2019 Conference on Lasers and Electro-Optics (CLEO).

[69]  P. Lakhani,et al.  Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks. , 2017, Radiology.

[70]  Thomas N. Theis,et al.  The End of Moore's Law: A New Beginning for Information Technology , 2017, Computing in Science & Engineering.

[71]  H. Larochelle,et al.  Deep learning with coherent nanophotonic circuits , 2016, Nature Photonics.

[72]  D. Miller,et al.  Attojoule Optoelectronics for Low-Energy Information Processing and Communications , 2016, Journal of Lightwave Technology.

[73]  Humphreys,et al.  An Optimal Design for Universal Multiport Interferometers , 2016, 1603.08788.

[74]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[75]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[76]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[77]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[78]  Martin Stemmler,et al.  Power Consumption During Neuronal Computation , 2014, Proceedings of the IEEE.

[79]  Pierre Ambs,et al.  Optical Computing: A 60-Year Adventure , 2010 .

[80]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[81]  R. Schaller,et al.  Moore's law: past, present and future , 1997 .

[82]  H. Ozaktas,et al.  Fractional Fourier optics , 1995 .

[83]  Reck,et al.  Experimental realization of any discrete unitary operator. , 1994, Physical review letters.

[84]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[85]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[86]  Leonard J. Porcello,et al.  Optical data processing and filtering systems , 1960, IRE Trans. Inf. Theory.

[87]  Qian Chen,et al.  A Review of Optical Neural Networks , 2020, IEEE Access.

[88]  Marco Cococcioni,et al.  Photonic Neural Networks: A Survey , 2019, IEEE Access.

[89]  G.E. Moore,et al.  Cramming More Components Onto Integrated Circuits , 1998, Proceedings of the IEEE.

[90]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[91]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[92]  Kunihiko Fukushima,et al.  Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition , 1982 .

[93]  Harold S. Stone,et al.  A Logic-in-Memory Computer , 1970, IEEE Transactions on Computers.