Machine Vision using Diffractive Spectral Encoding

Machine vision systems mostly rely on lens-based optical imaging architectures that relay the spatial information of objects onto high pixel-count opto-electronic sensor arrays, followed by digital processing of this information. Here, we demonstrate an optical machine vision system that uses trainable matter in the form of diffractive layers to transform and encode the spatial information of objects into the power spectrum of the diffracted light, which is used to perform optical classification of objects with a single-pixel spectroscopic detector. Using a time-domain spectroscopy setup with a plasmonic nanoantenna-based detector, we experimentally validated this framework at terahertz spectrum to optically classify the images of handwritten digits by detecting the spectral power of the diffracted light at ten distinct wavelengths, each representing one class/digit. We also report the coupling of this spectral encoding achieved through a diffractive optical network with a shallow electronic neural network, separately trained to reconstruct the images of handwritten digits based on solely the spectral information encoded in these ten distinct wavelengths within the diffracted light. These reconstructed images demonstrate task-specific image decompression and can also be cycled back as new inputs to the same diffractive network to improve its optical object classification. This unique framework merges the power of deep learning with the spatial and spectral processing capabilities of trainable matter, and can also be extended to other spectral-domain measurement systems to enable new 3D imaging and sensing modalities integrated with spectrally encoded classification tasks performed through diffractive networks.

[1]  Daniel Brunner,et al.  Diffractive Coupling For Photonic Networks: How Big Can We Go? , 2019, IEEE Journal of Selected Topics in Quantum Electronics.

[2]  Dmitry K. Polyushkin,et al.  Ultrafast machine vision with 2D material neural network image sensors , 2020, Nature.

[3]  Dirk Englund,et al.  Deep learning with coherent nanophotonic circuits , 2017, 2017 Fifth Berkeley Symposium on Energy Efficient Electronic Systems & Steep Transistors Workshop (E3S).

[4]  Paul R. Prucnal,et al.  Principles of Neuromorphic Photonics , 2017, ArXiv.

[5]  A. Sideris,et al.  A multilayered neural network controller , 1988, IEEE Control Systems Magazine.

[6]  Yi Luo,et al.  Analysis of Diffractive Optical Neural Networks and Their Integration With Electronic Neural Networks , 2018, IEEE Journal of Selected Topics in Quantum Electronics.

[7]  Lianqing Zhu,et al.  Miniaturized Diffraction Grating Design and Processing for Deep Neural Network , 2019, IEEE Photonics Technology Letters.

[8]  D Psaltis,et al.  Optical implementation of the Hopfield model. , 1985, Applied optics.

[9]  Yi Luo,et al.  Class-specific differential detection in diffractive optical neural networks improves inference accuracy , 2019, Advanced Photonics.

[10]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Didier Sornette,et al.  Encyclopedia of Complexity and Systems Science , 2009 .

[12]  D. Psaltis,et al.  Holography in artificial neural networks , 1990, Nature.

[13]  Iu,et al.  All-optical neural network with nonlinear activation functions: supplementary material , 2019 .

[14]  K Wagner,et al.  Multilayer optical learning networks. , 1987, Applied optics.

[15]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[16]  Yi Luo,et al.  Design of task-specific optical systems using broadband diffractive neural networks , 2019, Light, science & applications.

[17]  Eli Shlizerman,et al.  An Optical Frontend for a Convolutional Neural Network , 2018, Applied optics.

[18]  Laurent Larger,et al.  Reinforcement Learning in a large scale photonic Recurrent Neural Network , 2017, Optica.

[19]  Christopher W. Berry,et al.  High-Power Terahertz Generation Using Large-Area Plasmonic Photoconductive Emitters , 2015, IEEE Transactions on Terahertz Science and Technology.

[20]  N. Engheta,et al.  Inverse-designed metastructures that solve equations , 2019, Science.

[21]  Paul R. Prucnal,et al.  Progress in neuromorphic photonics , 2017 .

[22]  David R. Smith,et al.  Learned Integrated Sensing Pipeline: Reconfigurable Metasurface Transceivers as Trainable Physical Layer in an Artificial Neural Network , 2019, Advanced science.

[23]  Yue Jiang,et al.  All-optical neural network with nonlinear activation functions , 2019, Optica.

[24]  J. Feldmann,et al.  All-optical spiking neurosynaptic networks with self-learning capabilities , 2019, Nature.

[25]  Yi Luo,et al.  All-optical machine learning using diffractive deep neural networks , 2018, Science.

[26]  Gordon Wetzstein,et al.  Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification , 2018, Scientific Reports.

[27]  Nezih Tolga Yardimci,et al.  High Sensitivity Terahertz Detection through Large-Area Plasmonic Nano-Antenna Arrays , 2016, Scientific Reports.

[28]  Jian Xu,et al.  Performing optical logic operations by a diffractive neural network , 2020, Light: Science & Applications.

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[31]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[32]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[33]  Lei Ying,et al.  Nanophotonic media for artificial neural inference , 2018, Photonics Research.