Ultrafast machine vision with 2D material neural network image sensors

Machine vision technology has taken huge leaps in recent years, and is now becoming an integral part of various intelligent systems, including autonomous vehicles and robotics. Usually, visual information is captured by a frame-based camera, converted into a digital format and processed afterwards using a machine-learning algorithm such as an artificial neural network (ANN) 1 . The large amount of (mostly redundant) data passed through the entire signal chain, however, results in low frame rates and high power consumption. Various visual data preprocessing techniques have thus been developed 2 – 7 to increase the efficiency of the subsequent signal processing in an ANN. Here we demonstrate that an image sensor can itself constitute an ANN that can simultaneously sense and process optical images without latency. Our device is based on a reconfigurable two-dimensional (2D) semiconductor 8 , 9 photodiode 10 – 12 array, and the synaptic weights of the network are stored in a continuously tunable photoresponsivity matrix. We demonstrate both supervised and unsupervised learning and train the sensor to classify and encode images that are optically projected onto the chip with a throughput of 20 million bins per second. A two-dimensional semiconductor photodiode array senses and processes optical images simultaneously without latency, and is trained to classify and encode images with high throughput, acting as an artificial neural network.

[1]  Dirk Englund,et al.  Deep learning with coherent nanophotonic circuits , 2017, 2017 Fifth Berkeley Symposium on Energy Efficient Electronic Systems & Steep Transistors Workshop (E3S).

[2]  P. Jarillo-Herrero,et al.  Optoelectronic devices based on electrically tunable p-n diodes in a monolayer dichalcogenide. , 2013, Nature nanotechnology.

[3]  Young Min Song,et al.  Human eye-inspired soft optoelectronic device using high-density MoS2-graphene curved image sensor array , 2017, Nature Communications.

[4]  A. Kis,et al.  Nonvolatile memory cells based on MoS2/graphene heterostructures. , 2013, ACS nano.

[5]  Yi Luo,et al.  All-optical machine learning using diffractive deep neural networks , 2018, Science.

[6]  M. Gottardi,et al.  A 33 $\mu$ W 64$\,\times\,$ 64 Pixel Vision Sensor Embedding Robust Dynamic Background Subtraction for Event Detection and Scene Interpretation , 2013, IEEE Journal of Solid-State Circuits.

[7]  Helmut Fischer,et al.  New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device (PMD) , 1997, Other Conferences.

[8]  Ryan Hamerly,et al.  Large-Scale Optical Neural Networks based on Photoelectric Multiplication , 2018, Physical Review X.

[9]  Ole Bethge,et al.  A microprocessor based on a two-dimensional semiconductor , 2016, Nature Communications.

[10]  Vibhor Singh,et al.  Deterministic transfer of two-dimensional materials by all-dry viscoelastic stamping , 2013, 1311.4829.

[11]  D. Psaltis,et al.  Holography in artificial neural networks , 1990, Nature.

[12]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[13]  A. Kis,et al.  2D transition metal dichalcogenides , 2017 .

[14]  Jiaming Zhang,et al.  Analogue signal and image processing with large memristor crossbars , 2017, Nature Electronics.

[15]  Shimeng Yu,et al.  Optoelectronic resistive random access memory for neuromorphic vision sensors , 2019, Nature Nanotechnology.

[16]  Helgs Kolb,et al.  Much of the construction of an image takes place in the retina itself through the use of specialized neural circuits , 2002 .

[17]  Tobi Delbrück,et al.  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output , 2014, Proceedings of the IEEE.

[18]  R. Gerchberg.  A practical algorithm for the determination of phase from image and diffraction plane pictures , 1972 .

[19]  Misha Mahowald,et al.  A silicon model of early visual processing , 1993, Neural Networks.

[20]  T. Mueller,et al.  Solar-energy conversion and light emission in an atomic monolayer p-n diode. , 2013, Nature Nanotechnology.

[21]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[22]  T. Sugeta,et al.  Metal-Semiconductor-Metal Photodetector for High-Speed Optoelectronic Circuits , 1980 .

[23]  Nan Zhang,et al.  Reconfigurable two-dimensional optoelectronic devices enabled by local ferroelectric polarization , 2019, Nature Communications.

[24]  S. Goossens,et al.  Broadband image sensor array based on graphene–CMOS integration , 2017, Nature Photonics.

[25]  Yoshua Bengio,et al.  Practical Recommendations for Gradient-Based Training of Deep Architectures , 2012, Neural Networks: Tricks of the Trade.

[26]  Geoffrey E. Hinton,et al.  Semantic hashing , 2009, Int. J. Approx. Reason..

[27]  Aaron M. Jones,et al.  Electrically tunable excitonic light-emitting diodes based on monolayer WSe2 p-n junctions. , 2013, Nature nanotechnology.

[28]  Ermin Malic,et al.  Exciton physics and device application of two-dimensional transition metal dichalcogenide semiconductors , 2018, npj 2D Materials and Applications.

[29]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[30]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[31]  Luke P. Lee,et al.  Biologically Inspired Artificial Compound Eyes , 2006, Science.

[32]  Kazuo Kyuma,et al.  Artificial retinas — fast, versatile image processors , 1994, Nature.

[33]  Takashi Taniguchi,et al.  Dissociation of two-dimensional excitons in monolayer WSe2 , 2018, Nature Communications.

[34]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[35]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[36]  Wei D. Lu,et al.  Sparse coding with memristor networks. , 2017, Nature nanotechnology.

[37]  P. Ajayan,et al.  Two-dimensional non-volatile programmable p-n junctions. , 2017, Nature nanotechnology.

[38]  H. Kolb.  How the Retina Works , 2003, American Scientist.

[39]  Narayan Srinivasa,et al.  A functional hybrid memristor crossbar-array/CMOS system for data storage and neuromorphic applications. , 2012, Nano letters.