An artificial sensory neuron with visual-haptic fusion

Human behaviors are extremely sophisticated, relying on the adaptive, plastic and event-driven network of sensory neurons. Such neuronal system analyzes multiple sensory cues efficiently to establish accurate depiction of the environment. Here, we develop a bimodal artificial sensory neuron to implement the sensory fusion processes. Such a bimodal artificial sensory neuron collects optic and pressure information from the photodetector and pressure sensors respectively, transmits the bimodal information through an ionic cable, and integrates them into post-synaptic currents by a synaptic transistor. The sensory neuron can be excited in multiple levels by synchronizing the two sensory cues, which enables the manipulating of skeletal myotubes and a robotic hand. Furthermore, enhanced recognition capability achieved on fused visual/haptic cues is confirmed by simulation of a multi-transparency pattern recognition task. Our biomimetic design has the potential to advance technologies in cyborg and neuromorphic systems by endowing them with supramodal perceptual capabilities.

[1]  Yeongjun Lee,et al.  Flexible Neuromorphic Electronics for Computing, Soft Robotics, and Neuroprosthetics , 2019, Advanced materials.

[2]  Hua Chun Zeng,et al.  Preparation of Hollow Anatase TiO2 Nanospheres via Ostwald Ripening. , 2004, The journal of physical chemistry. B.

[3]  Stefan Wermter,et al.  Evaluating Integration Strategies for Visuo-Haptic Object Recognition , 2017, Cognitive Computation.

[4]  Jong-Hyun Ahn,et al.  Biomimetic Tactile Sensors Based on Nanomaterials. , 2020, ACS nano.

[5]  Xiaodong Chen,et al.  An Artificial Somatic Reflex Arc , 2019, Advanced materials.

[6]  Yang Hui Liu,et al.  Flexible Metal Oxide/Graphene Oxide Hybrid Neuromorphic Transistors on Flexible Conducting Graphene Substrates , 2016, Advanced materials.

[7]  D. Lewkowicz.  Perception of auditory-visual temporal synchrony in human infants. , 1996, Journal of experimental psychology. Human perception and performance.

[8]  Y. S. Zhang,et al.  Electrically Driven Microengineered Bioinspired Soft Robots , 2018, Advanced materials.

[9]  Ingrid M. L. C. Vogels,et al.  Detection of Temporal Delays in Visual-Haptic Interfaces , 2004, Hum. Factors.

[10]  Zhengya Zhang,et al.  A fully integrated reprogrammable memristor–CMOS system for efficient multiply–accumulate operations , 2019, Nature Electronics.

[11]  Zhenan Bao,et al.  Multifunctional materials for implantable and wearable photonic healthcare devices , 2020, Nature Reviews Materials.

[12]  Danica Kragic,et al.  What's in the container? Classifying object contents from vision and touch , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Alex Ming Shen,et al.  A Carbon Nanotube Synapse with Dynamic Logic and Learning , 2013, Advanced materials.

[14]  Yi Shi,et al.  Organic/inorganic hybrid synaptic transistors gated by proton conducting methylcellulose films , 2016 .

[15]  Yu Wang,et al.  Highly Stretchable, Compliant, Polymeric Microelectrode Arrays for In Vivo Electrophysiological Interfacing , 2017, Advanced materials.

[16]  Xin Huang,et al.  Two-dimensional additive diethylammonium iodide promoting crystal growth for efficient and stable perovskite solar cells , 2019, RSC Advances.

[17]  Zhenan Bao,et al.  A bioinspired flexible organic artificial afferent nerve , 2018, Science.

[18]  Zhong Lin Wang,et al.  Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing , 2018, Nature Communications.

[19]  Min Ku Kim,et al.  Soft-packaged sensory glove system for human-like natural interaction and control of prosthetic hands , 2019, NPG Asia Materials.

[20]  Byung-Gook Park,et al.  Silicon synaptic transistor for hardware-based spiking neural network and neuromorphic system , 2017, Nanotechnology.

[21]  Richard Stafford,et al.  A bio-inspired visual collision detection mechanism for cars: Optimisation of a model of a locust neuron to a novel environment , 2006, Neurocomputing.

[22]  Xiaodong Chen,et al.  Skin‐Inspired Haptic Memory Arrays with an Electrically Reconfigurable Architecture , 2016, Advanced materials.

[23]  Olivier Massin,et al.  In touch. , 2002, Emergency nurse : the journal of the RCN Accident and Emergency Nursing Association.

[24]  Yong-Hoon Kim,et al.  Environment‐Adaptable Artificial Visual Perception Behaviors Using a Light‐Adjustable Optoelectronic Neuromorphic Device Array , 2019, Advanced materials.

[25]  David C. Plaut,et al.  ‘What’ Is Happening in the Dorsal Visual Pathway , 2016, Trends in Cognitive Sciences.

[26]  Z. Suo,et al.  Hydrogel ionotronics , 2018, Nature Reviews Materials.

[27]  Shigang Yue,et al.  Collision detection in complex dynamic scenes using an LGMD-based visual neural network with feature enhancement , 2006, IEEE Transactions on Neural Networks.

[28]  Zhenan Bao,et al.  Stretchable organic optoelectronic sensorimotor synapse , 2018, Science Advances.

[29]  Wei Lu,et al.  The future of electronics based on memristive systems , 2018, Nature Electronics.

[30]  Si Li,et al.  A neuro-inspired artificial peripheral nervous system for scalable electronic skins , 2019, Science Robotics.

[31]  Xiaodong Chen,et al.  Artificial Sensory Memory , 2019, Advanced materials.

[32]  H-S Philip Wong,et al.  Artificial optic-neural synapse for colored and color-mixed pattern recognition , 2018, Nature Communications.

[33]  Arindam Basu,et al.  Synergistic Gating of Electro‐Iono‐Photoactive 2D Chalcogenide Neuristors: Coexistence of Hebbian and Homeostatic Synaptic Metaplasticity , 2018, Advanced materials.

[34]  D. Ginty,et al.  The Sensory Neurons of Touch , 2013, Neuron.

[35]  Wei Zhang,et al.  Locally coupled electromechanical interfaces based on cytoadhesion-inspired hybrids to identify muscular excitation-contraction signatures , 2020, Nature Communications.

[36]  Megan L. McCain,et al.  A tissue-engineered jellyfish with biomimetic propulsion , 2012, Nature Biotechnology.

[37]  Dirk Englund,et al.  Deep learning with coherent nanophotonic circuits , 2017, 2017 Fifth Berkeley Symposium on Energy Efficient Electronic Systems & Steep Transistors Workshop (E3S).

[38]  Yuchao Yang,et al.  Ion Gated Synaptic Transistors Based on 2D van der Waals Crystals with Tunable Diffusive Dynamics , 2018, Advanced materials.

[39]  M. Davare,et al.  Interactions between dorsal and ventral streams for controlling skilled grasp , 2015, Neuropsychologia.

[40]  Christian M. Siket,et al.  Instant tough bonding of hydrogels for soft machines and electronics , 2017, Science Advances.

[41]  Allister F. McGuire,et al.  Soft conductive micropillar electrode arrays for biologically relevant electrophysiological recording , 2018, Proceedings of the National Academy of Sciences.

[42]  Satinder Singh,et al.  Artificial intelligence: Learning to play Go from scratch , 2017, Nature.

[43]  Leslie G. Ungerleider,et al.  Object vision and spatial vision: two cortical pathways , 1983, Trends in Neurosciences.

[44]  N. V. Thakor,et al.  Soft sensors for a sensing-actuation system with high bladder voiding efficiency , 2020, Science Advances.

[45]  Jeong-Woo Choi,et al.  Phototactic guidance of a tissue-engineered soft-robotic ray , 2016, Science.

[46]  Seung Hwan Lee,et al.  Temporal data classification and forecasting using a memristor-based reservoir computing system , 2019, Nature Electronics.

[47]  Ellen A. Lumpkin,et al.  Mechanisms of sensory transduction in the skin , 2007, Nature.

[48]  Xiaodong Chen,et al.  An Artificial Sensory Neuron with Tactile Perceptual Learning , 2018, Advanced materials.

[49]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.