Bio-inspired smart vision sensor: toward a reconfigurable hardware modeling of the hierarchical processing in the brain

Biological vision systems inspire processing methods in computer vision applications. This paper employs the insights of vision systems in hardware and presents a pixel-parallel, reconfigurable, and layer-based hierarchical architecture for smart image sensors. The architecture aims to bring computation close to the sensor to achieve high acceleration for different machine vision applications while consuming low power. We logically divide the image into multiple regions and perform pixel-level and region-level processing after removing spatiotemporal redundancy. Those processors use bio-inspired algorithms to activate the regions with region of interest of a scene. The hierarchical processing breaks the traditional sequential image processing and introduces parallelism for machine vision applications. Also, we make the hardware design reconfigurable even after fabrication to make the hardware reusable for different applications. Simulation results show that the area overhead and power penalty for adding reconfigurable features stay in an acceptable range. We emphasize to maximize the operating speed and obtain 800 MHz. Besides, the design saves 84.01% and 96.91% dynamic power at the first and second stages of the hierarchy by removing redundant information. Furthermore, the sequential deployment of high-level reasoning only on the selected regions of the image becomes computationally inexpensive to execute a complex task in real time.

[1]  Tai Sing Lee,et al.  Hierarchical Bayesian inference in the visual cortex. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[2]  M. Ikeda,et al.  A 375 /spl times/ 365 high-speed 3-D range-finding image sensor using row-parallel search architecture and multisampling technique , 2005, IEEE Journal of Solid-State Circuits.

[3]  S. Hochstein,et al.  The reverse hierarchy theory of visual perceptual learning , 2004, Trends in Cognitive Sciences.

[4]  B. Tyrrell,et al.  Time Delay Integration and In-Pixel Spatiotemporal Filtering Using a Nanoscale Digital CMOS Focal Plane Readout , 2009, IEEE Transactions on Electron Devices.

[5]  Tobias Delbrück,et al.  Frame-free dynamic digital vision , 2008 .

[6]  D C Van Essen,et al.  Information processing in the primate visual system: an integrated systems perspective. , 1992, Science.

[7]  Hongbo Zhu,et al.  A Real-Time Motion-Feature-Extraction VLSI Employing Digital-Pixel-Sensor-Based Parallel Architecture , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[8]  E. Callaway,et al.  Parallel processing strategies of the primate visual system , 2009, Nature Reviews Neuroscience.

[9]  R. Desimone,et al.  Attention Increases Sensitivity of V4 Neurons , 2000, Neuron.

[10]  Steve Eugene Watkins,et al.  An overview of biomimetic sensor technology , 2009 .

[11]  Christophe Bobda,et al.  Pixel-Parallel Architecture for Neuromorphic Smart Image Sensor with Visual Attention , 2018, 2018 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[12]  Michael W. Hoffman,et al.  A CMOS Imager With Focal Plane Compression Using Predictive Coding , 2007, IEEE Journal of Solid-State Circuits.

[13]  Bhaskar Choubey,et al.  Advances on CMOS image sensors , 2016 .

[14]  Pierre Kornprobst,et al.  Bio-inspired computer vision: Towards a synergistic approach of artificial and biological vision , 2016, Comput. Vis. Image Underst..

[15]  Stephen A Baccus,et al.  Insights from the retina into the diverse and general computations of adaptation, detection, and prediction , 2014, Current Opinion in Neurobiology.

[16]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[17]  Rajesh P. N. Rao,et al.  Predictive Coding , 2019, A Blueprint for the Hard Problem of Consciousness.

[18]  Eugenio Culurciello,et al.  Activity-driven, event-based vision sensors , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[19]  S. Hochstein,et al.  View from the Top Hierarchies and Reverse Hierarchies in the Visual System , 2002, Neuron.

[20]  Peter Földiák,et al.  SPARSE CODING IN THE PRIMATE CORTEX , 2002 .

[21]  Wancheng Zhang,et al.  A Programmable Vision Chip Based on Multiple Levels of Parallel Processors , 2011, IEEE Journal of Solid-State Circuits.

[22]  Bernabé Linares-Barranco,et al.  A 32$\,\times\,$ 32 Pixel Convolution Processor Chip for Address Event Vision Sensors With 155 ns Event Latency and 20 Meps Throughput , 2011, IEEE Transactions on Circuits and Systems I: Regular Papers.

[23]  Paulo Da Cunha Possa,et al.  P2IP: A novel low-latency Programmable Pipeline Image Processor , 2015, Microprocess. Microsystems.

[24]  Peter Elias,et al.  Predictive coding-I , 1955, IRE Trans. Inf. Theory.

[25]  Derek Abbott,et al.  An insect vision-based motion detection chip , 1997, IEEE J. Solid State Circuits.

[26]  Bernabé Linares-Barranco,et al.  A 128$\,\times$ 128 1.5% Contrast Sensitivity 0.9% FPN 3 µs Latency 4 mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance Preamplifiers , 2013, IEEE Journal of Solid-State Circuits.

[27]  Shinya Miyata,et al.  A 6.9- $\mu$ m Pixel-Pitch Back-Illuminated Global Shutter CMOS Image Sensor With Pixel-Parallel 14-Bit Subthreshold ADC , 2018, IEEE Journal of Solid-State Circuits.

[28]  P. Milner.  A model for visual shape recognition. , 1974, Psychological review.

[29]  Frank Vahid,et al.  Embedded system design - a unified hardware / software introduction , 2001 .

[30]  Michael S. Landy,et al.  Computational models of visual attention , 2011, Vision Research.

[31]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Moshe Gur,et al.  Space reconstruction by primary visual cortex activity: a parallel, non-computational mechanism of object representation , 2015, Trends in Neurosciences.

[33]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[34]  C. Caltagirone,et al.  Neural networks engaged in milliseconds and seconds time processing: evidence from transcranial magnetic stimulation and patients with cortical or subcortical dysfunction , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[35]  John K. Tsotsos,et al.  Towards the Quantitative Evaluation of Visual Attention Models Bottom−up Top-down Dynamic Static 0 0 0 , 2022 .

[36]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[37]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[38]  Fred Rieke,et al.  Review the Challenges Natural Images Pose for Visual Adaptation , 2022 .

[39]  Christophe Bobda,et al.  Visual Cortex Inspired Pixel-Level Re-configurable Processors for Smart Image Sensors , 2019, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[40]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[41]  Joseph J. Atick,et al.  Towards a Theory of Early Visual Processing , 1990, Neural Computation.

[42]  B. C. Motter.  Focal attention produces spatially selective processing in visual cortical areas V1, V2, and V4 in the presence of competing stimuli. , 1993, Journal of neurophysiology.

[43]  Bernabé Linares-Barranco,et al.  On Real-Time AER 2-D Convolutions Hardware for Neuromorphic Spike-Based Cortical Processing , 2008, IEEE Transactions on Neural Networks.

[44]  M. Larkum.  A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex , 2013, Trends in Neurosciences.

[45]  R. Desimone,et al.  Attention Increases Sensitivity of V4 Neurons , 2000, Neuron.

[46]  R. W. Rodieck.  The First Steps in Seeing , 1998 .

[47]  Nanjian Wu,et al.  A Novel Vision Chip for High-Speed Target Tracking , 2006 .

[48]  Marjan Asadinia,et al.  Design of a Reconfigurable 3D Pixel-Parallel Neuromorphic Architecture for Smart Image Sensor , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[49]  H. Kennedy,et al.  Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas , 2016, Neuron.

[50]  E. Culurciello,et al.  A biomorphic digital image sensor , 2003, IEEE J. Solid State Circuits.

[51]  Amine Bermak,et al.  Arbitrated Time-to-First Spike CMOS Image Sensor With On-Chip Histogram Equalization , 2007, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.