Fast Vision Through Frameless Event-Based Sensing and Convolutional Processing: Application to Texture Recognition

Address-event representation (AER) is an emergent hardware technology which shows a high potential for providing in the near future a solid technological substrate for emulating brain-like processing structures. When used for vision, AER sensors and processors are not restricted to capturing and processing still image frames, as in commercial frame-based video technology, but sense and process visual information in a pixel-level event-based frameless manner. As a result, vision processing is practically simultaneous to vision sensing, since there is no need to wait for sensing full frames. Also, only meaningful information is sensed, communicated, and processed. Of special interest for brain-like vision processing are some already reported AER convolutional chips, which have revealed a very high computational throughput as well as the possibility of assembling large convolutional neural networks in a modular fashion. It is expected that in a near future we may witness the appearance of large scale convolutional neural networks with hundreds or thousands of individual modules. In the meantime, some research is needed to investigate how to assemble and configure such large scale convolutional networks for specific applications. In this paper, we analyze AER spiking convolutional neural networks for texture recognition hardware applications. Based on the performance figures of already available individual AER convolution chips, we emulate large scale networks using a custom made event-based behavioral simulator. We have developed a new event-based processing architecture that emulates with AER hardware Manjunath's frame-based feature recognition software algorithm, and have analyzed its performance using our behavioral simulator. Recognition rate performance is not degraded. However, regarding speed, we show that recognition can be achieved before an equivalent frame is fully sensed and transmitted.

[1]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[2]  K Fukushima,et al.  Handwritten alphanumeric character recognition by the neocognitron , 1991, IEEE Trans. Neural Networks.

[3]  Chien-Chang Chen,et al.  Filtering methods for texture discrimination , 1999, Pattern Recognit. Lett..

[4]  Fang Liu,et al.  Real-time recognition with the entire Brodatz texture database , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Kwabena Boahen,et al.  Optic nerve signals in a neuromorphic chip I: Outer and inner retina models , 2004, IEEE Transactions on Biomedical Engineering.

[6]  M. Pi,et al.  Fractal indexing with the joint statistical properties and its application in texture image retrieval , 2008 .

[7]  Bernabé Linares-Barranco,et al.  On algorithmic rate-coded AER generation , 2006, IEEE Transactions on Neural Networks.

[8]  Tobi Delbrück,et al.  A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change , 2006, 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers.

[9]  J. J. Kulikowski,et al.  Fourier analysis and spatial representation in the visual cortex , 1981, Experientia.

[10]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[11]  P.K. Biswas,et al.  Rotation-Invariant Texture Image Retrieval Using Rotated Complex Wavelet Filters , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[12]  Shutao Li,et al.  Comparison and fusion of multiresolution features for texture classification , 2004, Proceedings of 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826).

[13]  Michael Unser,et al.  Texture classification and segmentation using wavelet frames , 1995, IEEE Trans. Image Process..

[14]  John Wawrzynek,et al.  Silicon Auditory Processors as Computer Peripherals , 1992, NIPS.

[15]  E. Culurciello,et al.  A biomorphic digital image sensor , 2003, IEEE J. Solid State Circuits.

[16]  Mark J. T. Smith,et al.  Texture classification with a biorthogonal directional filter bank , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[17]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[18]  Bernabé Linares-Barranco,et al.  On Real-Time AER 2-D Convolutions Hardware for Neuromorphic Spike-Based Cortical Processing , 2008, IEEE Transactions on Neural Networks.

[19]  Pierre-Yves Burgi,et al.  A 128 /spl times/ 128 pixel 120 dB dynamic range vision sensor chip for image contrast and orientation extraction , 2003, 2003 IEEE International Solid-State Circuits Conference, 2003. Digest of Technical Papers. ISSCC..

[20]  Claus Nebauer,et al.  Evaluation of convolutional neural networks for visual recognition , 1998, IEEE Trans. Neural Networks.

[21]  Phil Brodatz,et al.  Textures: A Photographic Album for Artists and Designers , 1966 .

[22]  RussLL L. Ds Vnlos,et al.  SPATIAL FREQUENCY SELECTIVITY OF CELLS IN MACAQUE VISUAL CORTEX , 2022 .

[23]  Wan-Chi Siu,et al.  A Novel Fast and Reduced Redundancy Structure for Multiscale Directional Filter Banks , 2007, IEEE Transactions on Image Processing.

[24]  Bernabé Linares-Barranco,et al.  Fully digital AER convolution chip for vision processing , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[25]  Richard C. Dubes,et al.  Performance evaluation for four classes of textural features , 1992, Pattern Recognit..

[26]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[27]  David A. Clausi,et al.  Comparison and fusion of co‐occurrence, Gabor and MRF texture features for classification of SAR sea‐ice imagery , 2001 .

[28]  Minh N. Do,et al.  Ieee Transactions on Image Processing the Contourlet Transform: an Efficient Directional Multiresolution Image Representation , 2022 .

[29]  Tobi Delbrück,et al.  A Multichip Pulse-Based Neuromorphic Infrastructure and Its Application to a Model of Orientation Selectivity , 2007, IEEE Transactions on Circuits and Systems I: Regular Papers.

[30]  D. G. Albrecht,et al.  Spatial frequency selectivity of cells in macaque visual cortex , 1982, Vision Research.

[31]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[32]  Kai-Kuang Ma,et al.  Rotation-invariant and scale-invariant Gabor features for texture image retrieval , 2007, Image Vis. Comput..

[33]  C.-C. Jay Kuo,et al.  Texture analysis and classification with tree-structured wavelet transform , 1993, IEEE Trans. Image Process..

[34]  Tobi Delbrück,et al.  CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory–Processing– Learning–Actuating System for High-Speed Visual Object Recognition and Tracking , 2009, IEEE Transactions on Neural Networks.

[35]  Gert Cauwenberghs,et al.  An analog VLSI chip with asynchronous interface for auditory feature extraction , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[36]  William T. Freeman,et al.  Presented at: 2nd Annual IEEE International Conference on Image , 1995 .

[37]  Yingxue Wang,et al.  Quantification of a Spike-Based Winner-Take-All VLSI Network , 2008, IEEE Transactions on Circuits and Systems I: Regular Papers.

[38]  Rama Chellappa,et al.  Estimation and choice of neighbors in spatial-interaction models of images , 1983, IEEE Trans. Inf. Theory.

[39]  Gustavo Deco,et al.  Computational neuroscience of vision , 2002 .

[40]  Gert Cauwenberghs,et al.  Probabilistic synaptic weighting in a reconfigurable network of VLSI integrate-and-fire neurons , 2001, Neural Networks.

[41]  M.,et al.  Statistical and Structural Approaches to Texture , 2022 .

[42]  Stephen Grossberg,et al.  Synthetic aperture radar processing by a multiple scale neural system for boundary and surface representation , 1995, Neural Networks.

[43]  Mehdi Azadmehr,et al.  A foveated AER imager chip [address event representation] , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[44]  Anil K. Jain,et al.  Markov Random Field Texture Models , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Byung-Woo Hong,et al.  Locally Rotation, Contrast, and Scale Invariant Descriptors for Texture Analysis , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  David A. Clausi,et al.  Design-based texture feature fusion using Gabor filters and co-occurrence probabilities , 2005, IEEE Transactions on Image Processing.

[47]  Mehdi Azadmehr.  A foveated aer imager chip , 2005 .

[48]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[49]  P. Schiller,et al.  Quantitative studies of single-cell properties in monkey striate cortex. III. Spatial frequency. , 1976, Journal of neurophysiology.

[50]  Amine Bermak,et al.  A low power CMOS imager based on time-to-first-spike encoding and fair AER , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[51]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[52]  T. Sacktor.  The Synaptic Organization of the Brain (3rd Ed.) , 1991 .

[53]  Eugenio Culurciello,et al.  Event-based imaging with active illumination in sensor networks , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[54]  Bernabé Linares-Barranco,et al.  A Spatial Contrast Retina With On-Chip Calibration for Neuromorphic Spike-Based AER Vision Systems , 2007, IEEE Transactions on Circuits and Systems I: Regular Papers.

[55]  Thomas Sikora,et al.  The MPEG-7 visual standard for content description-an overview , 2001, IEEE Trans. Circuits Syst. Video Technol..

[56]  Bernabé Linares-Barranco,et al.  A Neuromorphic Cortical-Layer Microchip for Spike-Based Event Processing Vision Systems , 2006, IEEE Transactions on Circuits and Systems I: Regular Papers.

[57]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Arnaud Delorme,et al.  Networks of integrate-and-fire neurons using Rank Order Coding B: Spike timing dependent plasticity and emergence of orientation selectivity , 2001, Neurocomputing.

[59]  Glenn Healey,et al.  Feature extraction for texture discrimination via random field models with random spatial interaction , 1996, IEEE Trans. Image Process..

[60]  Kwabena Boahen,et al.  Retinomorphic chips that see quadruple images , 1999, Proceedings of the Seventh International Conference on Microelectronics for Neural, Fuzzy and Bio-Inspired Systems.

[61]  Cordelia Schmid,et al.  A sparse texture representation using local affine regions , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  G. Shepherd.  The Synaptic Organization of the Brain , 1979 .

[63]  Saurabh Dighe,et al.  An 80-Tile 1.28TFLOPS Network-on-Chip in 65nm CMOS , 2007, 2007 IEEE International Solid-State Circuits Conference. Digest of Technical Papers.

[64]  Alejandro Linares-Barranco,et al.  AER tools for communications and debugging , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[65]  Gert Cauwenberghs,et al.  An analog VLSI chip with asynchronous interface for auditory feature extraction , 1998 .

[66]  Andreas G. Andreou,et al.  AER image filtering architecture for vision-processing systems , 1999 .

[67]  B. S. Manjunath,et al.  Rotation-invariant texture classification using a complete space-frequency model , 1999, IEEE Trans. Image Process..

[68]  Trygve Randen,et al.  Filtering for Texture Classification: A Comparative Study , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[69]  Paul Scheunders,et al.  Statistical texture characterization from discrete wavelet representations , 1999, IEEE Trans. Image Process..

[70]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[71]  R. L. Valois,et al.  The orientation and direction selectivity of cells in macaque visual cortex , 1982, Vision Research.

[72]  Hong Zhang,et al.  A Fast and Effective Model for Wavelet Subband Histograms and Its Application in Texture Image Retrieval , 2006, IEEE Transactions on Image Processing.

[73]  Minh N. Do,et al.  Pyramidal directional filter banks and curvelets , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[74]  Amir Hussain,et al.  Texture Recognition by Fusion of Optimized Moment Based and Gabor Energy Features , 2008 .

[75]  Wan-Chi Siu,et al.  Multiscale directional filter bank with applications to structured and random texture retrieval , 2007, Pattern Recognit..

[76]  Lianping Chen,et al.  Effects of different Gabor filters parameters on image retrieval by texture , 2004, 10th International Multimedia Modelling Conference, 2004. Proceedings..

[77]  Massimo A. Sivilotti,et al.  Wiring considerations in analog VLSI systems, with application to field-programmable networks , 1992 .