A Multitask Grocery Assist System for the Visually Impaired: Smart glasses, gloves, and shopping carts provide auditory and tactile feedback

According to the World Health Organization, "285 million people are estimated to be visually impaired worldwide" [1]. Several technologies such as automatic text readers, Braille note makers, and navigation assistance canes have been developed to assist the visually impaired. Concurrent advances in computer vision and hardware technologies provide opportunities for a visual-assistance system that can be used in multiple contexts. As part of the Visual Cortex on Silicon program, we have been developing interfaces, algorithms, and hardware platforms to assist the visually impaired with a focus on grocery shopping. This article describes the various features that we have incorporated into this visual-assistance system so that it can be used in multiple contexts.

[1]  Yuan Gao,et al.  RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[2]  Jeffrey Stuecheli,et al.  CAPI: A Coherent Accelerator Processor Interface , 2015, IBM J. Res. Dev..

[3]  T. Stanford,et al.  The neural basis of multisensory integration in the midbrain: Its organization and maturation , 2009, Hearing Research.

[4]  Narayanan Vijaykrishnan,et al.  Enabling New Computation Paradigms with HyperFET - An Emerging Device , 2016, IEEE Transactions on Multi-Scale Computing Systems.

[5]  Hans Limburg,et al.  Visual impairment and blindness in Hungary , 2018, Acta ophthalmologica.

[6]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[8]  Donald M. Chiarulli,et al.  An image processing pipeline using coupled oscillators , 2014, 2014 14th International Workshop on Cellular Nanoscale Networks and their Applications (CNNA).

[9]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[10]  Narayanan Vijaykrishnan,et al.  Visual co-occurrence network: using context for large-scale object recognition in retail , 2015, 2015 13th IEEE Symposium on Embedded Systems For Real-time Multimedia (ESTIMedia).

[11]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[12]  John K. Tsotsos,et al.  An Information Theoretic Model of Saliency and Visual Search , 2008, WAPCV.

[13]  Iasonas Kokkinos,et al.  HOP: Hierarchical object parsing , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Narayanan Vijaykrishnan,et al.  A scalable architecture for multi-class visual object detection , 2015, 2015 25th International Conference on Field Programmable Logic and Applications (FPL).