Sensing and Navigation of Wearable Assistance Cognitive Systems for the Visually Impaired

This article develops a wearable vision-based assistance system to provide situational awareness for blind and visually impaired (BVI) people in indoor scenarios. The system is built upon nonintrusive wearable devices, including an RGB-D camera, an embedded computer, and haptic modules. First, the depth map and color images of the scene are obtained from an RGB-D camera, which provides 3-D environmental information. The modular work modes are then designed for different tasks, such as navigation and multitarget recognition. Then, the cognition results are summarized and presented to the user through verbal or haptic feedback. Our system is evaluated by a pilot test to validate its effectiveness of improving the navigation capabilities and multitarget recognition capabilities for the BVI in indoor environments. We present study results with different tasks, including navigation, object localization, face recognition, and text reading. The experiments prove that the system can meet the needs of the BVI in daily use.

[1]  Md Atiqur Rahman Ahad,et al.  An AI-Based Visual Aid With Integrated Reading Assistant for the Completely Blind , 2020, IEEE Transactions on Human-Machine Systems.

[2]  Liang-Bi Chen,et al.  Design and Implementation of an Intelligent Assistive System for Visually Impaired People for Aerial Obstacle Avoidance and Fall Detection , 2020, IEEE Sensors Journal.

[3]  Yi-Zeng Hsieh,et al.  Development of a wearable guide device based on convolutional neural network for blind or visually impaired persons , 2020, Multimedia Tools and Applications.

[4]  Irene Kotsia,et al.  RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jinsheng Xing,et al.  SSD Object Detection Model Based on Multi-Frequency Feature Theory , 2020, IEEE Access.

[6]  Peng Wang,et al.  Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Muhammad Haroon Yousaf,et al.  Melanoma Lesion Detection and Segmentation Using YOLOv4-DarkNet and Active Contour , 2020, IEEE Access.

[9]  Guijin Han,et al.  Face Recognition Based on The Improved MobileNet , 2019, 2019 IEEE Symposium Series on Computational Intelligence (SSCI).

[10]  Liang Sun,et al.  Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Kailas Patil,et al.  An Astute Assistive Device for Mobility and Object Recognition for Visually Impaired People , 2019, IEEE Transactions on Human-Machine Systems.

[12]  Quoc V. Le,et al.  Searching for MobileNetV3 , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Bin Jiang,et al.  Wearable Vision Assistance System Based on Binocular Sensors for Visually Impaired Users , 2019, IEEE Internet of Things Journal.

[14]  Bing Li,et al.  Vision-Based Mobile Indoor Assistive Navigation Aid for Blind People , 2019, IEEE Transactions on Mobile Computing.

[15]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Yang Tao,et al.  Validation and Optimization Framework for Indoor Navigation Systems Using User Comments in Spatial-Temporal Context , 2019, IEEE Access.

[17]  Chien-Hsing Chou,et al.  A Finger-Worn Device for Exploring Chinese Printed Text With Using CNN Algorithm on a Micro IoT Processor , 2019, IEEE Access.

[18]  Senem Velipasalar,et al.  Deep Learning-Based Obstacle Detection and Classification With Portable Uncalibrated Patterned Light , 2018, IEEE Sensors Journal.

[19]  Rongrong Ji,et al.  GroupCap: Group-Based Image Captioning with Structured Relevance and Diversity Constraints , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Pratik Rane,et al.  Self-Critical Sequence Training for Image Captioning , 2018 .

[21]  Rabia Jafri,et al.  Visual and Infrared Sensor Data-Based Obstacle Detection for the Visually Impaired Using the Google Project Tango Tablet Development Kit and the Unity Engine , 2018, IEEE Access.

[22]  Dongheui Lee,et al.  RGB-D SLAM in Dynamic Environments Using Static Point Weighting , 2017, IEEE Robotics and Automation Letters.

[23]  Pei-Ying Chiang,et al.  Simple Smartphone-Based Guiding System for Visually Impaired People , 2017, Sensors.

[24]  Mun-Cheon Kang,et al.  An enhanced obstacle avoidance method for the visually impaired using deformable grid , 2017, IEEE Transactions on Consumer Electronics.

[25]  Laura Giarré,et al.  Enabling independent navigation for visually impaired people through a wearable vision-based feedback system , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Jorma Laaksonen,et al.  Paying Attention to Descriptions Generated by Image Captioning Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Cang Ye,et al.  An Indoor Wayfinding System Based on Geometric Features Aided Graph SLAM for the Visually Impaired , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[28]  Sven P. Heinrich,et al.  Two-Tactor Vibrotactile Navigation Information for the Blind: Directional Resolution and Intuitive Interpretation , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[29]  Gang Wang,et al.  An Empirical Study of Language CNN for Image Captioning , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Juan D. Tardós,et al.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras , 2016, IEEE Transactions on Robotics.

[31]  Iyad Abu Doush,et al.  ISAB: Integrated Indoor Navigation System for the Blind , 2016, Interact. Comput..

[32]  Xiang Bai,et al.  An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Narayanan Vijaykrishnan,et al.  A Multitask Grocery Assist System for the Visually Impaired: Smart glasses, gloves, and shopping carts provide auditory and tactile feedback , 2017, IEEE Consumer Electronics Magazine.

[35]  Pan He,et al.  Detecting Text in Natural Image with Connectionist Text Proposal Network , 2016, ECCV.

[36]  Josechu J. Guerrero,et al.  Navigation Assistance for the Visually Impaired Using RGB-D Sensor With Range Expansion , 2016, IEEE Systems Journal.

[37]  Takeo Kanade,et al.  Special issue on Assistive Computer Vision and Robotics - Part I , 2016, Comput. Vis. Image Underst..

[38]  R. Usamentiaga,et al.  Face Recognition and Spoofing Detection System Adapted To Visually-Impaired People , 2016, IEEE Latin America Transactions.

[39]  Sethuraman Panchanathan,et al.  Social Interaction Assistant: A Person-Centered Approach to Enrich Social Interactions for Individuals With Visual Impairments , 2016, IEEE Journal of Selected Topics in Signal Processing.

[40]  Shuo Yang,et al.  WIDER FACE: A Face Detection Benchmark , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Yi Sun,et al.  A SLAM Based Semantic Indoor Navigation System for Visually Impaired Users , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[42]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Gérard G. Medioni,et al.  Wearable RGBD Indoor Navigation System for the Blind , 2014, ECCV Workshops.

[45]  Tianze Chen,et al.  A Union Matching Method for SAR Images Based on SIFT and Edge Strength , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[46]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[47]  Shuai Yuan,et al.  Assistive Clothing Pattern Recognition for Visually Impaired People , 2014, IEEE Transactions on Human-Machine Systems.

[48]  Klaus D. McDonald-Maier,et al.  An Algorithm for the Contextual Adaption of SURF Octave Selection With Good Matching Performance: Best Octaves , 2012, IEEE Transactions on Image Processing.

[49]  Joel A. Hesch,et al.  Design and Analysis of a Portable Indoor Localization Aid for the Visually Impaired , 2010, Int. J. Robotics Res..

[50]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[51]  Daniel E. Koditschek,et al.  Exact robot navigation using artificial potential functions , 1992, IEEE Trans. Robotics Autom..