A Wearable Assistive Technology for the Visually Impaired with Door Knob Detection and Real-Time Feedback for Hand-to-Handle Manipulation

The visually impaired are consistently faced with mobility restrictions due to the lack of truly accessible environments. Even in structured settings, people with low vision may still have trouble navigating efficiently and safely due to hallway and threshold ambiguity. Assistive technologies that are currently available do not provide door and door-handle object detections nor do they concretely help the visually impaired reaching towards the object. In this paper, we propose an AI-driven wearable assistive technology that integrates door handle detection, user's real-time hand position in relation to this targeted object, and audio feedback for "joy stick-like command" for acquisition of the target and subsequent hand-to-handle manipulation. When fully envisioned, this platform will help end users locate doors and door handles and reach them with feedback, enabling them to travel safely and efficiently when navigating through environments with thresholds. Compared to the usual computer vision models, the one proposed in this paper requires significantly fewer computational resources, which allows it to pair with a stereoscopic camera running on a small graphics processing unit (GPU). This permits us to take advantage of its convenient portability. We also introduce a dataset containing different types of door handles and door knobs with bounding-box annotations, which can be used for training and testing in future research.

[1]  Stefano Mattoccia,et al.  3D Glasses as Mobility Aid for Visually Impaired People , 2014, ECCV Workshops.

[2]  Rogério Schmidt Feris,et al.  A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection , 2016, ECCV.

[3]  Basilio Sierra,et al.  DOOR HANDLE IDENTIFICATION: A THREE-STAGE APPROACH , 2007 .

[4]  Xiaodong Yang,et al.  Robust door detection in unfamiliar environments by combining edge and corner features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[5]  Basilio Sierra,et al.  Approaches to Door Identification for Robot Navigation , 2010 .

[6]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  M. S. Panse,et al.  Character detection and recognition system for visually impaired people , 2016, 2016 IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT).

[8]  John-Ross Rizzo,et al.  Sensor fusion for ecologically valid obstacle identification: Building a comprehensive assistive technology platform for the visually impaired , 2017, 2017 7th International Conference on Modeling, Simulation, and Applied Optimization (ICMSAO).

[9]  Matti Siekkinen,et al.  How low energy is bluetooth low energy? Comparative measurements with ZigBee/802.15.4 , 2012, 2012 IEEE Wireless Communications and Networking Conference Workshops (WCNCW).

[10]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[11]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  James M. Coughlan,et al.  CamIO: a 3D computer vision system enabling audio/haptic interaction with physical objects by blind users , 2013, ASSETS.

[13]  Michael Beetz,et al.  Laser-based perception for door and handle identification , 2009, 2009 International Conference on Advanced Robotics.

[14]  Shuihua Wang,et al.  RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs , 2014, J. Vis. Commun. Image Represent..

[15]  ZhangChenyang,et al.  RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs , 2014 .

[16]  W. Marsden I and J , 2012 .

[17]  Rangachar Kasturi,et al.  Machine vision , 1995 .

[18]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Rahmat A. Shoureshi,et al.  Smart Wearable Systems for Enhanced Monitoring and Mobility , 2016 .

[21]  Oliver Bittel,et al.  Real-Time Door Detection Based on AdaBoost Learning Algorithm , 2009, Eurobot Conference.

[22]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Giovanni Maria Farinella,et al.  An Electronic Travel Aid to Assist Blind and Visually Impaired People to Avoid Obstacles , 2015, CAIP.

[24]  Florica Moldoveanu,et al.  Real-Time Indoor Staircase Detection on Mobile Devices , 2017, 2017 21st International Conference on Control Systems and Computer Science (CSCS).

[25]  Paramvir Bahl,et al.  GLIMPSE: Continuous, Real-Time Object Recognition on Mobile Devices , 2016, GetMobile Mob. Comput. Commun..