RGB-D Sensor-Based Computer Vision Assistive Technology for Visually Impaired Persons

A computer vision-based wayfinding and navigation aid can improve the mobility of blind and visually impaired people to travel independently. In this chapter, we focus on RGB-D sensor-based computer vision technologies in application to assist blind and visually impaired persons. We first briefly review the existing computer vision based assistive technology for the visually impaired. Then we provide a detailed description of the recent RGB-D sensor based assistive technology to help blind or visually impaired people. Next, we present the prototype system to detect and recognize stairs and pedestrian crosswalks based on RGB-D images. Since both stairs and pedestrian crosswalks are featured by a group of parallel lines, Hough transform is applied to extract the concurrent parallel lines based on the RGB (Red, Green, and Blue) channels. Then, the Depth channel is employed to recognize pedestrian crosswalks and stairs. The detected stairs are further identified as stairs going up (upstairs) and stairs going down (downstairs). The distance between the camera and stairs is also estimated for blind users. The detection and recognition results on our collected datasets demonstrate the effectiveness and efficiency of our developed prototype. We conclude the chapter by the discussion of the future directions.

[1]  Yingli Tian,et al.  Portable Camera-Based Assistive Text and Product Label Reading From Hand-Held Objects for Blind Persons , 2014, IEEE/ASME Transactions on Mechatronics.

[2]  Yingli Tian,et al.  Assistive Text Reading from Complex Background for Blind Persons , 2011, CBDAR.

[3]  Roman Kuc,et al.  Binaural sonar electronic travel aid provides vibrotactile cues for landmark, reflector motion and surface texture classification , 2002, IEEE Transactions on Biomedical Engineering.

[4]  Subhas Chandra Mukhopadhyay,et al.  Wearable and Autonomous Biomedical Devices and Systems for Smart Environment: Issues and Characterization , 2010 .

[5]  Xiaodong Yang,et al.  Robust and Effective Component-Based Banknote Recognition for the Blind , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[6]  Wolfgang L. Zagler,et al.  Computers Helping People with Special Needs, 12th International Conference, ICCHP 2010, Vienna, Austria, July 14-16, 2010, Proceedings, Part II , 2010, ICCHP.

[7]  James M. Coughlan,et al.  Detecting and locating crosswalks using a camera phone , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[8]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[9]  Shuihua Wang,et al.  RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs , 2014, J. Vis. Commun. Image Represent..

[10]  Michael Brady,et al.  Vision-based Detection of Stair-cases , 2003 .

[11]  Hong Liu,et al.  Segment and Label Indoor Scene Based on RGB-D for the Visually Impaired , 2014, MMM.

[12]  Ron Kupers,et al.  Navigation with a sensory substitution device in congenitally blind individuals , 2011, Neuroreport.

[13]  Xiaodong Yang,et al.  Toward a computer vision-based wayfinding aid for blind persons to access unfamiliar indoor environments , 2012, Machine Vision and Applications.

[14]  R K Kaminer,et al.  Blindness and visual impairment. , 1995, Pediatrics in review.

[15]  Chung Hyuk Park,et al.  Real-time haptic rendering and haptic telepresence robotic system for the visually impaired , 2013, 2013 World Haptics Conference (WHC).

[16]  Zhigang Zhu,et al.  KinDectect: Kinect Detecting Objects , 2012, ICCHP.

[17]  S M Genensky,et al.  SOME COMMENTS ON A CLOSED CIRCUIT TV SYSTEM FOR THE VISUALLY HANDICAPPED* , 1969, American journal of optometry and archives of American Academy of Optometry.

[18]  Shuai Yuan,et al.  Assistive Clothing Pattern Recognition for Visually Impaired People , 2014, IEEE Transactions on Human-Machine Systems.

[19]  N. A. Bradley,et al.  Assistive Technology For Visually Impaired And Blind People , 2008 .

[20]  Nikolaos G. Bourbakis,et al.  Wearable Obstacle Avoidance Electronic Travel Aids for Blind: A Survey , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[21]  Gordon E. Legge,et al.  Blind Navigation and the Role of Technology , 2008 .

[22]  Ron Kupers,et al.  Tactile–‘visual’ acuity of the tongue in early blind individuals , 2007, Neuroreport.

[23]  Yvonne Hsu-Lin Luo,et al.  The Argus II Retinal Prosthesis System , 2019, Prosthesis.

[24]  Stephen Se.  Zebra-crossing detection for the partially sighted , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[25]  M. Bousbia-Salah,et al.  An Ultrasonic Navigation System for Blind People , 2007, 2007 IEEE International Conference on Signal Processing and Communications.

[26]  A Arditi,et al.  Computing for the blind user , 1986 .

[27]  Yingli Tian,et al.  Text extraction from scene images by character appearance and structure modeling , 2013, Comput. Vis. Image Underst..

[28]  Ramiro Velazquez,et al.  Wearable Assistive Devices for the Blind , 2016, ArXiv.

[29]  Yingli Tian,et al.  A primary travelling assistant system of bus detection and recognition for visually impaired people , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[30]  Jizhong Xiao,et al.  Semantic Indoor Navigation with a Blind-User Oriented Augmented Reality , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[31]  Chucai Yi,et al.  Text String Detection From Natural Scenes by Structure-Based Partition and Grouping , 2011, IEEE Transactions on Image Processing.

[32]  James M. Coughlan,et al.  Crosswatch: A Camera Phone System for Orienting Visually Impaired Pedestrians at Traffic Intersections , 2008, ICCHP.

[33]  Shi,et al.  A Fast Algorithm for Finding Crosswalks using Figure-Ground Segmentation , 2006 .

[34]  Michael D. Williams,et al.  The Use of a Tactile-Vision Sensory Substitution System as an Augmentative Tool for Individuals with Visual Impairments , 2011 .

[35]  Roberto Manduchi,et al.  (Computer) vision without sight , 2012, Commun. ACM.

[36]  Han Wang,et al.  2D staircase detection using real AdaBoost , 2009, 2009 7th International Conference on Information, Communications and Signal Processing (ICICS).

[37]  David C. Mountain,et al.  DESIGN OF A SONAR SYSTEM FOR VISUALLY IMPAIRED HUMANS , 2008 .

[38]  Bitjoka Laurent,et al.  A sonar system modeled after spatial hearing and echolocating bats for blind mobility aid , 2007 .

[39]  Yingli Tian,et al.  Reading labels of cylinder objects for blind persons , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[40]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[41]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[42]  Sumi Helal,et al.  The Engineering Handbook of Smart Technology for Aging, Disability, and Independence , 2008 .

[43]  Shuai Yuan,et al.  Clothing Matching for Visually Impaired Persons. , 2011, Technology and disability.

[44]  Amirhossein Tamjidi,et al.  6-DOF pose estimation of a Portable Navigation Aid for the visually impaired , 2013, 2013 IEEE International Symposium on Robotic and Sensors Environments (ROSE).

[45]  Avinash C. Kak,et al.  Vision for Mobile Robot Navigation: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  A. Arditi,et al.  User Interface Preferences in the Design of a Camera-Based Navigation and Wayfinding Aid , 2013 .

[47]  Yingli Tian,et al.  Localizing Text in Scene Images by Boundary Clustering, Stroke Segmentation, and String Fragment Classification , 2012, IEEE Transactions on Image Processing.

[48]  Francisco Bonin-Font,et al.  Visual Navigation for Mobile Robots: A Survey , 2008, J. Intell. Robotic Syst..

[49]  Roberto Manduchi,et al.  Detection and Localization of Curbs and Stairways Using Stereo Vision , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[50]  Zhengyou Zhang,et al.  Auditory augmented reality: Object sonification for the visually impaired , 2012, 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP).

[51]  Hao Tang,et al.  From RGB-D to low-resolution tactile: Smart sampling and early testing , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[52]  Gert Jan Gelderblom,et al.  Inventory of Electronic Mobility Aids for Persons with Visual Impairments: A Literature Review , 2008 .

[53]  L. Kay,et al.  A sonar aid to enhance spatial perception of the blind: engineering design and evaluation , 1974 .

[54]  K. Karacs,et al.  Advanced crosswalk detection for the Bionic Eyeglass , 2010, 2010 12th International Workshop on Cellular Nanoscale Networks and their Applications (CNNA 2010).

[55]  Mohammad Shorif Uddin,et al.  Bipolarity and Projective Invariant-Based Zebra-Crossing Detection for the Visually Impaired , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[56]  Gérard G. Medioni,et al.  Real-time staircase detection from a wearable stereo system , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).