CNN-based object recognition and tracking system to assist visually impaired people

Visually impaired persons (VIPs) comprise a significant portion of the population, and they are present around the globe and in every part of the world. In recent times, technology proved its presence in every domain, and innovative devices assist humans in their daily lives. In this work, a smart and intelligent system is designed for VIPs to assist mobility and ensure their safety. The proposed system provides navigation in real-time using an automated voice. Though VIPs wouldn’t be able to see objects in their surroundings, they can sense and visualize the roaming environment. Moreover, a web-based application is developed to ensure their safety. The user of this application can turn the on-demand function for sharing his/her location with the family while compromising privacy. Through this application, the family members of VIPs would be able to track their movement (get location and snapshots) while being at their homes. Hence, the device allows VIPs to visualize the environment and ensure their security. Such a comprehensive device was a missing link in the existing literature. The application uses MobileNet architecture due to its low computational complexity to run on low-power end devices. To assess the efficacy of the proposed system, six pilot studies have been performed that reflected satisfactory results. For object detection and recognition, a deep Convolution Neural Network (CNN) model is employed with an accuracy of 83.3%, whereas the dataset contains more than 1000 categories. Moreover, a score-based quantitative comparative analysis is performed using the supported features of devices. It is found that the proposed system has outperformed the existing devices having a total score of 9.1/10, which is 8% higher than the second-best.

[1]  Liang-Bi Chen,et al.  An Artificial Intelligence Edge Computing-Based Assistive System for Visually Impaired Pedestrian Safety at Zebra Crossings , 2021, IEEE Transactions on Consumer Electronics.

[2]  Muhammad Sheikh Sadi,et al.  IoT Enabled Automated Object Recognition for the Visually Impaired , 2021 .

[3]  Liang-Bi Chen,et al.  Design and Implementation of an Intelligent Assistive System for Visually Impaired People for Aerial Obstacle Avoidance and Fall Detection , 2020, IEEE Sensors Journal.

[4]  Miguel Cazorla,et al.  Enhancing perception for the visually impaired with deep learning techniques and low-cost wearable sensors , 2020, Pattern Recognit. Lett..

[5]  Titus Zaharia,et al.  Wearable assistive devices for visually impaired: A state of the art survey , 2020, Pattern Recognit. Lett..

[6]  Md. Milon Islam,et al.  Obstacle and Fall Detection to Guide the Visually Impaired People with Real Time Monitoring , 2020, SN Computer Science.

[7]  D. Ashley.  Django , 2020, Foundation Dynamic Web Pages with Python.

[8]  Zeki Yetgin,et al.  Design and Implementation of an Embedded Real-Time System for Guiding Visually Impaired Individuals , 2019, 2019 International Artificial Intelligence and Data Processing Symposium (IDAP).

[9]  Kailas Patil,et al.  An Astute Assistive Device for Mobility and Object Recognition for Visually Impaired People , 2019, IEEE Transactions on Human-Machine Systems.

[10]  Md. Milon Islam,et al.  “BlindShoe”: An Electronic Guidance System for the Visually Impaired People , 2019 .

[11]  Yi Zhu,et al.  An ARCore Based User Centric Assistive Navigation System for Visually Impaired People , 2019, Applied Sciences.

[12]  Liang-Bi Chen,et al.  An Implementation of an Intelligent Assistance System for Visually Impaired/Blind People , 2019, 2019 IEEE International Conference on Consumer Electronics (ICCE).

[13]  Mariusz Kubanek,et al.  Device for Acoustic Support of Orientation in the Surroundings for Blind People † , 2018, Sensors.

[14]  Rachel Huang,et al.  YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[15]  Ali Javed,et al.  X-EYE: A Bio-smart Secure Navigation Framework for Visually Impaired People , 2018, 2018 International Conference on Signal Processing and Information Security (ICSPIS).

[16]  Hossein Mahvash Mohammadi,et al.  Obstacle Detection Using GoogleNet , 2018, 2018 8th International Conference on Computer and Knowledge Engineering (ICCKE).

[17]  Rajesh Kannan Megalingam,et al.  Autonomous Path Guiding Robot for Visually Impaired People , 2018, Cognitive Informatics and Soft Computing.

[18]  Guoqiang Li,et al.  A CNN-SIFT Hybrid Pedestrian Navigation Method Based on First-Person Vision , 2018, Remote. Sens..

[19]  Andreas Kunz,et al.  Virtual Navigation Environment for Blind and Low Vision People , 2018, ICCHP.

[20]  Yogesh K. Meghrajani,et al.  Multisensor - Based Object Detection in Indoor Environment for Visually Impaired People , 2018, 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS).

[21]  Emanuele Frontoni,et al.  Mechatronic System to Help Visually Impaired Users During Walking and Running , 2018, IEEE Transactions on Intelligent Transportation Systems.

[22]  Xiangyu Zhang,et al.  ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Akihiro Yamashita,et al.  Pedestrian Navigation System for Visually Impaired People Using HoloLens and RFID , 2017, 2017 Conference on Technologies and Applications of Artificial Intelligence (TAAI).

[24]  Mun-Cheon Kang,et al.  Collision detection method using image segmentation for the visually impaired , 2017, IEEE Transactions on Consumer Electronics.

[25]  Adrian Burlacu,et al.  Computer Vision for the Visually Impaired: the Sound of Vision System , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[26]  Petra Poulova,et al.  Google Glass Used as Assistive Technology Its Utilization for Blind and Visually Impaired People , 2017, MobiWIS.

[27]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[28]  Khaled M. Elleithy,et al.  Sensor-Based Assistive Devices for Visually-Impaired People: Current Status, Challenges, and Future Directions , 2017, Sensors.

[29]  Anderson Rocha,et al.  A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users , 2017, IEEE Transactions on Human-Machine Systems.

[30]  Ramiro Velazquez,et al.  Wearable Assistive Devices for the Blind , 2016, ArXiv.

[31]  Jian Bai,et al.  Expanding the Detection of Traversable Area with RealSense for the Visually Impaired , 2016, Sensors.

[32]  Ruxandra Tapu,et al.  When Ultrasonic Sensors and Computer Vision Join Forces for Efficient Obstacle Detection and Recognition , 2016, Sensors.

[33]  Jörg Conradt,et al.  A mobility device for the blind with improved vertical resolution using dynamic vision sensors , 2016, 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom).

[34]  Josechu J. Guerrero,et al.  Navigation Assistance for the Visually Impaired Using RGB-D Sensor With Range Expansion , 2016, IEEE Systems Journal.

[35]  Tugrul Çavdar,et al.  Ultrasonic Assistive Headset for visually impaired people , 2016, 2016 39th International Conference on Telecommunications and Signal Processing (TSP).

[36]  Stefano Mattoccia,et al.  A wearable mobility aid for the visually impaired based on embedded 3D vision and deep learning , 2016, 2016 IEEE Symposium on Computers and Communication (ISCC).

[37]  Joan García-Haro,et al.  Design, Implementation and Evaluation of an Indoor Navigation System for Visually Impaired People , 2015, Sensors.

[38]  Yiting Yi,et al.  A design of blind-guide crutch based on multi-sensors , 2015, 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[39]  Hanghang Tong,et al.  Mobile Crowd Assisted Navigation for the Visually Impaired , 2015, 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom).

[40]  Paulus Insap Santosa,et al.  Obstacle Avoidance for Visually Impaired Using Auto-Adaptive Thresholding on Kinect's Depth Image , 2014, 2014 IEEE 11th Intl Conf on Ubiquitous Intelligence and Computing and 2014 IEEE 11th Intl Conf on Autonomic and Trusted Computing and 2014 IEEE 14th Intl Conf on Scalable Computing and Communications and Its Associated Workshops.

[41]  Kunal Pal,et al.  Development of an ultrasonic cane as a navigation aid for the blind people , 2014, 2014 International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT).

[42]  Suranga Nanayakkara,et al.  FingerReader: a wearable device to support text reading on the go , 2014, CHI Extended Abstracts.

[43]  Kalevi Hyyppä,et al.  Obstacle avoidance using haptics and a laser rangefinder , 2013, 2013 IEEE Workshop on Advanced Robotics and its Social Impacts.

[44]  Joao Fradinho Oliveira,et al.  The path force feedback belt , 2013, 2013 8th International Conference on Information Technology in Asia (CITA).

[45]  Tan Phu Vuong,et al.  Antenna Design for Tongue electrotactile assistive device for the blind and visually-impaired , 2013, 2013 7th European Conference on Antennas and Propagation (EuCAP).

[46]  B. R. Prudhvi,et al.  Silicon eyes: GPS-GSM based navigation assistant for visually impaired using capacitive touch braille keypad and smart SMS facility , 2013, 2013 World Congress on Computer and Information Technology (WCCIT).

[47]  Jizhong Xiao,et al.  A low cost outdoor assistive navigation system for blind people , 2013, 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA).

[48]  Rohan Thakker,et al.  Substitute Eyes for Blind with Navigator Using Android , 2013, 2013 Texas Instruments India Educators' Conference.

[49]  L. Dunai,et al.  3D CMOS sensor based acoustic object detection and navigation system for blind people , 2012, IECON 2012 - 38th Annual Conference on IEEE Industrial Electronics Society.

[50]  Eduardo Bayro-Corrochano,et al.  Cognitive guidance system for the blind , 2012, World Automation Congress 2012.

[51]  Mathieu Raynal,et al.  Navigation and space perception assistance for the visually impaired: The NAVIG project Assistance à la navigation et à la perception de l'espace pour les déficients visuels : le projet NAVIG , 2012 .

[52]  Ariffin Abdul Mutalib,et al.  Smart Cane: Assistive Cane for Visually-impaired People , 2011, ArXiv.

[53]  A. Duran,et al.  Electronic long cane for locomotion improving on visual impaired people. A case study , 2011, 2011 Pan American Health Care Exchanges.

[54]  Christophe Jouffrais,et al.  Fusion of Artificial Vision and GPS to Improve Blind Pedestrian Positioning , 2011, 2011 4th IFIP International Conference on New Technologies, Mobility and Security.

[55]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[56]  M. F. Saaid,et al.  Radio Frequency Identification Walking Stick (RFIWS): A device for the blind , 2009, 2009 5th International Colloquium on Signal Processing & Its Applications.

[57]  Werapon Chiracharit,et al.  Banknote and coin speaker device for blind people , 2009, 2009 11th International Conference on Advanced Communication Technology.

[58]  M. Bouzit,et al.  Evaluation of RU-Netra - Tactile Feedback Navigation System For The Visually Impaired , 2006, 2006 International Workshop on Virtual Rehabilitation.

[59]  Derek Ferguson.  Mobile .NET , 2002, Apress.