Can we cover navigational perception needs of the visually impaired by panoptic segmentation?

Navigational perception for visually impaired people has been substantially promoted by both classic and deep learning based segmentation methods. In classic visual recognition methods, the segmentation models are mostly object-dependent, which means a specific algorithm has to be devised for the object of interest. In contrast, deep learning based models such as instance segmentation and semantic segmentation allow to individually recognize part of the entire scene, namely things or stuff, for blind individuals. However, both of them can not provide a holistic understanding of the surroundings for the visually impaired. Panoptic segmentation is a newly proposed visual model with the aim of unifying semantic segmentation and instance segmentation. Motivated by that, we propose to utilize panoptic segmentation as an approach to navigating visually impaired people by offering both things and stuff awareness in the proximity of the visually impaired. We demonstrate that panoptic segmentation is able to equip the visually impaired with a holistic real-world scene perception through a wearable assistive system.

[1]  Gretchen A. Stevens,et al.  Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. , 2017, The Lancet. Global health.

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Dong Liu,et al.  Real-time pedestrian crossing lights detection algorithm for the visually impaired , 2017, Multimedia Tools and Applications.

[4]  Ruiqi Cheng,et al.  A Comparative Study in Real-Time Scene Sonification for Visually Impaired People , 2020, Sensors.

[5]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Chih-Yang Lin,et al.  Content-Aware Video Analysis to Guide Visually Impaired Walking on the Street , 2019, IVIC.

[7]  Zhengcai Cao,et al.  Rapid Detection of Blind Roads and Crosswalks by Using a Lightweight Semantic Segmentation Network , 2021, IEEE Transactions on Intelligent Transportation Systems.

[8]  Peter Kontschieder,et al.  The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[9]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[10]  Luc Van Gool,et al.  Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Andrea Vedaldi,et al.  Semi-convolutional Operators for Instance Segmentation , 2018, ECCV.

[13]  Laura Giarré,et al.  Enabling independent navigation for visually impaired people through a wearable vision-based feedback system , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Luis Miguel Bergasa,et al.  Assisting the Visually Impaired: Obstacle Detection and Warning System by Acoustic Feedback , 2012, Sensors.

[15]  Jian Bai,et al.  Expanding the Detection of Traversable Area with RealSense for the Visually Impaired , 2016, Sensors.

[16]  Shiguo Lian,et al.  Small Obstacle Avoidance Based on RGB-D Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[17]  Lorenzo Porzi,et al.  Seamless Scene Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Luis Miguel Bergasa,et al.  Intersection Perception Through Real-Time Semantic Segmentation to Assist Navigation of Visually Impaired Pedestrians , 2018, 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[20]  Gabriel J. Brostow,et al.  Footprints and Free Space From a Single Color Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Anders Grunnet-Jepsen,et al.  Intel(R) RealSense(TM) Stereoscopic Depth Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Carsten Rother,et al.  Panoptic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Kyungjun Lee,et al.  Pedestrian Detection with Wearable Cameras for the Blind: A Two-way Perspective , 2020, CHI.

[25]  Ningbo Long,et al.  Unifying obstacle detection, recognition, and fusion based on millimeter wave radar and RGB-depth sensors for the visually impaired. , 2019, The Review of scientific instruments.

[26]  Yuning Jiang,et al.  SOLO: Segmenting Objects by Locations , 2020, ECCV.

[27]  Martin Lauer,et al.  An Intuitive Mobility Aid for Visually Impaired People Based on Stereo Vision , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[28]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Kaiming He,et al.  Panoptic Feature Pyramid Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Hao Chen,et al.  FCOS: Fully Convolutional One-Stage Object Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[33]  Jian Bai,et al.  Assisting the visually impaired: multitarget warning through millimeter wave radar and RGB-depth sensors , 2019, J. Electronic Imaging.

[34]  Luis Miguel Bergasa,et al.  Unifying Terrain Awareness for the Visually Impaired through Real-Time Semantic Segmentation , 2018, Sensors.