Real Time Indoor 3D Pipeline for an Advanced Sensory Substitution Device

In this paper, we present the indoor 3D pipeline of an assistive system for visually impaired people, whose goal is to scan the environment, extract information of interest and send it to the user through haptics and sounds. The particularities of indoor scenes, containing man-made objects, with many planar faces, led us to the idea of developing the 3D object recognition algorithms around a planar segmentation, based on normal vectors. The 3D pipeline starts with acquiring depth frames from a range camera and synchronized IMU data from an inertial sensor. The pre-processing stage computes normal vectors in the 3D points of the scanned environment and filters them to reduce the noise from the input data. The next stages are planar segmentation and object labeling, which divides the scene into ground, ceiling, walls and generic objects. The whole 3D pipeline works in real-time on a consumer laptop at approximately 15 fps. We describe each step of the pipeline, with the focus on the labeling stage, and present experimental results and ideas for further improvements.