3-D Passive-Vision-Aided Pedestrian Dead Reckoning for Indoor Positioning

The vision-aided pedestrian dead reckoning (PDR) systems have become increasingly popular, thanks to the ubiquitous mobile phone embedded with several sensors. This is particularly important for indoor use, where other indoor positioning technologies require additional installation or body attachment of specific sensors. This paper proposes and develops a novel 3-D passive vision-aided PDR system that uses multiple surveillance cameras and smartphone-based PDR. The proposed system can continuously track users’ movement on different floors by integrating results of inertial navigation and Faster Region-Based Convolutional Neutral Network (Faster R-CNN)-based real-time pedestrian detection, while utilizing existing camera locations and embedded barometers to provide floor/height information to identify user positions in 3-D space. This novel system provides a relatively low-cost and user-friendly solution, which requires no modifications to currently available mobile devices and also the existing indoor infrastructures at many public buildings for the purpose of 3-D indoor positioning. This paper shows the case of testing the prototype in a four-floor building, where it can provide the horizontal accuracy of 0.16 m and the vertical accuracy of 0.5 m. This level of accuracy is even better than required accuracy targeted by several emergency services, including the Federal Communications Commission (FCC). This system is developed for both Android and iOS-running devices.

[1]  Youngnam Han,et al.  Improved heading estimation for smartphone-based indoor positioning systems , 2012, 2012 IEEE 23rd International Symposium on Personal, Indoor and Mobile Radio Communications - (PIMRC).

[2]  Yu-Liang Hsu,et al.  A Wearable Inertial Pedestrian Navigation System With Quaternion-Based Extended Kalman Filter for Pedestrian Localization , 2017, IEEE Sensors Journal.

[3]  T. Moore,et al.  The use of high sensitivity GPS for initialisation of a foot mounted inertial navigation system , 2012, Proceedings of the 2012 IEEE/ION Position, Location and Navigation Symposium.

[4]  Magnus Jansson,et al.  Vision-Aided Inertial Navigation Based on Ground Plane Feature Detection , 2014, IEEE/ASME Transactions on Mechatronics.

[5]  Stergios I. Roumeliotis,et al.  A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[6]  Anahid Bassiri,et al.  Vision-aided indoor pedestrian dead reckoning , 2018, 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[7]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Oliver J. Woodman,et al.  An introduction to inertial navigation , 2007 .

[9]  Robert B. McGhee,et al.  Estimation of Human Foot Motion During Normal Walking Using Inertial and Magnetic Sensor Measurements , 2012, IEEE Transactions on Instrumentation and Measurement.

[10]  H. Inooka,et al.  Non-restricted measurement of walking distance , 2000, Smc 2000 conference proceedings. 2000 ieee international conference on systems, man and cybernetics. 'cybernetics evolving to systems, humans, organizations, and their complex interactions' (cat. no.0.

[11]  Xiaogang Wang,et al.  Using Multiple Barometers to Detect the Floor Location of Smart Phones with Built-in Barometric Sensors for Indoor Positioning , 2015, Sensors.

[12]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[13]  Zhaozheng Yin,et al.  Combining passive visual cameras and active IMU sensors for persistent pedestrian tracking , 2017, J. Vis. Commun. Image Represent..

[14]  S. Javadi,et al.  People Tracking in Outdoor Environment Using Kalman Filter , 2012, 2012 Third International Conference on Intelligent Systems Modelling and Simulation.

[15]  Gordon E. Legge,et al.  Variability in the Length and Frequency of Steps of Sighted and Visually Impaired Walkers , 2005, Journal of visual impairment & blindness.

[16]  Katinka Wolter,et al.  A survey of experimental evaluation in indoor localization research , 2015, 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[17]  Kangbin Yim,et al.  An Indoor Positioning System Using Bluetooth RSSI with an Accelerometer and a Barometer on a Smartphone , 2015, 2015 10th International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA).

[18]  Peter Martini,et al.  Indoor tracking for mission critical scenarios: A survey , 2011, Pervasive Mob. Comput..

[19]  Pei-Chun Lin,et al.  Design and Implementation of a Nine-Axis Inertial Measurement Unit , 2012, IEEE/ASME Transactions on Mechatronics.

[20]  Pingqiang Zhou,et al.  Integrating low-resolution surveillance camera and smartphone inertial sensors for indoor positioning , 2018, 2018 IEEE/ION Position, Location and Navigation Symposium (PLANS).

[21]  Anastasios I. Mourikis,et al.  Estimator initialization in vision-aided inertial navigation with unknown camera-IMU calibration , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Chris Hide,et al.  Low cost vision-aided IMU for pedestrian navigation , 2010, 2010 Ubiquitous Positioning Indoor Navigation and Location Based Service.

[24]  Sebastian Tilch,et al.  Survey of optical indoor positioning systems , 2011, 2011 International Conference on Indoor Positioning and Indoor Navigation.

[25]  Qiang Shen,et al.  A Handheld Inertial Pedestrian Navigation System With Accurate Step Modes and Device Poses Recognition , 2015, IEEE Sensors Journal.

[26]  De Xiang Zhang,et al.  Video Tracking of Human with Occlusion Based on MeanShift and Kalman Filter , 2013 .

[27]  Magnus Jansson,et al.  Image-based floor segmentation in visual inertial navigation , 2013, 2013 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[28]  Marcin Grzegorzek,et al.  Multi sensor 3D indoor localisation , 2015, 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[29]  Aladine Chetouani,et al.  Indoor Pedestrian Localization With a Smartphone: A Comparison of Inertial and Vision-Based Methods , 2016, IEEE Sensors Journal.

[30]  Dirk Baumbach,et al.  Stereo-vision-aided inertial navigation for unknown indoor and outdoor environments , 2014, 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[31]  Tsung-Han Tsai,et al.  Vision based indoor positioning for intelligent buildings , 2016, 2016 2nd International Conference on Intelligent Green Building and Smart Grid (IGBSG).

[32]  G. Panahandeh,et al.  Exploiting ground plane constraints for visual-inertial navigation , 2012, Proceedings of the 2012 IEEE/ION Position, Location and Navigation Symposium.

[33]  Romit Roy Choudhury,et al.  Did you see Bob?: human localization using mobile phones , 2010, MobiCom.

[34]  Shamik Sural,et al.  Modelling, synthesis and characterisation of occlusion in videos , 2015, IET Comput. Vis..

[35]  Jian Lu,et al.  B-Loc: Scalable Floor Localization Using Barometer on Smartphone , 2014, 2014 IEEE 11th International Conference on Mobile Ad Hoc and Sensor Systems.

[36]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[37]  Tao Lin,et al.  Multiple sensors integration for pedestrian indoor navigation , 2015, 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[38]  Xiaoli Meng,et al.  Self-Contained Pedestrian Tracking During Normal Walking Using an Inertial/Magnetic Sensor Module , 2014, IEEE Transactions on Biomedical Engineering.

[39]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Tao Gu,et al.  B-Loc: Scalable Floor Localization Using Barometer on Smartphone , 2014, 2014 IEEE 11th International Conference on Mobile Ad Hoc and Sensor Systems.

[41]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Miao Zhang,et al.  Robust height tracking by proper accounting of nonlinearities in an integrated UWB/MEMS-based-IMU/baro system , 2012, Proceedings of the 2012 IEEE/ION Position, Location and Navigation Symposium.

[43]  Jaap H van Dieën,et al.  The effects of stride length and stride frequency on trunk coordination in human walking. , 2010, Gait & posture.

[44]  W. A. Clarke,et al.  Mean shift object tracking with occlusion handling , 2012 .

[45]  Guojun Dai,et al.  BarFi: Barometer-Aided Wi-Fi Floor Localization Using Crowdsourcing , 2015, 2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems.

[46]  Hui Fang,et al.  Design of a wireless assisted pedestrian dead reckoning system - the NavMote experience , 2005, IEEE Transactions on Instrumentation and Measurement.

[47]  Chris Hide,et al.  Vision-Aided IMU for Handheld Pedestrian Navigation , 2010 .

[48]  Changdon Kee,et al.  Implementation and performance analysis of smartphone-based 3D PDR system with hybrid motion and heading classifier , 2014, 2014 IEEE/ION Position, Location and Navigation Symposium - PLANS 2014.

[49]  Alonzo Kelly,et al.  A new approach to vision-aided inertial navigation , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[50]  M Bonnard,et al.  Stride variability in human gait: the effect of stride frequency and stride length. , 2003, Gait & posture.

[51]  Dariu Gavrila,et al.  Monocular Pedestrian Detection: Survey and Experiments , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Angelo M. Sabatini,et al.  A Sensor Fusion Method for Tracking Vertical Velocity and Height Based on Inertial and Barometric Altimeter Measurements , 2014, Sensors.

[53]  Cordelia Schmid,et al.  Occlusion and Motion Reasoning for Long-Term Tracking , 2014, ECCV.

[54]  Feng Zhao,et al.  A novel occlusion-adaptive multi-object tracking method for road surveillance applications , 2013, Proceedings of the 32nd Chinese Control Conference.

[55]  Mohamed F. Mokbel,et al.  Recommendations in location-based social networks: a survey , 2015, GeoInformatica.

[56]  Moustafa Youssef,et al.  No need to war-drive: unsupervised indoor localization , 2012, MobiSys '12.

[57]  Anahid Bassiri,et al.  Indoor Pedestrian Dead Reckoning Calibration by Visual Tracking and Map Information , 2018, 2018 Ubiquitous Positioning, Indoor Navigation and Location-Based Services (UPINLBS).

[58]  M. Bernardine Dias,et al.  Robust Indoor Localization on a Commercial Smart Phone , 2012, ANT/MobiWIS.

[59]  Jesus Zegarra Flores,et al.  Indoor Navigation System for the Visually Impaired Using One Inertial Measurement Unit (IMU) and Barometer to Guide in the Subway Stations and Commercial Centers , 2014, ICCHP.

[60]  Seong-Eun Kim Floor Detection Using a Barometer Sensor in a Smartphone , 2017 .

[61]  Xiaojing Song,et al.  A Kalman Filter-Integrated Optical Flow Method for Velocity Sensing of Mobile Robots , 2011, IEEE/ASME Transactions on Mechatronics.

[62]  Craig M. Hancock,et al.  Low-Cost Vision-Based Positioning System , 2018 .

[63]  Henk Luinge,et al.  Drift-free dynamic height sensor using MEMS IMU aided by MEMS pressure sensor , 2008, 2008 5th Workshop on Positioning, Navigation and Communication.

[64]  Adam C. Winstanley,et al.  Indoor location based services challenges, requirements and usability of current solutions , 2017, Comput. Sci. Rev..

[65]  Peter Brida,et al.  Pedestrian Dead Reckoning with Particle Filter for handheld smartphone , 2016, 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[66]  Henning Lenz,et al.  Fusion of Barometric Sensors, WLAN Signals and Building Information for 3--D Indoor/Campus Localization , 2006 .

[67]  Miao Zhang,et al.  Second order nonlinear uncertainty modeling in strapdown integration using MEMS IMUs , 2011, 14th International Conference on Information Fusion.

[68]  Chris Hide,et al.  Aiding Low Cost Inertial Navigation with Building Heading for Pedestrian Navigation , 2011, Journal of Navigation.

[69]  Hassen Fourati,et al.  Heterogeneous Data Fusion Algorithm for Pedestrian Navigation via Foot-Mounted Inertial Measurement Unit and Complementary Filter , 2015, IEEE Transactions on Instrumentation and Measurement.

[70]  Jay A. Farrell,et al.  Real-Time Computer Vision/DGPS-Aided Inertial Navigation System for Lane-Level Vehicle Navigation , 2012, IEEE Transactions on Intelligent Transportation Systems.

[71]  Thomas Gallagher,et al.  Using barometers to determine the height for indoor positioning , 2013, International Conference on Indoor Positioning and Indoor Navigation.

[72]  Zhaozheng Yin,et al.  Combining passive visual cameras and active IMU sensors to track cooperative people , 2015, 2015 18th International Conference on Information Fusion (Fusion).

[73]  Frank Bentley,et al.  Beyond the bar: the places where location-based services are used in the city , 2014, Personal and Ubiquitous Computing.

[74]  Anshul Kumar,et al.  Strap-down Pedestrian Dead-Reckoning system , 2011, 2011 International Conference on Indoor Positioning and Indoor Navigation.

[75]  H. Weinberg Using the ADXL202 in Pedometer and Personal Navigation Applications , 2002 .

[76]  Robert Harle,et al.  A Survey of Indoor Inertial Positioning Systems for Pedestrians , 2013, IEEE Communications Surveys & Tutorials.

[77]  Yan Zhou,et al.  Moving Human path tracking based on video surveillance in 3D indoor scenarious , 2016 .

[78]  Norbert Link,et al.  Indoor 3D position estimation using low-cost inertial sensors and marker-based video-tracking , 2010, IEEE/ION Position, Location and Navigation Symposium.

[79]  Xiaoyang Tan,et al.  Weakly supervised human body detection under arbitrary poses , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[80]  Lei Niu A Survey of Wireless Indoor Positioning Technology for Fire Emergency Routing , 2014 .

[81]  Jay A. Farrell,et al.  Inertial Navigation Aiding by Stationary Updates , 2012, IEEE Transactions on Intelligent Transportation Systems.

[82]  Lian Zhao,et al.  Synergism of INS and PDR in Self-Contained Pedestrian Tracking With a Miniature Sensor Module , 2010, IEEE Sensors Journal.

[83]  Eric Foxlin,et al.  Pedestrian tracking with shoe-mounted inertial sensors , 2005, IEEE Computer Graphics and Applications.

[84]  Peter Kazanzides,et al.  An Inertial and Optical Sensor Fusion Approach for Six Degree-of-Freedom Pose Estimation , 2015, Sensors.

[85]  Jian Lu,et al.  FTrack: Infrastructure-free floor localization via mobile phone sensing , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications.

[86]  Azeem J. Khan,et al.  Barometric phone sensors: more hype than hope! , 2014, HotMobile.

[87]  Anastasios I. Mourikis,et al.  Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera , 2013, 2013 IEEE International Conference on Robotics and Automation.

[88]  Klaus Wehrle,et al.  FootPath: Accurate map-based indoor navigation using smartphones , 2011, 2011 International Conference on Indoor Positioning and Indoor Navigation.