A Mobile Outdoor Augmented Reality Method Combining Deep Learning Object Detection and Spatial Relationships for Geovisualization

The purpose of this study was to develop a robust, fast and markerless mobile augmented reality method for registration, geovisualization and interaction in uncontrolled outdoor environments. We propose a lightweight deep-learning-based object detection approach for mobile or embedded devices; the vision-based detection results of this approach are combined with spatial relationships by means of the host device’s built-in Global Positioning System receiver, Inertial Measurement Unit and magnetometer. Virtual objects generated based on geospatial information are precisely registered in the real world, and an interaction method based on touch gestures is implemented. The entire method is independent of the network to ensure robustness to poor signal conditions. A prototype system was developed and tested on the Wuhan University campus to evaluate the method and validate its results. The findings demonstrate that our method achieves a high detection accuracy, stable geovisualization results and interaction.

[1]  Dieter Schmalstieg,et al.  Real-Time Detection and Tracking for Augmented Reality on Mobile Phones , 2010, IEEE Transactions on Visualization and Computer Graphics.

[2]  Bernt Schiele,et al.  What Makes for Effective Detection Proposals? , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Suya You,et al.  A robust hybrid tracking system for outdoor augmented reality , 2004, IEEE Virtual Reality 2004.

[4]  Min Chen,et al.  A Spatial Lattice Model Applied for Meteorological Visualization and Analysis , 2017, ISPRS Int. J. Geo Inf..

[5]  Mark Gahegan,et al.  Geovisualization for knowledge construction and decision support , 2004, IEEE Computer Graphics and Applications.

[6]  Christian Roux,et al.  Hough transform and uncertainty handling. Application to circular object detection in ultrasound medical images , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[7]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[8]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[9]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[10]  Manuel Graña,et al.  Real-time optical markerless tracking for augmented reality applications , 2010, Journal of Real-Time Image Processing.

[11]  Koen E. A. van de Sande,et al.  Segmentation as selective search for object recognition , 2011, 2011 International Conference on Computer Vision.

[12]  Dieter Schmalstieg,et al.  Global pose estimation using multi-sensor fusion for outdoor Augmented Reality , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[13]  Margaret S. Watters.  Geovisualization: an example from the Catholme Ceremonial Complex , 2006 .

[14]  Bo Yang,et al.  Registration Combining Wide and Narrow Baseline Feature Tracking Techniques for Markerless AR Systems , 2009, Sensors.

[15]  Gregory Dudek,et al.  Learning to generalize 3D spatial relationships , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Yongtian Wang,et al.  Sensor-Aware Recognition and Tracking for Wide-Area Augmented Reality on Mobile Phones , 2015, Sensors.

[17]  Alexei A. Efros,et al.  Putting Objects in Perspective , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Thomas Deselaers,et al.  What is an object? , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  Jihad El-Sana,et al.  Shape Recognition and Pose Estimation for Mobile Augmented Reality , 2009, IEEE Transactions on Visualization and Computer Graphics.

[20]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[21]  Hirokazu Kato,et al.  Explorations in the Use of Augmented Reality for Geographic Visualization , 2002, Presence: Teleoperators & Virtual Environments.

[22]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[23]  Lin Lei,et al.  Vehicle Detection in Aerial Images Based on Region Convolutional Neural Networks and Hard Negative Example Mining , 2017, Sensors.

[24]  A. MacEachren,et al.  Visualization in Modern Cartography: Setting the Agenda , 1994 .

[25]  Nie-Jia Yau,et al.  Integrating geographical information and augmented reality techniques for mobile escape guidelines on nuclear accident sites. , 2012, Journal of environmental radioactivity.

[26]  Bernd Girod,et al.  Streaming mobile augmented reality on mobile phones , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[27]  Ronald Azuma,et al.  Performance analysis of an outdoor augmented reality tracking system that relies upon a few mobile beacons , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[28]  Tom Drummond,et al.  Going out: robust model-based tracking for outdoor augmented reality , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[29]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[30]  Katrin Grünfeld.  Integrating spatio-temporal information in environmental monitoring data--a visualization approach applied to moss data. , 2005, The Science of the total environment.

[31]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[32]  Sanja Fidler,et al.  Monocular 3D Object Detection for Autonomous Driving , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Venkataraman Sundareswaran,et al.  Model-based visual tracking for outdoor augmented reality applications , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[34]  Р Ю Чуйков,et al.  Обнаружение транспортных средств на изображениях загородных шоссе на основе метода Single shot multibox Detector , 2017 .

[35]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[36]  Takeshi Naemura,et al.  Combining deep features for object detection at various scales: finding small birds in landscape images , 2016, IPSJ Transactions on Computer Vision and Applications.

[37]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[38]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[39]  Simon J. Julier,et al.  BARS: Battlefield Augmented Reality System , 2001 .

[40]  Bernd Girod,et al.  Outdoors augmented reality on mobile phone using loxel-based visual feature organization , 2008, MIR '08.

[41]  Bruce H. Thomas,et al.  Tinmith-Metro: new outdoor techniques for creating city models with an augmented reality wearable computer , 2001, Proceedings Fifth International Symposium on Wearable Computers.

[42]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[44]  Silvio Savarese,et al.  Understanding Indoor Scenes Using 3D Geometric Phrases , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[46]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[47]  David G. Lowe,et al.  Scene modelling, recognition and tracking with invariant image features , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[48]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[49]  Lei Guo,et al.  Object Detection in Optical Remote Sensing Images Based on Weakly Supervised Learning and High-Level Feature Learning , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[50]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[51]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Sangho Lee,et al.  BoreholeAR: A mobile tablet application for effective borehole database visualization using an augmented reality technology , 2015, Comput. Geosci..

[53]  Takeo Kanade,et al.  Probabilistic modeling of local appearance and spatial relationships for object recognition , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[54]  Ronald Azuma,et al.  A motion-stabilized outdoor augmented reality system , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[55]  Rui Li,et al.  Effects of Visual Variables on the Perception of Distance in Off-Screen Landmarks: Size, Color Value, and Crispness , 2016, LBS.

[56]  Ian D. Bishop,et al.  Integration of augmented reality and GIS: A new approach to realistic landscape visualisation , 2008 .

[57]  Günther Sagl,et al.  A Visual Analytics Approach for Extracting Spatio-Temporal Urban Mobility Information from Mobile Network Traffic , 2012, ISPRS Int. J. Geo Inf..

[58]  Eric Foxlin,et al.  Pedestrian tracking with shoe-mounted inertial sensors , 2005, IEEE Computer Graphics and Applications.

[59]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[60]  Silvio Savarese,et al.  Object Detection with Geometrical Context Feedback Loop , 2010, BMVC.

[61]  Daniel A. Keim,et al.  Geovisual analytics for spatial decision support: Setting the research agenda , 2007, Int. J. Geogr. Inf. Sci..

[62]  Wei Huang,et al.  A 3D GIS-based interactive registration mechanism for outdoor augmented reality system , 2016, Expert Syst. Appl..

[63]  Florent Perronnin,et al.  High-dimensional signature compression for large-scale image classification , 2011, CVPR 2011.

[64]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[65]  Reinhold Behringer,et al.  Registration for outdoor augmented reality applications using computer vision techniques and hybrid sensors , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[66]  José Luis Lerma,et al.  Augmented reality and photogrammetry: A synergy to visualize physical and virtual city environments , 2010 .

[67]  Vineet R. Kamat,et al.  Georeferenced Registration of Construction Graphics in Mobile Outdoor Augmented Reality , 2007 .

[68]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Junqing Yu,et al.  Fast Scene Recognition and Camera Relocalisation for Wide Area Augmented Reality Systems , 2010, Sensors.

[70]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[71]  Justin Manweiler,et al.  OverLay: Practical Mobile Augmented Reality , 2015, MobiSys.

[72]  Vincent Lepetit,et al.  Polyhedral object detection and pose estimation for augmented reality applications , 2002, Proceedings of Computer Animation 2002 (CA 2002).

[73]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Kil To Chong,et al.  A Real-Time Marker-Based Visual Sensor Based on a FPGA and a Soft Core Processor , 2016, Sensors.

[75]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[76]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.