State-of-the-Art Mobile Intelligence: Enabling Robots to Move Like Humans by Estimating Mobility with Artificial Intelligence

Mobility is a significant robotic task. It is the most important function when robotics is applied to domains such as autonomous cars, home service robots, and autonomous underwater vehicles. Despite extensive research on this topic, robots still suffer from difficulties when moving in complex environments, especially in practical applications. Therefore, the ability to have enough intelligence while moving is a key issue for the success of robots. Researchers have proposed a variety of methods and algorithms, including navigation and tracking. To help readers swiftly understand the recent advances in methodology and algorithms for robot movement, we present this survey, which provides a detailed review of the existing methods of navigation and tracking. In particular, this survey features a relation-based architecture that enables readers to easily grasp the key points of mobile intelligence. We first outline the key problems in robot systems and point out the relationship among robotics, navigation, and tracking. We then illustrate navigation using different sensors and the fusion methods and detail the state estimation and tracking models for target maneuvering. Finally, we address several issues of deep learning as well as the mobile intelligence of robots as suggested future research topics. The contributions of this survey are threefold. First, we review the literature of navigation according to the applied sensors and fusion method. Second, we detail the models for target maneuvering and the existing tracking based on estimation, such as the Kalman filter and its series developed form, according to their model-construction mechanisms: linear, nonlinear, and non-Gaussian white noise. Third, we illustrate the artificial intelligence approach—especially deep learning methods—and discuss its combination with the estimation method.

[1]  Patrick Gallinari,et al.  Deep learning for physical processes: incorporating prior scientific knowledge , 2017, ICLR.

[2]  Dong-Hwan Hwang,et al.  Vision/INS Integrated Navigation System for Poor Vision Navigation Environments , 2016, Sensors.

[3]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Mtm Marc Lambooij,et al.  Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review , 2009 .

[5]  R. Piché,et al.  Cubature-based Kalman filters for positioning , 2010, 2010 7th Workshop on Positioning, Navigation and Communication.

[6]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[7]  Kostas Daniilidis,et al.  Event-based feature tracking with probabilistic data association , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[8]  Y. Ahmet Sekercioglu,et al.  A loosely-coupled approach for metric scale estimation in monocular vision-inertial systems , 2017, 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI).

[9]  Sabbir Ahmed,et al.  Accurate UWB and IMU based indoor localization for autonomous robots , 2017, 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE).

[10]  Wei Zhou,et al.  A proposal of interaction system between visitor and collection in museum hall by iBeacon , 2015, 2015 10th International Conference on Computer Science & Education (ICCSE).

[11]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[12]  Jurgen Sieck,et al.  Location-Based Services with iBeacon Technology , 2014, 2014 2nd International Conference on Artificial Intelligence, Modelling and Simulation.

[13]  W. Gregorwich,et al.  Ultra-wideband for navigation and communications , 2001, 2001 IEEE Aerospace Conference Proceedings (Cat. No.01TH8542).

[14]  Avinash C. Kak,et al.  Vision for Mobile Robot Navigation: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Jeffrey K. Uhlmann,et al.  Unscented filtering and nonlinear estimation , 2004, Proceedings of the IEEE.

[17]  Kristin L. Wood,et al.  InnoGPS for Data-Driven Exploration of Design Opportunities and Directions: The Case of Google Driverless Car Project , 2017 .

[18]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[19]  Shi Yan,et al.  Parallel Irregular Fusion Estimation Based on Nonlinear Filter for Indoor RFID Tracking System , 2016, Int. J. Distributed Sens. Networks.

[20]  Christian Thom,et al.  Implementation of an IMU Aided Image Stacking Algorithm in a Digital Camera for Unmanned Aerial Vehicles , 2017, Sensors.

[21]  Thomas Ross Machines who think. , 1933, Science.

[22]  S. Haykin,et al.  Cubature Kalman Filters , 2009, IEEE Transactions on Automatic Control.

[23]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[24]  Aboelmagd Noureldin,et al.  GPS/INS integration utilizing dynamic neural networks for vehicular navigation , 2011, Inf. Fusion.

[25]  Yingwei Zhao Applying Time-Differenced Carrier Phase in Nondifferential GPS/IMU Tightly Coupled Navigation Systems to Improve the Positioning Performance , 2017, IEEE Transactions on Vehicular Technology.

[26]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[27]  Feipei Lai,et al.  A mobile indoor positioning system based on iBeacon technology , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[28]  Sachin Katti,et al.  SpotFi: Decimeter Level Localization Using WiFi , 2015, SIGCOMM.

[29]  Shaojie Su,et al.  Monocular Vision- and IMU-Based System for Prosthesis Pose Estimation During Total Hip Replacement Surgery , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[30]  V. Jilkov,et al.  Design and comparison of mode-set adaptive IMM algorithms for maneuvering target tracking , 1999 .

[31]  M.F. Golnaraghi,et al.  A quaternion-based orientation estimation algorithm using an inertial measurement unit , 2004, PLANS 2004. Position Location and Navigation Symposium (IEEE Cat. No.04CH37556).

[32]  Ruzena Bajcsy,et al.  Precise indoor localization using smart phones , 2010, ACM Multimedia.

[33]  Luc Van Gool,et al.  An adaptive color-based particle filter , 2003, Image Vis. Comput..

[34]  Erkan Bostanci,et al.  Sensor fusion of camera, GPS and IMU using fuzzy adaptive multiple motion models , 2015, Soft Comput..

[35]  Mamoun F. Abdel-Hafez,et al.  Estimating Vehicle State by GPS/IMU Fusion with Vehicle Dynamics , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[36]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[37]  Jonathan Kelly,et al.  Reducing drift in visual odometry by inferring sun direction using a Bayesian Convolutional Neural Network , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[38]  Ned Mohan,et al.  Design and implementation of an extended Kalman filter for the state estimation of a permanent magnet synchronous motor , 1991 .

[39]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[40]  Mamoun F. Abdel-Hafez,et al.  Non-Linear Autoregressive Delay-Dependent INS/GPS Navigation System Using Neural Networks , 2017, IEEE Sensors Journal.

[41]  Xiaoying Kong INS algorithm using quaternion model for low cost IMU , 2004, Robotics Auton. Syst..

[42]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[43]  Gerhard P. Hancke,et al.  Pose Estimation of a Mobile Robot Based on Fusion of IMU Data and Vision Data Using an Extended Kalman Filter , 2017, Sensors.

[44]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Lianhong Cai,et al.  Question detection from acoustic features using recurrent neural network with gated recurrent unit , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[46]  An Li,et al.  Transformed Unscented Kalman Filter , 2013, IEEE Transactions on Automatic Control.

[47]  M. Hoshiya,et al.  Structural Identification by Extended Kalman Filter , 1984 .

[48]  Liang Li,et al.  An innovative information fusion method with adaptive Kalman filter for integrated INS/GPS navigation of autonomous vehicles , 2018 .

[49]  P. Fearnhead,et al.  Improved particle filter for nonlinear problems , 1999 .

[50]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[51]  Veerachai Malyavej,et al.  Indoor robot localization by RSSI/IMU sensor fusion , 2013, 2013 10th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology.

[52]  Xiaoyue Zhang,et al.  A Vision/Inertia Integrated Positioning Method Using Position and Orientation Matching , 2017 .

[53]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[54]  P. Zhang,et al.  Navigation with IMU/GPS/digital compass with unscented Kalman filter , 2005, IEEE International Conference Mechatronics and Automation, 2005.

[55]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[56]  Mudhakar Srivatsa,et al.  Darnet: a deep learning solution for distracted driving detection , 2017, Middleware '17.

[57]  Fernando Seco Granja,et al.  Accurate Pedestrian Indoor Navigation by Tightly Coupling Foot-Mounted IMU and RFID Measurements , 2012, IEEE Transactions on Instrumentation and Measurement.

[58]  Junyu Dong,et al.  Monocular visual-IMU odometry using multi-channel image patch exemplars , 2017, Multimedia Tools and Applications.

[59]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[60]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[61]  Donatella Sciuto,et al.  Occupancy detection via iBeacon on Android devices for smart building management , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[62]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[63]  Kimon P. Valavanis,et al.  Evolutionary algorithm based offline/online path planner for UAV navigation , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[64]  Dai Hongde,et al.  Performance Comparison of EKF/UKF/CKF for the Tracking of Ballistic Target , 2012 .

[65]  N. Chopin A sequential particle filter method for static models , 2002 .

[66]  Margaret Mitchell,et al.  VQA: Visual Question Answering , 2015, International Journal of Computer Vision.

[67]  John J. Leonard,et al.  Optimal-State-Constraint EKF for Visual-Inertial Navigation , 2015, ISRR.

[68]  Ulrich W. Eisenecker,et al.  AI: The Tumultuous History of the Search for Artificial Intelligence , 1995 .

[69]  Jean-Bernard Hayet,et al.  Bayesian Scale Estimation for Monocular SLAM Based on Generic Object Detection for Correcting Scale Drift , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[70]  Seung-Ki Sul,et al.  Speed sensorless vector control of an induction motor using an extended Kalman filter , 1992, Conference Record of the 1992 IEEE Industry Applications Society Annual Meeting.

[71]  Huai-Rong Shao,et al.  WiFi-based indoor positioning , 2015, IEEE Communications Magazine.

[72]  T. Westerlund,et al.  Remarks on "Asymptotic behavior of the extended Kalman filter as a parameter estimator for linear systems" , 1980 .

[73]  Li Dan,et al.  Moving object tracking method based on improved lucas-kanade sparse optical flow algorithm , 2017, 2017 International Smart Cities Conference (ISC2).

[74]  Imrich Chlamtac,et al.  Indoor location tracking using RSSI readings from a single Wi-Fi access point , 2007, Wirel. Networks.

[75]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[76]  Bing Liu,et al.  Efficient Particle Swarm Optimized Particle Filter Based Improved Multiple Model Tracking Algorithm , 2017, Comput. Intell..

[77]  Alonzo Kelly,et al.  A new approach to vision-aided inertial navigation , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[78]  Arpan Pal,et al.  A Kalman Filter Based Approach to De-noise the Stereo Vision Based Pedestrian Position Estimation , 2011, 2011 UkSim 13th International Conference on Computer Modelling and Simulation.

[79]  Hoang-Duy Nguyen,et al.  Tightly-coupled INS/GPS integration with magnetic aid , 2017, 2017 2nd International Conference on Control and Robotics Engineering (ICCRE).

[80]  Dhruv Batra,et al.  Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions? , 2016, EMNLP.

[81]  Jing Shi,et al.  A comprehensive multi-factor analysis on RFID localization capability , 2011, Adv. Eng. Informatics.

[82]  Emanuele Frontoni,et al.  Feature group matching for appearance-based localization , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[83]  Pedro José Marrón,et al.  From Fast to Accurate Wireless Map Reconstruction for Human Positioning Systems , 2017, ROBOT.

[84]  Stuart A. Golden,et al.  Sensor Measurements for Wi-Fi Location with Emphasis on Time-of-Arrival Ranging , 2007, IEEE Transactions on Mobile Computing.

[85]  Akihiro Yamamoto,et al.  Visual Odometry by Multi-frame Feature Integration , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[86]  Bernt Schiele,et al.  Long-Term On-board Prediction of People in Traffic Scenes Under Uncertainty , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[87]  Xuebo Jin,et al.  Maneuvering target tracking by adaptive statistics model , 2013 .

[88]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[89]  Marcus Baum,et al.  Extended Kalman filter for extended object tracking , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[90]  Sanjay Sharma,et al.  Obstacle Avoidance Approaches for Autonomous Navigation of Unmanned Surface Vehicles , 2017, Journal of Navigation.

[91]  Nando de Freitas,et al.  The Unscented Particle Filter , 2000, NIPS.

[92]  Lennart Svensson,et al.  LIDAR-based driving path generation using fully convolutional neural networks , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[93]  Siyuan Ma,et al.  Indoor robot navigation by coupling IMU, UWB, and encode , 2016, 2016 10th International Conference on Software, Knowledge, Information Management & Applications (SKIMA).

[94]  P. Abbeel,et al.  Kalman filtering , 2020, IEEE Control Systems Magazine.

[95]  C. Ascher,et al.  Dual IMU Indoor Navigation with particle filter based map-matching on a smartphone , 2010, 2010 International Conference on Indoor Positioning and Indoor Navigation.

[96]  Robert B. McGhee,et al.  An extended Kalman filter for quaternion-based orientation estimation using MARG sensors , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[97]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[98]  Simon J. Julier,et al.  The scaled unscented transformation , 2002, Proceedings of the 2002 American Control Conference (IEEE Cat. No.CH37301).

[99]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[100]  Zhe Zhang,et al.  CAAD: Computer Architecture for Autonomous Driving , 2017, ArXiv.

[101]  Marcel van Gerven,et al.  Convolutional neural network-based encoding and decoding of visual object recognition in space and time , 2017, NeuroImage.

[102]  Yu-Wei Su,et al.  A Comparative Study of Wireless Protocols: Bluetooth, UWB, ZigBee, and Wi-Fi , 2007, IECON 2007 - 33rd Annual Conference of the IEEE Industrial Electronics Society.

[103]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[104]  Luitpold Babel Flight path planning for unmanned aerial vehicles with landmark-based visual navigation , 2014, Robotics Auton. Syst..

[105]  Yash Goyal,et al.  Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[106]  Denis Pomorski,et al.  GPS/IMU data fusion using multisensor Kalman filtering: introduction of contextual aspects , 2006, Inf. Fusion.

[107]  Shaohan Hu,et al.  DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing , 2016, WWW.

[108]  John McCarthy,et al.  A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955 , 2006, AI Mag..

[109]  Yuanxin Wu,et al.  On 'A Kalman Filter-Based Algorithm for IMU-Camera Calibration: Observability Analysis and Performance Evaluation' , 2013, ArXiv.

[110]  Eugenio Culurciello,et al.  ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation , 2016, ArXiv.

[111]  Phoebe Sengers,et al.  In-car gps navigation: engagement with and disengagement from the environment , 2008, CHI.

[112]  Roland Siegwart,et al.  Robust Real-Time Visual Odometry with a Single Camera and an IMU , 2011, BMVC.

[113]  R. Singer Estimating Optimal Tracking Filter Performance for Manned Maneuvering Targets , 1970, IEEE Transactions on Aerospace and Electronic Systems.

[114]  Fernando Seco Granja,et al.  Indoor Localization of Persons in AAL Scenarios Using an Inertial Measurement Unit (IMU) and the Signal Strength (SS) from RFID Tags , 2012, EvAAL.

[115]  Ondrej Straka,et al.  Stochastic Integration Filter , 2013, IEEE Transactions on Automatic Control.

[116]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[117]  A. M. Mousa,et al.  GPS travelling wave fault locator systems: investigation into the anomalous measurements related to lightning strikes , 1996 .

[118]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[119]  Behzad Moshiri,et al.  Maneuvering Target Tracking , 2003 .

[120]  Moshe Kam,et al.  Sensor Fusion for Mobile Robot Navigation , 1997, Proc. IEEE.

[121]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[122]  Sun Yan,et al.  Data Fusion for Indoor Mobile Robot Positioning Based on Tightly Coupled INS/UWB , 2017 .

[123]  Yoshua Bengio,et al.  Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes , 2016, ArXiv.

[124]  James R. Bergen,et al.  Visual odometry , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[125]  Elecia White,et al.  Fusion Filter Algorithm Enhancements For a MEMS GPS/IMU , 2001 .

[126]  Alberto Elfes,et al.  Using occupancy grids for mobile robot perception and navigation , 1989, Computer.

[127]  Xiangyu Zhang,et al.  ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[128]  Zhen Ding,et al.  Comparison of the unscented and cubature Kalman filters for radar tracking applications , 2012 .

[129]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[130]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[131]  Daniel Cremers,et al.  Online Photometric Calibration of Auto Exposure Video for Realtime Visual Odometry and SLAM , 2017, IEEE Robotics and Automation Letters.

[132]  Angelo M. Sabatini,et al.  Quaternion-based extended Kalman filter for determining orientation by inertial and magnetic sensing , 2006, IEEE Transactions on Biomedical Engineering.

[133]  Sebastian Thrun,et al.  Learning Metric-Topological Maps for Indoor Mobile Robot Navigation , 1998, Artif. Intell..

[134]  Hong Liu,et al.  3D action recognition using multi-temporal skeleton visualization , 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[135]  Shaojie Shen,et al.  CSI-based WiFi-inertial state estimation , 2016, 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI).

[136]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[137]  Jan Beutel,et al.  GPS-Equipped Wireless Sensor Network Node for High-Accuracy Positioning Applications , 2012, EWSN.