Predicting Performance of SLAM Algorithms

Among the abilities that autonomous mobile robots should exhibit, map building and localization are definitely recognized as fundamental. Consequently, countless algorithms for solving the Simultaneous Localization And Mapping (SLAM) problem have been proposed. Currently, their evaluation is performed ex post, according to outcomes obtained when running the algorithms on data collected by robots in real or simulated environments. In this paper, we present a novel method that allows the ex ante prediction of the performance of a SLAM algorithm in an unseen environment, before it is actually run. Our method collects the performance of a SLAM algorithm in a number of simulated environments, builds a model that represents the relationship between the observed performance and some geometrical features of the environments, and exploits such model to predict the performance of the algorithm in an unseen environment starting from its features.

[1]  Francesco Amigoni,et al.  Toward generalization of experimental results for autonomous robots , 2017, Robotics Auton. Syst..

[2]  Ching Y. Suen,et al.  A fast parallel algorithm for thinning digital patterns , 1984, CACM.

[3]  Noury Bouraqadi,et al.  Evaluation of Out-of-the-Box ROS 2D SLAMs for Autonomous Exploration of Unknown Indoor Environments , 2018, ICIRA.

[4]  Yuandong Tian,et al.  Building Generalizable Agents with a Realistic and Rich 3D Environment , 2018, ICLR.

[5]  Jitendra Malik,et al.  Gibson Env: Real-World Perception for Embodied Agents , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Jitendra Malik,et al.  Habitat: A Platform for Embodied AI Research , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[8]  Maria L. Gini,et al.  Good Experimental Methodologies for Robotic Mapping: A Proposal , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[9]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[10]  Francesco Amigoni,et al.  Improving Repeatability of Experiments by Automatic Evaluation of SLAM Algorithms , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[11]  Giulio Fontana,et al.  Rawseeds ground truth collection systems for indoor self-localization and mapping , 2009, Auton. Robots.

[12]  Lucas Beyer,et al.  The STRANDS Project: Long-Term Autonomy in Everyday Environments , 2016, IEEE Robotics Autom. Mag..

[13]  Javier Ruiz-del-Solar,et al.  RoboCup@Home: Analysis and results of evolving competitions for domestic and service robots , 2015, Artif. Intell..

[14]  Angel P. del Pobil,et al.  Toward Replicable and Measurable Robotics Research [From the Guest Editors] , 2015, IEEE Robotics Autom. Mag..

[15]  Domenico G. Sorrenti,et al.  Rawseeds : Building a Benchmarking Toolkit for Autonomous Robotics , 2014 .

[16]  M. Newman,et al.  Mixing patterns in networks. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[17]  Adrian Bowyer,et al.  Computing Dirichlet Tessellations , 1981, Comput. J..

[18]  Brian Yamauchi,et al.  A frontier-based approach for autonomous exploration , 1997, Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation'.

[19]  S. Balakirsky,et al.  Towards Quantitative Comparisons of Robot Algorithms : Experiences with SLAM in Simulation and Real World Systems , 2007 .

[20]  Francesco Amigoni,et al.  Extracting Structure of Buildings Using Layout Reconstruction , 2018, IAS.

[21]  Andreas Birk,et al.  Map evaluation using matched topology graphs , 2016, Auton. Robots.

[22]  Keiichi Abe,et al.  Topological structural analysis of digitized binary images by border following , 1985, Comput. Vis. Graph. Image Process..

[23]  Samuel Karlin,et al.  A First Course on Stochastic Processes , 1968 .

[24]  Wolfram Burgard,et al.  Efficient Sparse Pose Adjustment for 2D mapping , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Monica Reggiani,et al.  An insightful comparison between experiments in mobile robotics and in science , 2009, Auton. Robots.

[26]  T. Collins,et al.  Occupancy grid mapping: An empirical evaluation , 2007, 2007 Mediterranean Conference on Control & Automation.

[27]  Kirill Krinkin,et al.  2D SLAM quality evaluation methods , 2017, 2017 21st Conference of Open Innovations Association (FRUCT).

[28]  Cyrill Stachniss,et al.  On measuring the accuracy of SLAM algorithms , 2009, Auton. Robots.

[29]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[30]  Shahram Sarkani,et al.  A Framework for Predicting Future System Performance in Autonomous Unmanned Ground Vehicles , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[31]  David Portugal,et al.  An evaluation of 2D SLAM techniques available in Robot Operating System , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[32]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[33]  Wenzhe Li,et al.  Room segmentation: Survey, implementation, and analysis , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Matthias Nießner,et al.  Matterport3D: Learning from RGB-D Data in Indoor Environments , 2017, 2017 International Conference on 3D Vision (3DV).

[35]  Wolfram Burgard,et al.  A benchmark for the evaluation of RGB-D SLAM systems , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[37]  Alberto Quattrini Li,et al.  A System for Building Semantic Maps of Indoor Environments Exploiting the Concept of Building Typology , 2013, RoboCup.

[38]  Doris M. Turnage.  Simulation results for localization and mapping algorithms , 2016, 2016 Winter Simulation Conference (WSC).

[39]  Maren Bennewitz,et al.  Predicting travel time from path characteristics for wheeled robot navigation , 2017, 2017 European Conference on Mobile Robots (ECMR).

[40]  Thomas A. Funkhouser,et al.  MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments , 2017, ArXiv.