Quantitative Assessment of Robot-Generated Maps

Mobile robotic mapping is now considered to be a sufficiently mature field with demonstrated successes in various domains. While much progress has been made in the development of computationally efficient and consistent mapping schemes, it is still murky, at best, on how these maps can be evaluated. We are motivated by the absence of an accepted standard for quantitatively measuring the performance of robotic mapping systems against user-defined requirements. It is our belief that the development of standardized methods for quantitatively evaluating existing robotic technologies will improve the utility of mobile robots in already established application areas, such as vacuum cleaning, robot surveillance, and bomb disposal. This approach will also enable the proliferation and acceptance of such technologies in emerging markets. This chapter summarizes our preliminary efforts by bringing together the research community towards addressing this important problem which has ramifications not only from researchers’ perspective but also from consumers’, robot manufacturers’, and developers’ viewpoints.

[1]  Kristine L. Bell,et al.  Bayesian Bounds for Parameter Estimation and Nonlinear Filtering/Tracking , 2007 .

[2]  Andrea Censi,et al.  On achievable accuracy for pose tracking , 2009, 2009 IEEE International Conference on Robotics and Automation.

[3]  Arnaud Doucet,et al.  A survey of convergence results on particle filtering methods for practitioners , 2002, IEEE Trans. Signal Process..

[4]  Andreas Birk,et al.  Determining Map Quality through an Image Similarity Metric , 2009, RoboCup.

[5]  Raj Madhavan,et al.  Multi robot mapping using force field simulation , 2007, J. Field Robotics.

[6]  D. Kendall,et al.  The Riemannian Structure of Euclidean Shape Spaces: A Novel Environment for Statistics , 1993 .

[7]  D. Mumford,et al.  Riemannian Geometries on Spaces of Plane Curves , 2003, math/0312384.

[8]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[9]  G. Dissanayake,et al.  Information Efficient 3D Visual SLAM in Unstructured Domains , 2007, 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information.

[10]  Roland Siegwart,et al.  EKF-based 3D SLAM for structured environment reconstruction , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  Sergey Ablameyko,et al.  Fast Line and Rectangle Detection by Clustering and Grouping , 1997, CAIP.

[12]  Andreas Birk,et al.  The Jacobs map analysis toolkit , 2008 .

[13]  Edward Tunstel.  Mobile robotic surveying performance for planetary surface site characterization , 2008, PerMIS.

[14]  V. Rich.  Personal communication , 1989, Nature.

[15]  A. Nuchter,et al.  Mapping of rescue environments with Kurt3D , 2005, IEEE International Safety, Security and Rescue Rototics, Workshop, 2005..

[16]  Dieter Fox,et al.  Adapting the Sample Size in Particle Filters Through KLD-Sampling , 2003, Int. J. Robotics Res..

[17]  Gamini Dissanayake,et al.  Convergence analysis for extended Kalman filter based SLAM , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[18]  Udo Frese,et al.  A Discussion of Simultaneous Localization and Mapping , 2006, Auton. Robots.

[19]  Wolfram Burgard,et al.  Information Gain-based Exploration Using Rao-Blackwellized Particle Filters , 2005, Robotics: Science and Systems.

[20]  Raj Madhavan,et al.  Performance metrics for intelligent systems (PERMIS) workshop , 2008 .

[21]  Anuj Srivastava,et al.  Riemannian Structures on Shape Spaces: A Framework for Statistical Inferences , 2006, Statistics and Analysis of Shapes.

[22]  Stergios I. Roumeliotis,et al.  Analysis and improvement of the consistency of extended Kalman filter based SLAM , 2008, 2008 IEEE International Conference on Robotics and Automation.

[23]  Wolfram Burgard,et al.  Improving Grid-based SLAM with Rao-Blackwellized Particle Filters by Adaptive Proposals and Selective Resampling , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[24]  D. H. Ballard,et al.  GENERALIZING THE HOUGH TRANSFORM TO DETECT ARBITRARY SHAPES**The research described in this report was supported in part by NIH Grant R23-HL-2153-01 and in part by the Alfred P. Sloan Foundation Grant 78-4-15. , 1987 .

[25]  Raj Madhavan,et al.  Quantitative Performance Evaluation of Navigation Solutions for Mobile Robots , 2008 .

[26]  J BeslPaul,et al.  A Method for Registration of 3-D Shapes , 1992 .

[27]  Andrea Censi,et al.  On achievable accuracy for range-finder localization , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[28]  Paul Newman,et al.  Assessing Map Quality Using Conditional Random Fields , 2007, FSR.

[29]  Stergios I. Roumeliotis,et al.  On the treatment of relative-pose measurements for mobile robot localization , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[30]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[31]  Zhou Wang,et al.  Why is image quality assessment so difficult? , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[32]  Paul Newman,et al.  ASSESSING MAP QUALITY AND ERROR CAUSATION USING CONDITIONAL RANDOM FIELDS , 2007 .

[33]  Daniel C. Asmar,et al.  Towards benchmarks for vision SLAM algorithms , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[34]  Christian Freksa,et al.  Computational Modeling of Reasoning with Mental Images: Basic Requirements * , 2006 .

[35]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[36]  Sebastian Thrun,et al.  Learning Metric-Topological Maps for Indoor Mobile Robot Navigation , 1998, Artif. Intell..

[37]  Josef Kittler,et al.  A survey of the hough transform , 1988, Comput. Vis. Graph. Image Process..

[38]  S. Balakirsky,et al.  Stable Navigation Solutions for Robots in Complex Environments , 2007, 2007 IEEE International Workshop on Safety, Security and Rescue Robotics.

[39]  Rajmohan Madhavan,et al.  Performance analysis for stable mobile robot navigation solutions , 2008, SPIE Defense + Commercial Sensing.

[40]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[41]  Mark Hedley,et al.  Fast corner detection , 1998, Image Vis. Comput..

[42]  Gamini Dissanayake,et al.  Convergence and Consistency Analysis for Extended Kalman Filter Based SLAM , 2007, IEEE Transactions on Robotics.