Open framework for objective evaluation of crater detection algorithms with first test-field subsystem based on MOLA data

Abstract Crater Detection Algorithms (CDAs) applications range from estimation of lunar/planetary surface age to autonomous landing on planets and asteroids and advanced statistical analyses. A large amount of work on CDAs has already been published. However, problems arise when evaluation results of some new CDA have to be compared with already published evaluation results. The problem is that different authors use different test-fields, different Ground-Truth (GT) catalogues, and even different methodologies for evaluation of their CDAs. Re-implementation of already published CDAs or its evaluation environment is a time-consuming and unpractical solution to this problem. In addition, implementation details are often insufficiently described in publications. As a result, there is a need in research community to develop a framework for objective evaluation of CDAs. A scientific question is how CDAs should be evaluated so that the results are easily and reliably comparable. In attempt to solve this issue we first analyzed previously published work on CDAs. In this paper, we propose a framework for solution of the problem of objective CDA evaluation. The framework includes: (1) a definition of the measure for differences between craters; (2) test-field topography based on the 1/64° MOLA data; (3) the GT catalogue wherein each of 17,582 craters is aligned with MOLA data and confirmed with catalogues by N.G. Barlow et al. and J.F. Rodionova et al.; (4) selection of methodology for training and testing; and (5) a Free-response Receiver Operating Characteristics (F-ROC) curves as a way to measure CDA performance. The handling of possible improvements of the framework in the future is additionally addressed as a part of discussion of results. Possible extensions with additional test-field subsystems based on visual images, data sets for other planets, evaluation methodologies for CDAs developed for different purposes than cataloguing of craters, are proposed as well. The goal of the proposed framework is to contribute to the research community by establishing guidelines for objective evaluation of CDAs.

[1]  Larry H. Matthies,et al.  The Mars Exploration Rovers Descent Image Motion Estimation System , 2004, IEEE Intell. Syst..

[2]  Osamu Konishi,et al.  Data Mining System for Planetary Images - Crater Detection and Categorization , 2000 .

[3]  Brian D. Bue,et al.  Machine Detection of Martian Impact Craters From Digital Topography Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Steven P. Brumby,et al.  Automatic feature extraction for panchromatic Mars Global Surveyor Mars Orbiter camera imagery , 2002, SPIE Optics + Photonics.

[5]  E. Mjolsness,et al.  Systematic comparison of automated geological feature detection methods for impact craters , 2001 .

[6]  Yadvinder Malhi,et al.  Seasonality in CO2 and H2O flux at an eastern Amazonian rain forest , 2002 .

[7]  Goran Salamuniccar,et al.  Applying the mathematical theory of stochastic processes to lunar and planetary science: the ancient oceans on Mars case , 2004 .

[8]  Pietro Perona,et al.  Learning to Recognize Volcanoes on Venus , 1998, Machine Learning.

[9]  Jan-Peter Muller,et al.  Quantitative assessment of automated crater detection on Mars , 2004 .

[10]  Hisashi Yamamoto,et al.  Shape Recognition of Craters in Clementine Images , 1999 .

[11]  Alejandro Flores-Méndez,et al.  Crater Marking and Classification Using Computer Vision , 2003, CIARP.

[12]  L. Perugini,et al.  Craters detection via possibilistic shell clustering , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[13]  Shuichi Rokugawa,et al.  Automated detection and classification of lunar craters using multiple approaches , 2006 .

[14]  William K. Pratt,et al.  Digital Image Processing: PIKS Inside , 2001 .

[15]  Larry H. Matthies,et al.  MER-DIMES: a planetary landing application of computer vision , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[16]  G. Michael,et al.  Coordinate registration by automated crater recognition , 2003 .

[17]  Eric Mjolsness,et al.  Training of a crater detection algorithm for Mars crater imagery , 2002, Proceedings, IEEE Aerospace Conference.

[18]  Tomasz F. Stepinski,et al.  Robust System for Automated Identification of Martian Impact Craters , 2006 .

[19]  Rie Honda,et al.  Learning to Detect Small Impact Craters , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[20]  Charless C. Fowlkes,et al.  Diamond Eye: a distributed architecture for image data mining , 1999, Defense, Security, and Sensing.

[21]  Michael C. Burl,et al.  Automated detection of craters and other geological features , 2001 .

[22]  Naoko Matsumoto,et al.  Automatic Crater Recognition on Digital Terrain Model , 2005 .

[23]  Eric Mjolsness,et al.  Can Tectonic and Fluvial Structures be Used to Date a Planetary Surface , 2002 .

[24]  Rie Honda,et al.  Crater extraction and classification system for lunar images , 2000 .

[25]  James H. Roark,et al.  Buried impact craters: A topographic analysis of quasi‐circular depressions, Utopia Basin, Mars , 2005 .

[26]  A. C. Chicarro,et al.  Unravelling earth's geological history from space using impact craters , 2003 .

[27]  Werner Frei,et al.  Fast Boundary Detection: A Generalization and a New Algorithm , 1977, IEEE Transactions on Computers.

[28]  Pedro Pina,et al.  Enhancing impact crater contours to increase recognition rates , 2006, VISAPP.

[29]  Herbert Jahn.  Crater detection by linear filters representing the Hough Transform , 1994, Other Conferences.

[30]  Rie Honda,et al.  Mining of Topographic Feature from Heterogeneous Imagery and Its Application to Lunar Craters , 2002, Progress in Discovery Science.

[31]  Stanley R. Deans,et al.  Hough Transform from the Radon Transform , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  James H. Roark,et al.  Ancient lowlands on Mars , 2002 .

[33]  Larry H. Matthies,et al.  Crater detection for autonomous landing on asteroids , 2001, Image Vis. Comput..

[34]  E Mjolsness,et al.  Machine learning for science: state of the art and future prospects. , 2001, Science.

[35]  Arnaldo de Albuquerque Araújo,et al.  Using the Hough transform to detect circular forms in satellite imagery , 2001, Proceedings XIV Brazilian Symposium on Computer Graphics and Image Processing.

[36]  Larry H. Matthies,et al.  Autonomous Navigation Results from the Mars Exploration Rover (MER) Mission , 2004, ISER.

[37]  Boris A. Ivanov,et al.  IMPACT CRATER COLLAPSE , 1999 .

[38]  Pedro Pina,et al.  Automatic Recognition of Impact Craters on the Surface of Mars , 2004, ICIAR.

[39]  Sven Loncaric,et al.  Crater Detection Algorithms Based On Pixel-Difference, Separated-Pixel-Difference, Roberts, Prewitt, Sobel and Frei-Chen Gradient Edge Detectors , 2007 .

[40]  Christian Koeberl,et al.  Automatic Recognition of Crater-like Structures in Terrestrial and Planetary Images , 2005 .

[41]  Andrew E. Johnson,et al.  Field Testing of the Mars Exploration Rovers Descent Image Motion Estimation System , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[42]  Clark F. Olson,et al.  Localization of Mars rovers using descent and surface‐based image data , 2002 .

[43]  William M. Farrell,et al.  MARSIS radar sounder evidence of buried basins in the northern lowlands of Mars , 2006, Nature.

[44]  Gordon R. J. Cooper,et al.  The detection of circular features in irregularly spaced data , 2004, Comput. Geosci..

[45]  J. Muller,et al.  Automated crater detection, a new tool for Mars cartography and chronology , 2005 .

[46]  P. G. Marchetti,et al.  Recognition and Detection of Impact Craters from EO Products , 2004 .