Rover Localization for Tube Pickup: Dataset, Methods and Validation for Mars Sample Return Planning

The Mars 2020 rover mission is intended to collect samples which will be stored in metal tubes and left on the surface of Mars, for possible retrieval and return to Earth by a future mission. In the proposed Mars Sample Return (MSR) campaign concept, a follow-up mission would collect the sample tubes and load them into a Mars Ascent Vehicle to be launched into orbit for subsequent transfer and return to Earth. In this work, we study the problem of autonomous tube localization and pickup by a “Fetch” rover during the MSR campaign. This is a challenging problem as, over time, the sample tubes may become partially covered by dust and sand, thereby making it difficult to recover their pose by direct visual observation. We propose an indirect approach, in which the Fetch rover localizes itself relative to a map built from Mars 2020 images. The map encodes the position of rocks that are sufficiently tall not to be affected by sand drifts. Because we are confident that tubes will remain immobile until Fetch arrives, their pose within the Mars 2020 map can be used to plan pickup maneuvers without directly observing the tubes in Fetch images. To support this approach, we present a dataset composed of 4160 images collected from two sets of stereo cameras placed at thirteen different view angles, two different heights from the ground, two distances from a tube, in five different lighting conditions, and ground-truthed with a motion capture setup. This dataset allows us to quantify the sensitivity of terrain-relative tube localization with respect to lighting conditions and camera pose.