2011 solutions in perception challenge performance metrics and results

The 2011 Solutions in Perception Challenge presented an international collection of teams with the opportunity to develop algorithms that could accurately detect, recognize, and locate in space an arbitrary collection of artifacts. Researchers at the National Institute of Standards and Technology (NIST) generated a series of artifacts synonymous with parts found in industrial settings, a modular fixturing system capable of accurately and precisely positioning the artifacts within a work volume, and a relative pose scoring metric to quantify an algorithm's performance. Teams were presented with training and validation data sets consisting of red-green-blue color images and 3D point cloud data of the artifacts, and the top performers achieved over 70 % accuracy in translation and pose estimation. In this paper we discuss the design of NIST's contributions, and present the teams' results from the Challenge.