An Extensible Benchmarking Infrastructure for Motion Planning Algorithms

OTION planning is a key problem in robotics concerned with finding a path that satisfies a goal specification subject to constraints. In its simplest form, the solution to this problem consists of finding a path connecting two states and the only constraint is to avoid collisions. Even for this version of the motion planning problem there is no efficient solution for the general case [1]. The addition of differential constraints on robot motion or more general goal specifications make motion planning even harder. Given its complexity, most planning algorithms forego completeness and optimality for slightly weaker notions such as resolution completeness or probabilistic completeness [2] and asymptotic optimality. Sampling-based planning algorithms are the most common probabilistically complete algorithms and are widely used on many robot platforms. Within this class of algorithms, many variants have been proposed over the last 20 years, yet there is still no characterization of which algorithms are wellsuited for which classes of problems. This has motivated us to develop a benchmarking infrastructure for motion planning algorithms (see Figure 1). It consists of three main components. First, we have created an extensive benchmarking software framework that is included with the Open Motion Planning Library (OMPL), a C++ library that contains implementations of many sampling-based algorithms [3]. One can immediately compare any new planning algorithm to the 29 other planning algorithms that currently exist within OMPL. There is also much flexibility in the types of motion planning problems that can be benchmarked, as discussed in Section II-A. Second, we have defined extensible formats for storing benchmark results. The formats are fairly straightforward so that other planning libraries could easily produce compatible output. Finally, we have created an interactive, versatile visualization tool for compact presentation of collected benchmark data. The tool and underlying database facilitate the analysis of performance across benchmark problems and planners. While the three components described above emphasize generality, we have also created—as an example—a simple command line tool specifically for rigid body motion planning that takes as input a plain text description of a motion planning problem. Benchmarking sampling-based planners is non-trivial for several reasons. Since these planners rely on sampling, performance cannot be judged from a single run. Instead, benchmarks need to be run repeatedly to obtain a distribution of some