An Automatic Benchmarking System

It is standard to write unit tests to answer the question ‘does my software work?’, but it is not always common to answer questions related to performance such as ‘is my software fast?’. Existing approaches to this problem are often manually invoked, limited in scope, or require a fairly large amount of maintenance. For our machine learning library mlpack, we have developed an automated benchmarking system which is integrated into the Jenkins continuous integration tool; this allows our developers to quickly compare the performance of their implementations with earlier revisions or other libraries’ implementations. The system is flexible and easily configurable; thus it would be straightforward to deploy for other projects.