The DLib Test Suite and Metrics Working Group: Harvesting the Experience from the Digital Library Initiative

DARPA’s DLib Test Suite project was an early attempt at organizing a rigorous and wellsupported test bed to enable comparative evaluation of digital library technologies and capabilities. The test suite, part of the DLib Forum, included a diverse and heterogeneous set of resources deliberately selected to foster research in interoperability. The DLib Forum also sponsored a Metrics Working Group (MWG) to develop quantitative performance measures. The test suite went largely underutilized by the research community, and the MWG, while making significant progress, found the stated objective daunting. Clearly, much remains to be done in both the conception of effective test beds and the instrumentation to assess progress.