Charting the digital library evaluation domain with a semantically enhanced mining methodology

The digital library evaluation field has an evolving nature and it is characterized by a noteworthy proclivity to enfold various methodological orientations. Given the fact that the scientific literature in the specific domain is vast, researchers require tools that will exhibit either commonly acceptable practices, or areas for further investigation. In this paper, a data mining methodology is proposed to identify prominent patterns in the evaluation of digital libraries. Using Machine Learning techniques, all papers presented in the ECDL and JCDL conferences between the years 2001 and 2011 were categorized as relevant or non-relevant to the DL evaluation domain. Then, the relevant papers were semantically annotated according to the Digital Library Evaluation Ontology (DiLEO) vocabulary. The produced set of annotations was clustered to evaluation patterns for the most frequently used tools, methods and goals of the domain. Our findings highlight the expressive nature of DiLEO, place emphasis on semantic annotation as a necessary step in handling domain-centric corpora and underline the potential of the proposed methodology in the profiling of evaluation activities.