Benchmarking the Effectiveness of Associating Chains of Links for Exploratory Semantic Search

Linked Data offers an entity-based infrastructure to resolve indirect relations between resources, expressed as chains of links. If we could benchmark how effective retrieving chains of links from these sources is, we can motivate why they are a reliable addition for exploratory search interfaces. A vast number of applications could reap the benefits from encouraging insights in this field. Especially all kinds of knowledge discovery tasks related for instance to adhoc decision support and digital assistance systems. In this paper, we explain a benchmark model for evaluating the effectiveness of associating chains of links with keyword-based queries. We illustrate the benchmark model with an example case using academic library and conference metadata where we measured precision involving targeted expert users and directed it towards search effectiveness. This kind of typical semantic search engine evaluation focusing on information retrieval metrics such as precision is typically biased towards the final result only. However, in an exploratory search scenario, the dynamics of the intermediary links that could lead to potentially relevant discoveries are not to be neglected.

[1]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[2]  Heiko Paulheim,et al.  Adoption of the Linked Data Best Practices in Different Topical Domains , 2014, SEMWEB.

[3]  Rik Van de Walle,et al.  Discovering Meaningful Connections between Resources in the Web of Data , 2013, LDOW.

[4]  Fabien L. Gandon,et al.  Survey of Linked Data Based Exploration Systems , 2014, IESD@ISWC.

[5]  Rik Van de Walle,et al.  A visual workflow to explore the web of data for scholars , 2014, WWW '14 Companion.

[6]  Rik Van de Walle,et al.  COLINDA: Modeling, Representing and Using Scientific Events in the Web of Data , 2015, DeRiVE@ESWC.

[7]  Giovanni Tummarello,et al.  Searching web data: An entity retrieval and high-performance indexing model , 2012, J. Web Semant..

[8]  Enrico Motta,et al.  Reflections on five years of evaluating semantic search systems , 2010, Int. J. Metadata Semant. Ontologies.

[9]  Haofen Wang,et al.  Hermes: Data Web search on a pay-as-you-go integration infrastructure , 2009, J. Web Semant..

[10]  Liliana Cabral,et al.  Evaluating semantic web service tools using the SEALS Platform , 2010, IWEST@ISWC.

[11]  Georg Lausen,et al.  SP2Bench: A SPARQL Performance Benchmark , 2008, Semantic Web Information Management.

[12]  Rik Van de Walle,et al.  A Search Interface for Researchers to Explore Affinities in a Linked Data Knowledge Base , 2013, International Semantic Web Conference.

[13]  Rik Van de Walle,et al.  Finding and Exploring Commonalities between Researchers Using the ResXplorer , 2014, HCI.

[14]  Gary Marchionini,et al.  Report on ACM SIGIR 2006 workshop on evaluating exploratory search systems , 2006, SIGF.

[15]  Gary Marchionini,et al.  Evaluating exploratory search systems: Introduction to special topic issue of information processing and management , 2008, Inf. Process. Manag..

[16]  Rik Van de Walle,et al.  A Visual Exploration Workflow as Enabler for the Exploitation of Linked Open Data , 2014, IESD@ISWC.

[17]  Wessel Kraaij,et al.  Task based evaluation of exploratory search systems , 2006 .

[18]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[19]  Gary Marchionini,et al.  Examining the effectiveness of real-time query expansion , 2007, Inf. Process. Manag..

[20]  Abraham Bernstein,et al.  Evaluating Semantic Search Systems to Identify Future Directions of Research , 2012, ESWC.

[21]  Erik Duval,et al.  Components of a Research 2.0 Infrastructure , 2010, EC-TEL.

[22]  Daniel Schwabe,et al.  Exploration of Semi-Structured Data Sources , 2014, IESD@ISWC.