HOBBIT: A platform for benchmarking Big Linked Data

An increasing number of solutions aim to support the steady increase of the number of requirements and requests for Linked Data at scale. This plethora of solutions leads to a growing need for objective means that facilitate the selection of adequate solutions for particular use cases. We hence present HOBBIT, a distributed benchmarking platform designed for the unified execution of benchmarks for Linked Data solutions. The HOBBIT benchmarking platform is based on the FAIR principles and is the first benchmarking platform able to scale up to benchmarking real-world scenarios for Big Linked Data solutions. Our online instance of the platform has more than 300 registered users and offers more than 40 benchmarks. It has been used in eleven benchmarking challenges and for more than 13000 experiments. We give an overview of the results achieved during 2 of these challenges and point to some of the novel insights that were gained from the results of the platform. HOBBIT is open-source and available at http://github.com/hobbit-project.

[1]  Axel-Cyrille Ngonga Ngomo,et al.  GERBIL - Benchmarking Named Entity Recognition and Linking consistently , 2017, Semantic Web.

[2]  Tim Berners-Lee,et al.  Linked Data - The Story So Far , 2009, Int. J. Semantic Web Inf. Syst..

[3]  Axel-Cyrille Ngonga Ngomo,et al.  HOBBIT link discovery benchmarks at ontology matching 2017 , 2017, OM@ISWC.

[4]  Dan Roth,et al.  Design Challenges and Misconceptions in Named Entity Recognition , 2009, CoNLL.

[5]  Axel-Cyrille Ngonga Ngomo,et al.  Ensemble Learning for Named Entity Recognition , 2014, SEMWEB.

[6]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[7]  Axel-Cyrille Ngonga Ngomo,et al.  Open Knowledge Extraction Challenge 2017 , 2017, SemWebEval@ESWC.

[8]  Beth M. Sundheim,et al.  TIPSTER/MUC-5 Information Extraction System Evaluation , 1993, TIPSTER.

[9]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[10]  Christian Bizer,et al.  DBpedia spotlight: shedding light on the web of documents , 2011, I-Semantics '11.

[11]  Felix Conrads,et al.  Open Knowledge Extraction Challenge 2018 , 2018, SemWebEval@ESWC.

[12]  Axel-Cyrille Ngonga Ngomo,et al.  The Scalable Question Answering Over Linked Data (SQA) Challenge 2018 , 2018, SemWebEval@ESWC.

[13]  Elena Cabrio,et al.  Question Answering over Linked Data (QALD-5) , 2014, CLEF.

[14]  Axel-Cyrille Ngonga Ngomo,et al.  MOCHA2018: The Mighty Storage Challenge at ESWC 2018 , 2018, SemWebEval@ESWC.

[15]  Jens Lehmann,et al.  Iguana: A Generic Framework for Benchmarking the Read-Write Performance of Triple Stores , 2017, SEMWEB.

[16]  Deborah L. McGuinness,et al.  PROV-O: The PROV Ontology , 2013 .

[17]  Axel-Cyrille Ngonga Ngomo,et al.  MOCHA2017: The Mighty Storage Challenge at ESWC 2017 , 2017, SemWebEval@ESWC.

[18]  Irini Fundulaki,et al.  Instance matching benchmarks in the era of Linked Data , 2016, J. Web Semant..

[19]  Felix Conrads,et al.  Benchmarking question answering systems , 2019, Semantic Web.

[20]  Markus Nentwig,et al.  A survey of current Link Discovery frameworks , 2016, Semantic Web.

[21]  Jens Lehmann,et al.  Introduction to Linked Data and Its Lifecycle on the Web , 2013, Reasoning Web.

[22]  Jim Gray,et al.  Thousands of DebitCredit Transactions-Per-Second: Easy and Inexpensive , 2007, ArXiv.

[23]  Jens Lehmann,et al.  DBpedia SPARQL Benchmark - Performance Assessment with Real Queries on Real Data , 2011, SEMWEB.

[24]  Blesson Varghese,et al.  Plug and play bench: Simplifying big data benchmarking using containers , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[25]  Axel-Cyrille Ngonga Ngomo,et al.  MAG: A Multilingual, Knowledge-base Agnostic and Deterministic Entity Linking Approach , 2017, K-CAP.

[26]  Vincenzo Gulisano,et al.  The DEBS 2017 Grand Challenge , 2017, DEBS.

[27]  Muhammad Saleem,et al.  FEASIBLE: A Feature-Based SPARQL Benchmark Generation Framework , 2015, SEMWEB.

[28]  Cássia Trojahn,et al.  Introducing the HOBBIT platform into the ontology alignment evaluation campaign , 2018, OM@ISWC.

[29]  Christian Bizer,et al.  The Berlin SPARQL Benchmark , 2009, Int. J. Semantic Web Inf. Syst..

[30]  Heiner Stuckenschmidt,et al.  Results of the Ontology Alignment Evaluation Initiative 2007 , 2006, OM.

[31]  Wolfgang Wahlster,et al.  New Horizons for a Data-Driven Economy , 2016, Springer International Publishing.

[32]  Massimiliano Ciaramita,et al.  A framework for benchmarking entity-annotation systems , 2013, WWW.

[33]  Vincenzo Gulisano,et al.  The DEBS 2018 Grand Challenge , 2018, DEBS.

[34]  Christopher D. Manning,et al.  Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling , 2005, ACL.

[35]  Jeff Heflin,et al.  LUBM: A benchmark for OWL knowledge base systems , 2005, J. Web Semant..

[36]  Heiner Stuckenschmidt,et al.  Results of the Ontology Alignment Evaluation Initiative , 2007 .

[37]  Carlo Curino,et al.  OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases , 2013, Proc. VLDB Endow..

[38]  Muhammad Saleem,et al.  LSQ: The Linked SPARQL Queries Dataset , 2015, SEMWEB.