Towards best practices for crowdsourcing ontology alignment benchmarks

Ontology alignment systems establish the links between ontologies that enable knowledge from various sources and domains to be used by applications in many different ways. Unfortunately, these systems are not perfect. Currently, the results of even the best-performing alignment systems need to be manually verified in order to be fully trusted. Ontology alignment researchers have turned to crowdsourcing platforms such as Amazon’s Mechanical Turk to accomplish this. However, there has been little systematic analysis of the accuracy of crowdsourcing for alignment verification and the establishment of best practices. In this work, we analyze the impact of the presentation of the context of potential matches and the way in which the question is presented to workers on the accuracy of crowdsourcing for alignment verification.

[1]  Mark A. Musen,et al.  Crowdsourcing the Verification of Relationships in Biomedical Ontologies , 2013, AMIA.

[2]  Gayo Diallo,et al.  ServOMBI at OAEI 2015 , 2015, OM.

[3]  Lukas Biewald,et al.  Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing , 2011, Human Computation.

[4]  Mark A. Musen,et al.  Ontology Quality Assurance with the Crowd , 2013, HCOMP.

[5]  Elena Paslaru Bontas Simperl,et al.  CrowdMap: Crowdsourcing Ontology Alignment with Microtasks , 2012, SEMWEB.

[6]  Jing Wang,et al.  Bonus, Disclosure, and Choice: What Motivates the Creation of High-Quality Paid Reviews? , 2012, ICIS.

[7]  Huajun Chen,et al.  The Semantic Web , 2011, Lecture Notes in Computer Science.

[8]  Matthias Thimm,et al.  Crowd Work CV: Recognition for Micro Work , 2014, SocInfo Workshops.

[9]  Jérôme Euzenat,et al.  Ontology Matching: State of the Art and Future Challenges , 2013, IEEE Transactions on Knowledge and Data Engineering.

[10]  Mark A. Musen,et al.  Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow , 2013, WebSci.

[11]  Ian Horrocks,et al.  Large-scale Interactive Ontology Matching: Algorithms and Implementation , 2012, ECAI.

[12]  Mark A. Musen,et al.  PROMPT: Algorithm and Tool for Automated Ontology Merging and Alignment , 2000, AAAI/IAAI.

[13]  Mark A. Musen,et al.  Crowdsourcing Ontology Verification , 2013, ICBO.

[14]  Michelle Cheatham,et al.  Semantic Data Integration , 2017, Handbook of Big Data Technologies.

[15]  Pascal Hitzler,et al.  Conference v2.0: An Uncertain Version of the OAEI Conference Benchmark , 2014, SEMWEB.

[16]  Thomas R. Gruber,et al.  The Role of Common Ontology in Achieving Sharable, Reusable Knowledge Bases , 1991, KR.

[17]  J. Euzenat,et al.  Ontology Matching , 2007, Springer Berlin Heidelberg.

[18]  Pascal Hitzler,et al.  The properties of property alignment , 2014, OM.

[19]  Cosmin Stroe,et al.  Interactive User Feedback in Ontology Matching Using Signature Vectors , 2012, 2012 IEEE 28th International Conference on Data Engineering.