Evaluating Semantic Web Service Matchmaking Effectiveness Based on Graded Relevance

Semantic web services (SWS) promise to take service oriented computing to a new level by allowing to semi-automate time-consuming programming tasks. At the core of SWS are solutions to the problem of SWS matchmaking, i.e., the problem of comparing semantic goal descriptions with semantic offer descriptions to determine services able to fulfill a given request. Approaches to this problem have so far been evaluated based on binary relevance despite the fact that virtually all SWS matchmakers support more fine-grained levels of match. In this paper, a solution to this discrepancy is presented. A graded relevance scale for SWS matchmaking is proposed as are measures to evaluate SWS matchmakers based on such graded relevance scales. The feasibility of the approach is shown by means of a preliminary evaluation of two hybrid OWL-S matchmakers based on the proposed measures.

[1]  Francesco M. Donini,et al.  Semantic Matchmaking as Non-Monotonic Reasoning: A Description Logic Approach , 2007, J. Artif. Intell. Res..

[2]  Tetsuya Sakai,et al.  On the reliability of information retrieval metrics based on graded relevance , 2007, Inf. Process. Manag..

[3]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[4]  Stefano Mizzaro,et al.  Measuring Retrieval Effectiveness with Average Distance Measure (ADM) , 2006 .

[5]  Chris Preist.  A Conceptual Architecture for Semantic Web Services ( Extended version ) , 2004 .

[6]  Matthias Klusch,et al.  Performance of Hybrid WSML Service Matching with WSMO-MX: Preliminary Results , 2007, SMRR.

[7]  Tetsuya Sakai.  Ranking the NTCIR Systems Based on Multigrade Relevance , 2004, AIRS.

[8]  Birgitta König-Ries,et al.  On the Empirical Evaluation of Semantic Web Service Approaches: Towards Common SWS Test Collections , 2008, 2008 IEEE International Conference on Semantic Computing.

[9]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[10]  Takahiro Kawamura,et al.  Semantic Matching of Web Services Capabilities , 2002, SEMWEB.

[11]  Stefano Mizzaro,et al.  Relevance: The Whole History , 1997, J. Am. Soc. Inf. Sci..

[12]  Tetsuya Sakai.  On Penalising Late Arrival of Relevant Documents in Information Retrieval Evaluation with Graded Relevance , 2007, EVIA@NTCIR.

[13]  Stathes Hadjiefthymiades,et al.  On the Evaluation of Semantic Web Service Matchmaking Systems , 2006, 2006 European Conference on Web Services (ECOWS'06).

[14]  Abraham Bernstein,et al.  The Creation and Evaluation of iSPARQL Strategies for Matchmaking , 2008, ESWC.

[15]  Matthias Klusch,et al.  Automated semantic web service discovery with OWLS-MX , 2006, AAMAS '06.

[16]  Dieter Fensel,et al.  Automatic Location of Services , 2005, ESWC.

[17]  Kazuaki Kishida.  Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments , 2005 .

[18]  Matthine Klusch,et al.  Semantic Web Service Coordination , 2008 .

[19]  Tetsuya Sakai,et al.  New Performance Metrics Based on Multigrade Relevance: Their Application to Question Answering , 2004, NTCIR.