On computing minimal conflicts for ontology debugging

Ontology debugging is an important stage of the ontology life-cycle and supports a knowledge engineer during the ontology development and maintenance processes. Model based diagnosis is the basis of many recently suggested ontology debugging methods. The main difference between the proposed approaches is the method of computing required conflict sets, i.e. a sets of axioms such that at least one axiom of each set should be changed (removed) to make ontology coherent. Conflict set computation is, however, the most time consuming part of the debugging process. Consequently, the choice of an efficient conflict set computation method is crucial for ensuring the practical applicability of an ontology debugging approach. In this paper we evaluate and compare two popular minimal conflict computation methods: QUICKXPLAIN and SINGLE JUST. First, we analyze best and worst cases of the required number of coherency checks of both methods on a theoretical basis assuming a black-box reasoner. Then, we empirically evaluate the run-time efficiency of the algorithms both in black-box and in glass-box settings. Although both algorithms were designed to view the reasoner as a black box, the exploitation of specific knowledge about the reasoning process (glass-box) can significantly speed up the run-time performance in practical applications. Therefore, we present modifications of the original algorithms that can also exploit specific data from the reasoning process. Both a theoretical analysis of bestand worst-case complexity as well as an empirical evaluation of run-time performance show that QUICKXPLAIN is preferable over SINGLE JUST. 1 MOTIVATION With an increasing number of applications that rely on ontologies, these knowledge bases are getting larger and more complex. Thus, corresponding knowledge bases can include definitions of thousands of concepts and roles from different domains. RDF search engines like Watson [2] for instance facilitate the creation of composite ontologies that reuse the definition of concepts and roles published on the Web. Moreover, the community of ontology users is getting more heterogeneous and nowadays includes many members from various industrial and scientific fields. Hence, different faults can be easily introduced during creation and maintenance of ontologies. Recent debugging methods as described in [4, 8, 9, 11] help the user to localize, understand, and correct faults in ontologies and are already implemented in popular ontology development tools like Protégé or Swoop. All currently suggested approaches for ontology debugging aim at the automated computation of a set of changes to the ontology 1 University Klagenfurt, Austria, email: firstname.lastname@ifit.uni-klu.ac.at 2 Dortmund University of Technology, Germany, email: dietmar.jannach@u do.edu 3 http://www.co-ode.org 4 http://code.google.com/p/swoop/ that restore the coherence of its terminology (diagnosis). In order to accomplish this task efficiently, current diagnosis approaches are based on the computation of axiom subsets that define an incoherent terminology (conflict sets). Diagnosis techniques: Currently, two approaches are used for the computation of diagnoses in ontology debugging: Pinpointing [12] and Reiter’s model-based diagnosis (MBD) [10]. Pinpoints are used to avoid the computation of minimal hitting sets of conflict sets by approximating minimal diagnoses by their supersets. However, the pinpoints themselves are computed on the basis of all minimal conflicts. In contrast, in MBD approaches (minimal) conflicts are computed on demand and diagnoses are computed with increasing cardinality by constructing a hitting-set tree (HSTREE). Consequently, this method will find those diagnoses first that suggest minimal changes and avoids both the computation of very implausible multi-fault diagnoses and the costly computation of the set of all minimal conflicts. Note that Reiter’s original proposal does not work correctly for non-minimal conflicts [5] and shows limited performance for non-minimal conflicts. A modified diagnosis method was however introduced in [4] which avoids these deficits. The general question whether pinpoints or leading diagnoses are more appropriate as an output of the debugging process is still open. Conflict computation: Current approaches like SINGLE JUST[8] and QUICKXPLAIN[4] either treat the underlying reasoner as a black-box or a glass-box. In (invasive) glass-box approaches the developer of the debugging method can exploit specifics of the theorem prover. In [9], for instance, a conflict computation approach was proposed which requires modifications of existing reasoning algorithms as its aim is to compute sets of conflicts during the reasoning process as efficiently as possible. The main drawback of such glass-box approaches however is that they can be used only for a particular description logic [1], like SHOIN (D) [9]. Hence, only a particular reasoner (or even a version of a reasoner) and a particular type of logic can be used. Moreover, glass-box modifications to reasoning systems often remove existing optimizations and thus are typically slower then their non-modified analogues. In addition, glass-box approaches to conflict set minimization do not guarantee the minimality of returned conflict sets and further (black-box) minimization is required [8]. On the other hand, black-box algorithms are completely independent from the reasoning process and just use the boolean outputs of the theorem prover. These algorithms are therefore logic-independent and can exploit the full power of highly optimized reasoning methods. Still, in case of an unsatisfiable set of axioms, all the axioms are considered as a conflict since no further information is available. Conflicts are typically minimized by additional calls to a theorem prover. In order to make black-box approaches applicable in cases where theorem proving is expensive, the number of such calls must be minimized. In current systems, two main approaches for conflict set computation are used, SINGLE JUST [8] and QUICKXPLAIN [4]. In general, both of them can be used in glass-box and black-box settings. In this paper we show that QUICKXPLAIN is preferable over SINGLE JUST in both settings based on a theoretical analysis of best and worst cases and an empirical performance evaluation for a simulated average case. In addition, we propose modifications to the original algorithms to further improve the run-time performance in glass-box settings. The reminder of the paper is organized as follows, Section 2 provides theoretical study of conflict set computation methods and includes a brief description of the main algorithms as well as the analysis of extreme cases. In Section 3 we present the results of empirical evaluation of QUICKXPLAIN and SINGLE JUST in both blackand glass-box settings. The paper closes with a discussion of the future work. 2 COMPUTING MINIMAL CONFLICTS We will focus on the comparison of two popular algorithms QUICKXPLAIN [6] and SINGLE JUST [8]. The presented comparison is possible because application scenarios and strategies of these algorithms are similar. Both methods are designed to compute only one minimal conflict set per execution. The combinations of QUICKXPLAIN + HSTREE [4] and SINGLE JUST + HSTREE (also referred as ALL JUST) [8] are used to obtain a set of minimal diagnoses diagnoses or to enumerate minimal conflict sets (justifications). Therefore, QUICKXPLAIN and SINGLE JUST can be compared both theoretically and empirically. Algorithm: QUICKXPLAIN(B, C) Input: trusted knowledge B, set of axioms C Output: minimal conflict set CS (1) if isCoherent(B ∪ C) or C = ∅ return ∅; (2) AX ← getFaultyAxioms(C); (3) if AX 6= ∅ then C ← C ∩AX; (4) return computeConflict(B, B, C) function computeConflict(B, ∆, C) (5) if ∆ 6= ∅ and not isCoherent(B) then return ∅; (6) if |C| = 1 then return C; (7) int n← |C|; int k := split(n) (8) C1 ← {ax1, . . . , axk} and C2 := {axk+1, . . . , axn}; (9) CS1 ← computeConflict(B ∪ C1, C1, C2); (10) if CS1 = ∅ then C1 ← getFaultyAxioms(C1); (11) CS2 ← computeConflict(B ∪ CS1, CS1, C1); (12) return CS ← CS1 ∪ CS2; function getFaultyAxioms(C) (13) AX ← getConflictSet glassBox(); (14) if AX = ∅ then return C; (15) else return AX; Figure 1. Generalized QUICKXPLAIN algorithm QUICKXPLAIN. This algorithm (listed in Figure 1) takes two parameters as an input, B a set of axioms that are considered as correct by a knowledge engineer and C a set of axioms, which should be analyzed by the algorithm. QUICKXPLAIN follows a divide-andconquer strategy and splits the input set of axioms C into two subsets C1 and C2 on each recursive call. If the conflict set is a subset of either C1 or C2, the algorithm significantly reduces the search space. If for instance the splitting function is defined as split(n) = n/2 then the search space will be reduced in half just with one call to a reasoner. Otherwise, the algorithm re-adds some axioms ax ∈ C2 to C1. With the splitting function defined above, the algorithm will add a half of all axioms of the set C2. The choice of the splitting function is crucial since it affects the number of required coherency checks. The knowledge engineer can define a very effective splitting function for a concrete problem, e.g., if there exists some a priori knowledge about faulty axioms of an ontology. However, in the general case it is recommended to use the function that splits the set C of all axioms into two subsets of the same size since the path length from the root of the recursion tree to a leaf will contain at most log2 n nodes. Thus, if the cardinality of the searched minimal conflict set |CS| = k in the best case, i.e. when all k elements belong to a single subset C1, the number of required coherency checks is log2 n k +2k. The worst case for QUICKXPLAIN is observed when the axioms of a minimal conflict set always belong to different sets C1 and C2, i.e., if for instance a minimal conflict se