Towards a benchmark for Semantic Web reasoners - an analysis of the DAML ontology library

Benchmarks are one important aspect of performance evaluation. This paper concentrates on the development of a representative benchmark for Semantic Web-type ontologies. To this extent we perform a statistical analysis of available Semantic Web ontologies, in our case the DAML ontology library, and derive parameters that can be used for the generation of synthetic ontologies. These synthetic ontologies can be used as workloads in benchmarks. Naturally, performance evaluation can also be performed using a real workload, viz. a workload that is observed on a reasoner being used for normal operations. However, such workloads can usually not be applied repeatedly in a controlled manner. Therefore synthetic workloads are typically used in performance evaluations. Synthetic workloads should be a representation or model of the real workload. Hence, it is necessary to measure and characterize the workload on existing reasoners to produce meaningful synthetic workloads. This should allow us to systematically evaluate different reasoners and reasoning techniques using a benchmark to gain realistic practical comparisons of individual systems.