A Benchmark for OCL Engine Accuracy, Determinateness, and Efficiency

The Object Constraint Language (OCL) is a central element in modeling and transformation languages like UML, MOF, and QVT. Consequently approaches for MDE (Model-Driven Engineering) depend on OCL. However, OCL is present not only in these areas influenced by the OMG but also in the Eclipse Modeling Framework (EMF). Thus the quality of OCL and its realization in tools seems to be crucial for the success of model-driven development. Surprisingly, up to now a benchmark for OCL to measure quality properties has not been proposed. This paper puts forward in the first part the concepts of a comprehensive OCL benchmark. Our benchmark covers (A) OCL engine accuracy (e.g., for the undefined value and the use of variables), (B) OCL engine determinateness properties (e.g., for the collection operations any and flatten), and (C) OCL engine efficiency (for data type and user-defined operations). In the second part, this paper empirically evaluates the proposed benchmark concepts by examining a number of OCL tools. The paper discusses several differences in handling particular OCL language features and underspecifications in the OCL standard.