Perceived vs. Measured Quality of Conceptual Schemas: An Experimental Comparison

Frequently the behaviour of an information system is functionally correct, but it does not meet some quality criteria, such as completeness, consistency, and usability. One way to enhance the capability of an information system is to consider its conceptual model quality as well as its functional behaviour. Conceptual model quality can be defined as a set of perceivable characteristics expressed with quantifiable parameters that may be objective and/or subjective. The aim of this empirical investigation is to evaluate and compare perceived and measured quality of different conceptual model versions of the same universe of discourse. This paper describes: a) a set of metrics (clarity, simplicity, expressiveness, minimality) applied to different versions of ER conceptual schemas, b) a framework enabling a comprehensive comparison of the conceptual schemas, c) an experimentation leading to the evaluation of the same schemas by information system (IS) stakeholders such as designers, end-users, and students, d) a comparison of the objective and subjective evaluations based on a sample of about 120 observations using different statistical methods. First results indicate that there exists a strong relationship between perceived and measured quality. A second result reveals a significant difference between groups of respondents in their ways to perceive conceptual schemas quality. Based on our experiment, we are able to identify quality criteria relevant to different groups of stakeholders, depending on several dimensions, such as their professional experience, and/or their specialization degree.