Applying AHP for Collaborative Modeling Evaluation: Experiences from a Modeling Experiment

Collaborative modeling is one of the approaches used to enhance productivity in many enterprise modeling and system development projects. Determining the success of such a collaborative effort needs an evaluation of a number of factors which affect the quality of not only the end-products – the models, but also that of other modeling artifacts: the modeling language, the modeling procedure and the support tool. Although a number of quality frameworks have been developed, few of these frameworks have received practical validation and many offer little guidance about how the evaluation is operationalized. The Collaborative Modeling Evaluation (COME) framework presented in this paper offers a holistic approach to the evaluation of the four modeling artifacts. It employs the Analytic Hierarchy Process (AHP), a well-established method from Operations Research, to score the artifacts’ quality dimensions and to aggregate the modelers’ priorities and preferences. Results from a modeling experiment demonstrate both the theoretical and practical significance of the framework.

[1]  Arthur H. J. Sale,et al.  Remarks on "A comparison between PASCAL, FORTRAN and PL/1" (With Author's Reply) , 1981, Aust. Comput. J..

[2]  Daniel L. Moody,et al.  Metrics for Evaluating the Quality of Entity Relationship Models , 1998, ER.

[3]  Gwendolyn L. Kolfschoten,et al.  Challenges in collaborative modelling: a literature review and research agenda , 2008, Int. J. Simul. Process. Model..

[4]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[5]  M. T. Escobar,et al.  Aggregation of Individual Preference Structures in Ahp-Group Decision Making , 2007 .

[6]  Reinhard Schütte,et al.  The Guidelines of Modeling - An Approach to Enhance the Quality in Information Models , 1998, ER.

[7]  Björn Niehaves,et al.  Evaluation of Conceptual Models - A Structuralist Approach , 2005, ECIS.

[8]  Robert O. Briggs,et al.  Collaboration Engineering: Designing Repeatable Processes for High-Value Collaborative Tasks , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[9]  Jan Mendling,et al.  Extending the Discussion of Model Quality: Why Clarity and Completeness may not always be enough , 2007, EMMSAD.

[10]  Yuji Iwahori,et al.  Construction of Shadow Model by Robust Features to Illumination Changes , 2013, Int. J. Softw. Innov..

[11]  John Krogstie,et al.  Process models representing knowledge for action: a revised quality framework , 2006, Eur. J. Inf. Syst..

[12]  Eric W. T. Ngai,et al.  Evaluation of knowledge management tools using AHP , 2005, Expert Syst. Appl..

[13]  S.J.B.A. Hoppenbrouwers,et al.  Navigating the Methodology Jungle - The communicative role of modelling techniques in information system development , 2005 .

[14]  Keqin Li,et al.  Organizational Patterns for Security and Dependability: From Design to Application , 2011, Int. J. Secur. Softw. Eng..

[15]  Stijn Hoppenbrouwers,et al.  Evaluating Modeling Sessions Using the Analytic Hierarchy Process , 2009, PoEM.

[16]  Theo P. van der Weide,et al.  Information modeling: The process and the required competencies of its participants , 2006, Data Knowl. Eng..

[17]  Robert O. Briggs,et al.  A theory and measurement of meeting satisfaction , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[18]  Douglas R. Vogel,et al.  Modeling with a group modeling tool: group support, model quality, and validation , 1994, 1994 Proceedings of the Twenty-Seventh Hawaii International Conference on System Sciences.

[19]  M. Conner,et al.  The Theory of Planned Behaviour , 2004 .

[20]  Frank Wolff,et al.  A Multi-Perspective Framework for Evaluating Conceptual Models in Organisational Change , 2005, ECIS.

[21]  Dimitris Plexousakis,et al.  Event Pattern Discovery in Multi-Cloud Service-Based Applications , 2015, Int. J. Syst. Serv. Oriented Eng..

[22]  I. Ajzen.  The theory of planned behavior , 1991 .

[23]  Adel Guitouni,et al.  Tentative guidelines to help choosing an appropriate MCDA method , 1998, Eur. J. Oper. Res..

[24]  Keng Siau,et al.  Evaluation of information modeling methods-a review , 1998, Proceedings of the Thirty-First Hawaii International Conference on System Sciences.

[25]  Pierre F. Tiako,et al.  Software Applications: Concepts, Methodologies, Tools, and Applications , 2009 .

[26]  Stijn Hoppenbrouwers,et al.  Assessing Collaborative Modeling Quality Based on Modeling Artifacts , 2010, EIS.

[27]  Graeme G. Shanks,et al.  What Makes a Good Data Model? A Framework for Evaluating and Improving the Quality of Entity Relationship Models , 1998, Aust. Comput. J..

[28]  Thomas L. Saaty,et al.  DECISION MAKING WITH THE ANALYTIC HIERARCHY PROCESS , 2008 .