Empirical and Heuristic-Based Evaluation of Collaborative Modeling Systems: An Evaluation Framework

Within the field of groupware systems, collaborative modeling systems play an important role. They are useful and promising tools for many fields of application. In this paper we begin with the evaluation of the usefulness of one of these tools, SPACE-DESIGN, by applying it to a specific domain and by comparing it to a single-user tool which is shared by means of a shared window system. We have performed a study in two phases in order to evaluate usability aspects and the awareness support of both software tools. The first phase is an empirical study with users who have worked with both software tools, whilst the second is a heuristic study with some heuristic models that are well known in the context of the research work. Taking into account this study, we propose a complete evaluation framework, to be used as a theoretical lab which could be used to evaluate interactive and usability aspects of any collaborative modeling tool. This is based on a comparison strategy and proposes a specific sequence of phases: selection of collaborative modeling tools, empirical evaluation, heuristic evaluation, analysis of results, and obtaining of conclusions.

[1]  Clarence A. Ellis,et al.  Groupware: some issues and experiences , 1991, CACM.

[2]  Brad A. Myers,et al.  User interface software tools , 1995, TCHI.

[3]  Kenneth L. Kraemer,et al.  The impact of technological support on groups: An assessment of the empirical research , 1989, Decis. Support Syst..

[4]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[5]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[6]  Jesús Gallardo,et al.  Using Specifications to Build Domain-Independent Collaborative Design Environments , 2004, CDVE.

[7]  Heinz Ulrich Hoppe,et al.  Group-Oriented Modelling Tools with Heterogeneous Semantics , 2002, Intelligent Tutoring Systems.

[8]  Michelle Potts Steves,et al.  Evaluating collaborative enterprises - a workshop report , 2001, Proceedings Tenth IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. WET ICE 2001.

[9]  Carl Gutwin,et al.  Workspace Awareness in Real-Time Distributed Groupware: Framework, Widgets, and Evaluation , 1996, BCS HCI.

[10]  Jesús Gallardo,et al.  Developing Collaborative Modeling Systems Following a Model-Driven Engineering Approach , 2008, OTM Workshops.

[11]  Keith C. C. Chan,et al.  The Effect of Pairs in Program Design Tasks , 2008, IEEE Transactions on Software Engineering.

[12]  H.M. Abdel-Wahab,et al.  XTV: a framework for sharing X Window clients in remote synchronous collaboration , 1991, Proceedings of TRICOMM `91: IEEE Conference on Communications Software: Communications for Distributed Applications and Systems.

[13]  Mary Beth Rosson,et al.  Survey on user interface programming , 1992, CHI.

[14]  César A. Collazos,et al.  An ontological conceptualization approach for awareness in domain-independent collaborative modeling systems: Application to a model-driven development method , 2011, Expert Syst. Appl..

[15]  César A. Collazos,et al.  Evaluating Collaborative Learning Processes using System-based Measurement , 2007, J. Educ. Technol. Soc..

[16]  Crescencio Bravo,et al.  A framework for process-solution analysis in collaborative learning environments , 2008, Int. J. Hum. Comput. Stud..

[17]  Nikolaos Avouris,et al.  Modelling interaction during small-group synchronous problem-solving activities: The Synergo approach . , 2004 .

[18]  Carl Gutwin,et al.  Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration , 2001, EHCI.

[19]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[20]  Carl Gutwin,et al.  A Descriptive Framework of Workspace Awareness for Real-Time Groupware , 2002, Computer Supported Cooperative Work (CSCW).

[21]  Kristina Höök,et al.  Using cognitive walkthrough for evaluating a CSCW application , 1994, CHI Conference Companion.

[22]  Fabio Paternò,et al.  ConcurTaskTrees: An Engineered Notation for Task Models , 2003 .

[23]  Fabio Paternò,et al.  CTTE: Support for Developing and Analyzing Task Models for Interactive System Design , 2002, IEEE Trans. Software Eng..

[24]  George Casaday,et al.  Inspections and design reviews: framework, history and reflection , 1994 .

[25]  Jesús Gallardo,et al.  Defining Tasks, Domains and Conversational Acts in CSCW Systems: the SPACE-DESIGN Case Study , 2008, J. Univers. Comput. Sci..

[26]  Jesús Gallardo,et al.  An Ontological Approach for Developing Domain-Independent Groupware , 2007, 16th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE 2007).

[27]  Paul Dourish,et al.  Awareness and coordination in shared workspaces , 1992, CSCW '92.

[28]  Flávia Maria Santoro,et al.  The CSCW Lab for Groupware Evaluation , 2002, CRIWG.

[29]  Carl Gutwin,et al.  Groupware walkthrough: adding context to groupware usability evaluation , 2002, CHI.