Supporting experimental collaborative systems evaluation

This paper proposes a framework and tool for conducting laboratory experiments with collaborative systems. The framework and tool seek to control a large number of experimental conditions, while obtaining fine-grained log data about the individual and collaborative activities. Furthermore, the proposed framework and tool allow reusing voice communication and questionnaire modules. The paper also reports a laboratory experiment where the framework and tool were used to analyze the impact of collaboration support on teams performing maintenance activities on a network infrastructure. The obtained results highlight the capacity of the framework and tool to facilitate collaborative systems evaluation.

[1]  B. Brehmer,et al.  Experiments with computer-simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study , 1993 .

[2]  E.,et al.  GROUPS : INTERACTION AND PERFORMANCE , 2001 .

[3]  Pedro Antunes,et al.  Structuring dimensions for collaborative systems evaluation , 2012, CSUR.

[4]  Wayne D. Gray.  Simulated task environments: The role of high-fidelity simulations, scaled worlds, synthetic environments, and laboratory tasks in basic and applied cognitive research. , 2002 .

[5]  Pedro Antunes,et al.  Evaluation Methods for Groupware Systems , 2007, CRIWG.

[6]  J. Funke.  Dynamic systems as tools for analysing human judgement , 2001 .

[7]  Mary Beth Rosson,et al.  Evaluating computer-supported cooperative work: models and frameworks , 2004, CSCW.

[8]  Marie-Eve Jobidon,et al.  Team Response to Workload Transition: The Role of Team Structure , 2006 .

[9]  Robert M. Davison.  An action research perspective of group support systems : how to improve meetings in Hong Kong , 1998 .

[10]  J. Schraagen,et al.  Improving Decision Making in Crisis Response through Critical Thinking Support , 2008 .

[11]  Kenneth L. Kraemer,et al.  The impact of technological support on groups: An assessment of the empirical research , 1989, Decis. Support Syst..

[12]  Starr Roxanne Hiltz,et al.  An Assessment of Group Support Systems Experimental Research: Methodology and Results , 1998, J. Manag. Inf. Syst..

[13]  Carl Gutwin,et al.  A review of groupware evaluations , 2000, Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000).

[14]  Pedro Antunes,et al.  An Emergency Response Model toward Situational Awareness Improvement , 2009 .

[15]  Pedro Antunes,et al.  A Review of CRIWG Research , 2010, CRIWG.