Teamwork evaluation with a microworld platform

We developed a platform supporting the evaluation of teamwork. The platform logs oral communications, collaborative tool usage and simulated physical action in the work environment. The paper brings forward observations derived from a comprehensive study of maintenance teams operating simulated network infrastructures. As positive results, we highlight that the platform supports a comprehensive analysis of teamwork at macro and micro levels, including individual and team activities, conducted in quasi naturalistic settings.

[1]  Jay F. Nunamaker,et al.  The technology transition model: a key to self-sustaining and growing communities of GSS users , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[2]  G. Rolo *,et al.  Decision-making processes evaluation using two methodologies: field and simulation techniques , 2005 .

[3]  Eduardo Salas,et al.  Emerging multi-disciplinary theoretical perspectives in team cognition: an overview , 2010 .

[4]  Pietro Carlo Cacciabue,et al.  Guide to Applying Human Factors Methods , 2004, Springer London.

[5]  E. Salas,et al.  Taking stock of naturalistic decision making , 2001 .

[6]  D O'Hare,et al.  Situational awareness ability and cognitive skills training in a complex real-world task , 2007, Ergonomics.

[7]  Robert Winter,et al.  Design science research in Europe , 2008 .

[8]  Rafael A. Gonzalez,et al.  Quo Vadis, Design Science? - A Survey of Literature , 2010, DESRIST.

[9]  Annette Kluge,et al.  The effects of heuristic rule training on operator performance in a simulated process control environment , 2008, Ergonomics.

[10]  John Patrick,et al.  Process tracing of complex cognitive work tasks , 2004 .

[11]  Daniel J. Garland,et al.  Situation Awareness Analysis and Measurement , 2009 .

[12]  Anne Cleven,et al.  Design alternatives for the evaluation of design science research artifacts , 2009, DESRIST.

[13]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[14]  Barry Kirwan.  Beyond human error: taxonomies and safety science , 2008 .

[15]  J Swanson,et al.  Business Dynamics—Systems Thinking and Modeling for a Complex World , 2002, J. Oper. Res. Soc..

[16]  Lea Fleischer,et al.  The Senses Considered As Perceptual Systems , 2016 .

[17]  Marie-Eve Jobidon,et al.  Team Response to Workload Transition: The Role of Team Structure , 2006 .

[18]  Carl Gutwin,et al.  A review of groupware evaluations , 2000, Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000).

[19]  Mark S. Young,et al.  The Field Guide to Understanding Human Error , 2008 .

[20]  Janice D. Gobert,et al.  Using educational data mining to assess students’ skills at designing and conducting experiments within a complex systems microworld , 2015 .

[21]  Wayne D. Gray.  Simulated task environments: The role of high-fidelity simulations, scaled worlds, synthetic environments, and laboratory tasks in basic and applied cognitive research. , 2002 .

[22]  Dana E. Sims,et al.  Is there a “Big Five” in Teamwork? , 2005 .

[23]  Ted Nettelbeck,et al.  Investigating the construct validity associated with microworld research: A comparison of performance under different management structures across expert and non-expert naturalistic decision-making groups , 2006 .

[24]  Robert O. Briggs,et al.  On theory-driven design and deployment of collaboration systems , 2006, Int. J. Hum. Comput. Stud..

[25]  Karol G. Ross,et al.  Macrocognition , 2003, IEEE Intell. Syst..

[26]  B. Brehmer,et al.  Micro-worlds and the circular relation between people and their environment , 2005 .

[27]  Stephen M Fiore,et al.  Augmenting team cognition in human-automation teams performing in complex operational environments. , 2007, Aviation, space, and environmental medicine.

[28]  Stephen A. Stumpf,et al.  Microworlds and Simuworlds: Practice Fields for the Learning Organization , 1996 .

[29]  Pedro Antunes,et al.  Supporting experimental collaborative systems evaluation , 2011, Proceedings of the 2011 15th International Conference on Computer Supported Cooperative Work in Design (CSCWD).

[30]  Stephen M. Fiore,et al.  Toward an Understanding of Macrocognition in Teams: Predicting Processes in Complex Collaborative Contexts , 2010, Hum. Factors.

[31]  C. Sapateiro,et al.  Evaluating mobile collaborative applications support of teamwork in critical incidents response management , 2013 .

[32]  Pedro Antunes,et al.  Structuring dimensions for collaborative systems evaluation , 2012, CSUR.

[33]  Cleotilde Gonzalez,et al.  The use of microworlds to study dynamic decision making , 2005, Comput. Hum. Behav..

[34]  Fred D. Davis.  A technology acceptance model for empirically testing new end-user information systems : theory and results , 1985 .

[35]  John D. Sterman,et al.  System Dynamics: Systems Thinking and Modeling for a Complex World , 2002 .

[36]  David A. Duce,et al.  Systems, interactions, and macrotheory , 2000, TCHI.

[37]  J. Shaoul.  Human Error , 1973, Nature.

[38]  John M. Flach,et al.  Mind the Gap: A Skeptical View of Macrocognition , 2008 .

[39]  Nancy J. Cooke,et al.  Advances in Measuring Team Cognition , 2003 .

[40]  Ronald L. Boring,et al.  A prototyping environment for research on human-machine interfaces in process control use of Microsoft WPF for microworld and distributed control system development , 2014, 2014 7th International Symposium on Resilient Control Systems (ISRCS).

[41]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[42]  Mirjana Ivanovic,et al.  Matching Observed with Empirical Reality - What you see is what you get? , 2014, Fundam. Informaticae.

[43]  Sébastien Tremblay,et al.  Decision support and vulnerability to interruption in a dynamic multitasking environment , 2015, Int. J. Hum. Comput. Stud..

[44]  G. Klein,et al.  A recognition-primed decision (RPD) model of rapid decision making. , 1993 .