What signal is your inspection team sending to each other? Using a shared collaborative interface to improve shared cognition and implicit coordination in error-detection teams

Coordination theory, created in a collaborative HCI context, has long emphasized the importance of implicit coordination. Such coordination is possible through interactivity design that creates shared cognition. Teams experience implicit coordination when they achieve tacit task coordination without direct or purposeful communication of task strategies among the participants, but this is impossible to achieve without shared cognition. Implicit coordination is often more effective than explicit coordination, and teams increasingly lack sufficient time or capabilities to coordinate explicitly; therefore, methods to foster implicit coordination could result in highly practical techniques to improve task performance. Implicit coordination, despite its promise, has not received much attention in the HCI literature. A particular problem we believe has hindered the application of implicit coordination is the dearth of objective measurement of implicit coordination and the related construct shared cognition. Teams have difficulty improving, implementing, and proving the value of implicit coordination without measurement. For our theoretical foundation, we use implicit coordination theory (ICT) to explain and predict how a simple shared interface in a group error-detection task can establish formalized group memory and group awareness, which foster the shared cognition necessary for implicit coordination to occur. To test ICT, we demonstrate innovative measurement of implicit coordination and shared cognition-without intrusive perceptual survey items-by leveraging measures from signal detection theory (SDT). We use the context of teams performing heuristic data-flow diagramming (HDFD) (which is similar to the Prepare, Collect, and Repair (PCR)-based process of heuristic evaluation) to measure the improved shared cognition and implicit coordination. Namely, we measured shared cognition with SDT's measure of response bias (c); we measured implicit coordination with SDT's measure of discrimination (d'). Supporting our ICT-based predictions, groups operating with a shared interface indeed experienced more shared cognition and implicit coordination-resulting in higher discrimination, more positive response bias, and more accurate hit rates and false-alarm rates than the control groups. Our results show great promise in making similar predictions, measurements, and improvements in other individual or group inspection tasks, which are prevalent in HCI development research and practice.

[1]  Ronald E. Rice,et al.  Attribution accuracy when using anonymity in group support systems , 1997, Int. J. Hum. Comput. Stud..

[2]  Neil A. Macmillan,et al.  Detection theory: A user's guide, 2nd ed. , 2005 .

[3]  Mitsuru Ikeda,et al.  Interactions of perceptual and conceptual processing: Expertise in medical image diagnosis , 2008, Int. J. Hum. Comput. Stud..

[4]  Alan R. Dennis,et al.  Process Structuring in Electronic Brainstorming , 1996, Inf. Syst. Res..

[5]  Gordon B. Davis,et al.  Can Humans Detect Errors in Data? Impact of Base Rates, Incentives, and Goals , 1997, MIS Q..

[6]  J. Richard Hackman,et al.  The interaction of task design and group performance strategies in determining group effectiveness , 1976 .

[7]  Ranmalee Eramudugolla,et al.  A dual-process account of auditory change detection. , 2010, Journal of experimental psychology. Human perception and performance.

[8]  Jay F. Nunamaker,et al.  Using Internet-based, distributed collaborative writing tools to improve coordination and group awareness in writing teams , 2003 .

[9]  Mary Beth Rosson,et al.  Notification and awareness: synchronizing task-oriented collaborative activity , 2003, Int. J. Hum. Comput. Stud..

[10]  Michael Rosemann,et al.  Toward Improving the Relevance of Information Systems Research to Practice: The Role of Applicability Checks , 2008, MIS Q..

[11]  Paul E. Tesluk,et al.  Overcoming roadblocks to effectiveness: Incorporating management of performance barriers into models of work group effectiveness. , 1999 .

[12]  J. Richardson,et al.  Continuous Recognition Memory Tests: are the assumptions of the theory of signal detection really met? , 1994, Journal of clinical and experimental neuropsychology.

[13]  Jiajie Zhang,et al.  Applying heuristic evaluation to improve the usability of a telemedicine system. , 2006, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[14]  Joey F. George,et al.  Modern Systems Analysis and Design , 1996 .

[15]  Laku Chidambaram,et al.  Is Out of Sight, Out of Mind? An Empirical Study of Social Loafing in Technology-Supported Groups , 2005, Inf. Syst. Res..

[16]  Sandra Waller Shelton,et al.  The Effect of Experience on the Use of Irrelevant Evidence in Auditor Judgment , 1999 .

[17]  E. Salas,et al.  Reflections on shared cognition , 2001 .

[18]  Gavriel Salvendy,et al.  Effect of evaluators' cognitive style on heuristic evaluation: Field dependent and field independent evaluators , 2009, Int. J. Hum. Comput. Stud..

[19]  J. Walsh Managerial and Organizational Cognition: Notes from a Trip Down Memory Lane , 1995 .

[20]  Gilbert A. Churchill,et al.  Using the Theory of Signal Detection to Improve Ad Recognition Testing , 1986 .

[21]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[22]  Simon S. K. Lam,et al.  The Effects of Group Decision Support Systems and Task Structures on Group Communication and Decision Quality , 1997, J. Manag. Inf. Syst..

[23]  J. Valacich,et al.  Group, Sub-Group, and Nominal Group Idea Generation: New Rules for a New Media? , 1994 .

[24]  Hee Sun Park,et al.  The Effects of Shared Cognition on Group Satisfaction and Performance , 2008, Commun. Res..

[25]  Charles Pavitt,et al.  Why We Still Have to Be Reductionists About Group Memory , 2003 .

[26]  Alistair G. Sutcliffe,et al.  Heuristic evaluation of virtual reality applications , 2004, Interact. Comput..

[27]  Amnon Rapoport,et al.  Individual strategies in a market entry game , 1995 .

[28]  J. Hackman,et al.  Group tasks, group interaction process, and group performance effectiveness: A review and proposed integration , 1975 .

[29]  Amy R. Pritchett,et al.  Predicting Interactions Between Agents in Agent-Based Modeling and Simulation of Sociotechnical Systems , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[30]  David E. Clement Group Performance in a Visual Signal-Detection Task , 1973 .

[31]  Tom L. Roberts,et al.  Toward Building Self-Sustaining Groups in PCR-Based Tasks Through Implicit Coordination: The Case of Heuristic Evaluation , 2009, J. Assoc. Inf. Syst..

[32]  Armen Tashchian,et al.  Signal Detection Analysis and Advertising Recognition: An Introduction to Measurement and Interpretation Issues , 1988 .

[33]  Mark S. Ackerman,et al.  Collaborative Support for Informal Information in Collective Memory Systems , 2000, Inf. Syst. Frontiers.

[34]  Soon Heung Chang,et al.  Development of the effectiveness measure for an advanced alarm system using signal detection theory , 1997 .

[35]  Alan R. Dennis,et al.  Information exchange and use in GSS and verbal group decision making: effects of minority influence , 1997, Proceedings of the Thirtieth Hawaii International Conference on System Sciences.

[36]  Richard E. Potter,et al.  The Role of Individual Memory and Attention Processes During Electronic Brainstorming , 2004, MIS Q..

[37]  Jay F. Nunamaker,et al.  Communication Concurrency and the New Media , 1993 .

[38]  Robert E. Kraut,et al.  Visual Information as a Conversational Resource in Collaborative Physical Tasks , 2003, Hum. Comput. Interact..

[39]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[40]  Ebba Þóra Hvannberg,et al.  Heuristic evaluation: Comparing ways of finding and reporting usability problems , 2007, Interact. Comput..

[41]  C. Gibson,et al.  Team Implicit Coordination Processes: A Team Knowledge–Based Approach , 2008 .

[42]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[43]  SauerChris,et al.  The Effectiveness of Software Development Technical Reviews , 2000 .

[44]  Souren Paul,et al.  Collective memory support and cognitive-conflict group decision-making: an experimental investigation , 2004, Decis. Support Syst..

[45]  Jay F. Nunamaker,et al.  Electronic meeting systems , 1991, CACM.

[46]  S. Mohammed,et al.  Team Mental Model: Construct or Metaphor? , 1994 .

[47]  Paul D. Cheney,et al.  The Impact of Group Size and Social Presence on Small-Group Communication , 2006 .

[48]  J. Wixted,et al.  A continuous dual-process model of remember/know judgments. , 2010, Psychological review.

[49]  Jerrold H. May,et al.  Evaluating and Tuning Predictive Data Mining Models Using Receiver Operating Characteristic Curves , 2004, J. Manag. Inf. Syst..

[50]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[51]  P. Lachenbruch Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[52]  S. Mohammed,et al.  Team mental models in a team knowledge framework: expanding theory and measurement across disciplinary boundaries , 2001 .

[53]  Matthew L. Jensen,et al.  Technology Dominance in Complex Decision Making: The Case of Aided Credibility Assessment , 2010, J. Manag. Inf. Syst..

[54]  Amnon Rapoport,et al.  Coordination in market entry games with symmetric players , 1995 .

[55]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[56]  Carl Gutwin,et al.  Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration , 2001, EHCI.

[57]  Stephen C. Hayne,et al.  The effectiveness of groups recognizing patterns , 2003, Int. J. Hum. Comput. Stud..

[58]  D. Wegner,et al.  Transactive memory in close relationships. , 1991, Journal of personality and social psychology.

[59]  A. Dennis,et al.  Paradox of richness: a cognitive model of media choice , 2005, IEEE Transactions on Professional Communication.

[60]  Robert E. Ployhart,et al.  An Evaluation of Generic Teamwork Skills Training with Action Teams: Effects on Cognitive and Skill-Based Outcomes , 2005 .

[61]  H Kanis Estimating the number of usability problems. , 2011, Applied ergonomics.

[62]  K. Newell Knowledge Of Results And Motor Learning , 1976, Exercise and sport sciences reviews.

[63]  Michael A. Cusumano,et al.  Who is liable for bugs and security flaws in software? , 2004, CACM.

[64]  M. E. Shaw A comparison of individuals and small groups in the rational solution of complex problems. , 1932 .

[65]  Timothy J. Pleskac,et al.  Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. , 2010, Psychological review.

[66]  Joan H. Johnston,et al.  Decision Support for Air Warfare: Detection of Deceptive Threats , 2004 .

[67]  J. Mathieu,et al.  Scaling the quality of teammates' mental models: equifinality and normative comparisons , 2005 .

[68]  Philip H. Siegel,et al.  A signal detection theory approach to analyzing the efficiency and effectiveness of auditing to detect management fraud , 1998 .

[69]  Pierre Dillenbourg,et al.  The effects of mutual location-awareness on group coordination , 2010, Int. J. Hum. Comput. Stud..

[70]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[71]  Robert W. Zmud,et al.  Inducing Sensitivity to Deception in Order to Improve Decision Making Performance: A Field Study , 2002, MIS Q..

[72]  Wendy A. Kellogg,et al.  Social translucence: an approach to designing systems that support social processes , 2000, TCHI.

[73]  H Stanislaw,et al.  Calculation of signal detection theory measures , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[74]  Monica J. Garfield,et al.  The Adoption and Use of GSS in Project Teams: Toward More Participative Processes and Outcomes , 2003, MIS Q..

[75]  J. G. Snodgrass,et al.  Pragmatics of measuring recognition memory: applications to dementia and amnesia. , 1988, Journal of experimental psychology. General.

[76]  Jay F. Nunamaker,et al.  The Impact of Process Structure on Novice, Internet-Based, Asynchronous-Distributed Collaborative Writing Teams , 2005 .

[77]  Alistair G. Sutcliffe Heuristic Evaluation of Website Attractiveness and Usability , 2001, DSV-IS.

[78]  R D Sorkin,et al.  Signal-detection analysis of group decision making. , 2001, Psychological review.

[79]  C. Gersick Time and Transition in Work Teams: Toward a New Model of Group Development , 1988 .

[80]  J. Valacich,et al.  Computer brainstorms: More heads are better than one. , 1993 .

[81]  Chris Sauer,et al.  Technical Reviews: A Behaviorally Motivated Program of Research , 2022 .

[82]  Douglas R. Vogel,et al.  Antecedents of Coordination Effectiveness of Software Developer Dyads From Interacting Teams: An Empirical Investigation , 2009, IEEE Transactions on Engineering Management.

[83]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[84]  Leonard Adelman,et al.  Using Brunswikian theory and a longitudinal design to study how hierarchical teams adapt to increasing levels of time pressure. , 2003, Acta psychologica.

[85]  Viswanath Venkatesh,et al.  Assessing a Firm's Web Presence: A Heuristic Evaluation Procedure for the Measurement of Usability , 2002, Inf. Syst. Res..

[86]  James Miller,et al.  A cognitive-based mechanism for constructing software inspection teams , 2004, IEEE Transactions on Software Engineering.

[87]  Claude Godart,et al.  A model to support collaborative work in virtual enterprises , 2003, Data Knowl. Eng..

[88]  Evan W. Duggan,et al.  Integrating nominal group technique and joint application development for improved systems requirements determination , 2004, Inf. Manag..

[89]  Ashutosh Deshmukh,et al.  Performance analysis of filtering software using Signal Detection Theory , 2006, Decis. Support Syst..

[90]  J.F. Nunamaker,et al.  The impact of process structure on novice, virtual collaborative writing teams , 2005, IEEE Transactions on Professional Communication.

[91]  Alan R. Dennis,et al.  Group Size Effects in Electronic Brainstorming , 2008 .

[92]  Tom L. Roberts,et al.  Improving Design Artifact Reviews with Group Support Systems and an Extension of Heuristic Evaluation Techniques , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[93]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[94]  M M Ayoub,et al.  Energy Expenditure and Aerobic Fitness of Male Low Seam Coal Miners , 1983, Human factors.

[95]  T. Dickinson,et al.  Task structure, work structure, and team performance. , 1969 .

[96]  Donald A. Norman,et al.  A non-parametric analysis of recognition experiments , 1964 .

[97]  Huanping Dai,et al.  Signal Detection Analysis of the Ideal Group , 1994 .

[98]  A. Hollingshead The Rank-Order Effect in Group Decision Making , 1996 .

[99]  Gavriel Salvendy,et al.  Optimizing Heuristic Evaluation Process in E-Commerce: Use of the Taguchi Method , 2007, Int. J. Hum. Comput. Interact..

[100]  Robert W. Zmud,et al.  A Contingency Approach to Software Project Coordination , 2002, J. Manag. Inf. Syst..

[101]  Matthew L. Jensen,et al.  Effects of Automated and Participative Decision Support in Computer-Aided Credibility Assessment , 2009, J. Manag. Inf. Syst..

[102]  Stephen J. Grove,et al.  Signal Detection Theory and Sales Effectiveness , 2013 .

[103]  Roger Simnett,et al.  The effect of information selection, information processing and task complexity on predictive accuracy of auditors☆ , 1996 .

[104]  Laurie L. Levesque,et al.  Cognitive divergence and shared mental models in software development project teams , 2001 .

[105]  Adrian Parkinson,et al.  Accommodating field-dependence: a cross-over study , 2004, ITiCSE '04.

[106]  Audris Mockus,et al.  Shared Mental Models and Coordination in Large-Scale, Distributed Software Development , 2001, ICIS.

[107]  K. Weick,et al.  Collective mind in organizations: Heedful interrelating on flight decks. , 1993 .

[108]  Verlin B. Hinsz,et al.  Cognitive and Consensus Processes in Group Recognition Memory Performance , 1990 .

[109]  Blair H. Sheppard,et al.  Memory performance by decision-making groups and individuals , 1989 .

[110]  J. Grier,et al.  Nonparametric indexes for sensitivity and bias: computing formulas. , 1971, Psychological bulletin.

[111]  Leslie A. DeChurch,et al.  The cognitive underpinnings of effective teamwork: a meta-analysis. , 2010, The Journal of applied psychology.

[112]  Gwen M. Wittenbaum,et al.  Tacit coordination in anticipation of small group task completion. , 1996 .

[113]  Thomas B. Sheridan How far to commit to open-loop action: a probabilistic decision approach with analogies to signal-detection theory , 1993, IEEE Trans. Syst. Man Cybern..

[114]  Elliot E. Entin,et al.  Adaptive Team Coordination , 1999, Hum. Factors.

[115]  Alan R. Dennis,et al.  Understanding Fit and Appropriation Effects in Group Support Systems via Meta-Analysis , 2001, MIS Q..

[116]  L.W. Nolte,et al.  Signal detection theory and reconstruction algorithms-performance for images in noise , 1994, IEEE Transactions on Biomedical Engineering.

[117]  Kevin Crowston,et al.  The interdisciplinary study of coordination , 1994, CSUR.

[118]  R. E. Pastore,et al.  SIGNAL DETECTION THEORY: CONSIDERATIONS FOR GENERAL APPLICATION , 1974 .

[119]  Zülal Güngör,et al.  The usability analysis with heuristic evaluation and analytic hierarchy process , 2009 .

[120]  Monica J. Garfield,et al.  The Creative Process: The Effects of Group Memory on Individual Idea Generation , 1999, J. Manag. Inf. Syst..

[121]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[122]  J. Mathieu,et al.  The influence of shared mental models on team process and performance. , 2000, The Journal of applied psychology.