PETRA: Participatory Evaluation Through Redesign and Analysis

Abstract Compared with single user-computer interactions, evaluating multi-user-computer interactions is much more complex. We argue for multiplicity — of theory, method and perspective — in the evaluation of computer-supported collaborative work (CSCW). This allows us to address both theoretical concerns and practical design issues, and to incorporate the expertise and experiences of both researchers and participants. We propose the PETRA framework, incorporating a theoretically-driven evaluators' perspective to investigate the collaborative activity, and a design-based, user-focused participants' perspective to evaluate the supporting tool. Our study investigated collaborative writing, both in a face-to-face context, and supported by a computer-based group editor. In our instantiation of the PETRA framework, we used distributed cognition and a form of breakdown analysis to investigate the development of shared understanding in the two different mediating settings; and devised a rapid prototyping session (inspired by participatory design) to elicit participant reactions to and redesigns of the tool interface. Our findings show that computer-supported shared understanding develops technologically, using social coordination as a repair mechanism; and that the collaborative tool must be particularly sensitive to issues of awareness, communication, focus and ownership.

[1]  Thomas P. Moran,et al.  The workaday world as a paradigm for CSCW design , 1990, CSCW '90.

[2]  Liam J. Bannon,et al.  Chapter 25 Use, design and evaluation: Steps towards an integration , 1996 .

[3]  Jonathan Grudin,et al.  Why CSCW Applications Fail: Problems in the Design and Evaluation of Organization of Organizational Interfaces. , 1988 .

[4]  S. Kiesler,et al.  Connections: New Ways of Working in the Networked Organization. , 1992 .

[5]  E. Hutchins.  Cognition in the wild , 1995 .

[6]  Edwin Hutchins,et al.  Distributed Cognition in an Airline Cockpit , 1996 .

[7]  Stephen A. R. Scrivener,et al.  The Use of Breakdown Analysis in Synchronous CSCW System Design , 1993, ECSCW.

[8]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[9]  Yvonne Rogers,et al.  Rethinking theoretical frameworks for HCI: report on an INTERCHI '93 workshop, Amsterdam, 24–25th April, 1993 , 1994, SGCH.

[10]  Steve M. Easterbrook,et al.  Coordination breakdowns: why groupware is so difficult to design , 1995, Proceedings of the Twenty-Eighth Annual Hawaii International Conference on System Sciences.

[11]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[12]  Wanda J. Orlikowski,et al.  Learning from Notes: organizational issues in groupware implementation , 1992, CSCW '92.

[13]  Jonathan Grudin,et al.  The case against user interface consistency , 1989, CACM.

[14]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[15]  Douglas Schuler,et al.  Participatory Design: Principles and Practices , 1993 .

[16]  Lee Sproull,et al.  Connections: New Ways of Working in the Networked Organization , 1991 .

[17]  T.M. Duffy,et al.  Scenario-Based Design: Envisioning Work and Technology in System Development [Book Review] , 1996, IEEE Transactions on Professional Communication.

[18]  Carla Simone,et al.  Computational Mechanisms of Interaction for CSCW , 1993 .

[19]  Mike Sharples,et al.  Research Issues in the Study of Computer Supported Collaborative Writing , 1993 .

[20]  Kari Kuutti,et al.  Identifying potential CSCW applications by means of activity theory concepts: a case example , 1992, CSCW '92.

[21]  Yvonne Rogers,et al.  What Are Workplace Studies For? , 1995, ECSCW.

[22]  Wendy E. Mackay,et al.  Bring media spaces into the real world , 1993 .

[23]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[24]  Finn Kensing,et al.  Generating visions: future workshops and metaphorical design , 1992 .

[25]  Yvonne Rogers,et al.  Rethinking theoretical frameworks for HCI: A Review. , 1994 .

[26]  Daniel G. Bobrow,et al.  WYSIWIS revised: early experiences with multiuser interfaces , 1987, TOIS.

[27]  Christian Heath,et al.  Collaborative Activity and Technological Design: Task Coordination in London Underground Control Rooms , 1991, ECSCW.

[28]  Lucy Suchman,et al.  Understanding practice: video as a medium for reflection and design (excerpt) , 1992 .

[29]  Stephanie D. Teasley,et al.  Perspectives on socially shared cognition , 1991 .

[30]  Tom Rodden,et al.  Moving out from the control room: ethnography in system design , 1994, CSCW '94.

[31]  Susan R. Fussell,et al.  Constructing shared communicative environments , 1991, Perspectives on socially shared cognition.

[32]  Yvonne Rogers,et al.  Ghosts in the network: distributed troubleshooting in a shared working environment , 1992, CSCW '92.

[33]  Paul Dourish,et al.  Awareness and coordination in shared workspaces , 1992, CSCW '92.

[34]  R. Young.  Surrogates and mappings: two kinds of conceptual models for interactive , 1983 .

[35]  Thomas Erickson,et al.  Working with interface metaphors , 1995 .

[36]  Vicki S. Napper.  Design at work: Cooperative design of computer systems, Greenbaum, J. and Kyng, M. (Eds.). The book, Design at Work: Cooperative design of computer systems, Lawrence Erlbaum Associates, 1991, (ISBN 0-8058-611-3) is available in hardback ($49.95) and paperback ($29.95) , 1994 .

[37]  Yvonne Rogers,et al.  Distributed cognition: an alternative framework for analysing and explaining collaborative working , 1994, J. Inf. Technol..

[38]  Mike Sharples,et al.  Computer Supported Collaborative Writing , 1993, Computer Supported Cooperative Work.

[39]  Christine A . Halverson,et al.  Distributed Cognition as a Theoretical Framework for HCI: Don't Throw the Baby Out With the Bathwater — The Importance of the Cursor in Air Traffic Control , 1994 .

[40]  P. Johnson-Laird.  Mental models , 1989 .

[41]  David E. Rowley,et al.  The cognitive jogthrough: a fast-paced user interface evaluation procedure , 1992, CHI.

[42]  H. Garfinkel.  Studies in Ethnomethodology , 1968 .

[43]  Wendy E. Mackay,et al.  Bringing Media Spaces into the Real World , 1993, ECSCW.

[44]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[45]  Keith Duncan,et al.  Cognitive Engineering , 2017, Encyclopedia of GIS.