Situated evaluation for cooperative systems

This paper discusses an evaluation of the MEAD prototype, a multi-user interface generator tool particularly for use in the context of Air Traffic Control (ATC). The procedures we adopted took the form of opportunistic and informal evaluation sessions with small user groups, including Air Traffic Controllers (ATCOs). We argue that informal procedures are a powerful and cost effective method for dealing with specific evaluation issues in the context of CSCW but that wider issues are more problematic. Most notably, identifying the “validity” or otherwise of CSCW systems requires that the context of use be taken seriously, necessitating a fundamental re-appraisal of the concept of evaluation.

[1]  John Bowers,et al.  Studies in computer supported cooperative work: theory, practice and design , 1990 .

[2]  Michael B. Twidale,et al.  Redressing the balance: the advantages of informal evaluation techniques for Intelligent Learning Environments , 1993 .

[3]  Richard Bentley,et al.  An architecture for tailoring cooperative multi-user displays , 1992, CSCW '92.

[4]  James A. Landay,et al.  Inferring graphical constraints with Rockit , 1993, INTERCHI.

[5]  Dan Shapiro,et al.  CSCW: Discipline or Paradigm? A Sociological Perspective , 1991, ECSCW.

[6]  Christian Heath,et al.  Collaborative Activity and Technological Design: Task Coordination in London Underground Control Rooms , 1991, ECSCW.

[7]  Robert E. Kraut,et al.  Iterative design of video communication systems , 1992, CSCW '92.

[8]  A. Darker,et al.  TOTAL QUALITY CONTROL , 1987 .

[9]  Jonathan Grudin,et al.  Why CSCW applications fail: problems in the design and evaluationof organizational interfaces , 1988, CSCW '88.

[10]  Dan Shapiro,et al.  Harmonious Working and CSCW: Computer Technology and Air Traffic Control , 1990 .

[11]  Thomas K. Landauer,et al.  Relations between cognitive psychology and computer system design , 1990 .

[12]  Liam J. Bannon,et al.  Chapter 25 Use, design and evaluation: Steps towards an integration , 1996 .

[13]  Daniel G. Bobrow,et al.  Design for Conversation: Lessons from Cognoter , 1991, Int. J. Man Mach. Stud..

[14]  Ian Sommerville,et al.  Developing a Tool to Support Collaborative Dialogues and Graphical Representation of Ideas , 1994 .

[15]  Dan Shapiro,et al.  Faltering from ethnography to design , 1992, CSCW '92.

[16]  L. Suchman.  Plans and situated actions , 1987 .

[17]  Tom Rodden,et al.  The Designers' Notepad: Supporting and Understanding Cooperative Design , 1993, ECSCW.

[18]  Kaj Grønbæk,et al.  Cooperative Prototyping: Users and Designers in Mutual Activity , 1990, Int. J. Man Mach. Stud..

[19]  Richard Bentley,et al.  Ethnographically-informed systems design for air traffic control , 1992, CSCW '92.

[20]  Jakob Nielsen,et al.  Usability engineering at a discount , 1989 .

[21]  Martyn Hammersley,et al.  Ethnography : Principles in Practice , 1983 .

[22]  Richard Michael Bentley,et al.  Supporting multi-user interface development for cooperative systems , 1994 .