Situating evaluation in scenarios of use

We report on the use of scenario-based methods for evaluating collaborative systems. We describe the method, the case study where it was applied, and provide results of its efficacy in the field. The results suggest that scenario-based evaluation is effective in helping to focus evaluation efforts and in identifying the range of technical, human, organizational and other contextual factors that impact system success. The method also helps identify specific actions, for example, prescriptions for design to enhance system effectiveness. However, we found the method somewhat less useful for identifying the measurable benefits gained from a CSCW implementation, which was one of our primary goals. We discuss challenges faced applying the technique, suggest recommendations for future research, and point to implications for practice.

[1]  M. Lombard,et al.  Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability , 2002 .

[2]  Leonard J. Bass,et al.  Scenario-Based Analysis of Software Architecture , 1996, IEEE Softw..

[3]  Tom Rodden,et al.  Moving out from the control room: ethnography in system design , 1994, CSCW '94.

[4]  Mary Beth Rosson,et al.  Getting around the task-artifact cycle: how to make claims and design by scenario , 1992, TOIS.

[5]  Regine Freitag,et al.  Making Use of Scenarios for Validating Analysis and Design , 1998, IEEE Trans. Software Eng..

[6]  Thomas K. Landauer,et al.  Trouble with Computers: Usefulness, Usability, and Productivity , 1996 .

[7]  David Pinelle,et al.  A Survey of Groupware Evaluations in CSCW Proceedings , 2000 .

[8]  Richard Bentley,et al.  Situated evaluation for cooperative systems , 1994, CSCW '94.

[9]  Jean Scholtz,et al.  Evaluation for collaborative systems , 1999, CSUR.

[10]  Armin B. Cremers,et al.  The Use of Cooperation Scenarios in the Design and Evaluation of a CSCW System , 1998, IEEE Trans. Software Eng..

[11]  Wanda J. Orlikowski,et al.  Technological frames: making sense of information technology in organizations , 1994, TOIS.

[12]  Judith S. Olson,et al.  User‐centered design of collaboration technology , 1991 .

[13]  Richard Harper,et al.  Looking at ourselves: an examination of the social organisation of two research laboratories , 1992, CSCW '92.

[14]  Yvonne Rogers,et al.  Grounding blue-sky research: how can ethnography help? , 1997, INTR.

[15]  T.M. Duffy,et al.  Scenario-Based Design: Envisioning Work and Technology in System Development [Book Review] , 1996, IEEE Transactions on Professional Communication.

[16]  Paul Dourish,et al.  Process descriptions as organisational accounting devices: the dual use of workflow technologies , 2001, GROUP.

[17]  R. Kolbe,et al.  Content-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity , 1991 .

[18]  Mary Beth Rosson,et al.  Notification and awareness: synchronizing task-oriented collaborative activity , 2003, Int. J. Hum. Comput. Stud..

[19]  Carl Gutwin,et al.  Empirical development of a heuristic evaluation methodology for shared workspace groupware , 2002, CSCW '02.

[20]  Bonnie E. John.  Evaluating usability evaluation techniques , 1996, CSUR.

[21]  M. Roseman,et al.  A usability study of awareness widgets in a shared workspace groupware system , 1996, CSCW '96.

[22]  Mark S. Ackerman,et al.  The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility , 2000, Hum. Comput. Interact..

[23]  C. Geertz.  Thick Description: Towards an Interpretive Theory of Culture , 1973 .

[24]  Dan Shapiro,et al.  Book preview: The design of computer supported cooperative work and groupware systems , 1996, INTR.

[25]  Leslie P. Willcocks,et al.  Measuring organizational IS effectiveness: an overview and update of senior management perspectives , 2002, DATB.

[26]  Leysia Palen,et al.  Social, individual and technological issues for groupware calendar systems , 1999, CHI '99.

[27]  Carl Gutwin,et al.  Groupware walkthrough: adding context to groupware usability evaluation , 2002, CHI.

[28]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[29]  Gilbert Cockton,et al.  Sale must end: should discount methods be cleared off HCI's shelves? , 2002, INTR.

[30]  P. Kidwell,et al.  The trouble with computers: Usefulness, usability and productivity , 1996, IEEE Annals of the History of Computing.

[31]  John M. Carroll,et al.  Making Use: Scenario-Based Design of Human-Computer Interactions , 2000 .

[32]  Liam J. Bannon,et al.  Chapter 25 Use, design and evaluation: Steps towards an integration , 1996 .

[33]  Yvonne Rogers,et al.  PETRA: Participatory Evaluation Through Redesign and Analysis , 1995, Interact. Comput..

[34]  Yvonne Rogers,et al.  Exploring obstacles: integrating CSCW in evolving organisations , 1994, CSCW '94.

[35]  Lucy A. Suchman,et al.  Plans and Situated Actions: The Problem of Human-Machine Communication (Learning in Doing: Social, , 1987 .

[36]  Jonathan Grudin,et al.  Why CSCW applications fail: problems in the design and evaluationof organizational interfaces , 1988, CSCW '88.

[37]  Magnus Ramage,et al.  The learning way : evaluating co-operative systems , 1999 .

[38]  Carl Gutwin,et al.  A comparison of usage evaluation and inspection methods for assessing groupware usability , 2001, GROUP.

[39]  BellottiVictoria,et al.  Grounding blue-sky research , 1997 .

[40]  Mary Beth Rosson,et al.  Usability Engineering: Scenario-based Development of Human-Computer Interaction , 2001 .