Multi-faceted evaluation for complex, distributed activities

Computer-supported cooperative learning presents challenges for evaluation methodology: Learning events and learning outcomes are dispersed in time and space, making causal relationships difficult to identify. We are developing techniques to address these challenges including systematic sampling, collation of multiple evaluation methods and data, and the use of collaborative critical incidents. In this paper we overview and discuss this emerging methodology.

[1]  A. Hunter,et al.  Multimethod Research: A Synthesis of Styles , 1989 .

[2]  R. Bogdan.  Qualitative research for education : an introduction to theory and methods / by Robert C. Bogdan and Sari Knopp Biklen , 1997 .

[3]  J. McGrath.  Methodology matters: doing research in the behavioral and social sciences , 1995 .

[4]  Karen Ruhleder,et al.  Capturing complex, distributed activities: video-based interaction analysis as a component of workplace ethnography , 1997 .

[5]  Valerie J. Caracelli,et al.  Toward a Conceptual Framework for Mixed-Method Evaluation Designs , 1989 .

[6]  Timothy L. Nyerges,et al.  Developing and Using Interaction Coding Systems for Studying Groupware Use , 1998, Hum. Comput. Interact..

[7]  N. Denzin.  The research act: A theoretical introduction to sociological methods , 1977 .

[8]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[9]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[10]  Abbas Tashakkori,et al.  Mixed Methodology: Combining Qualitative and Quantitative Approaches , 1998 .

[11]  Mary Beth Rosson,et al.  Managing evaluation goals for training , 1995, CACM.

[12]  Michael J. Gallivan,et al.  Value in triangulation: a comparison of two approaches for combining qualitative and quantitative methods , 1997 .

[13]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[14]  M. Scriven.  The methodology of evaluation , 1966 .

[15]  Saul Greenberg,et al.  Computer-supported cooperative work and groupware , 1991 .

[16]  Judy Kay,et al.  Studying long-term system use , 1995, CACM.

[17]  Clifford A. Shaffer,et al.  Designing collaborative applications for classroom use: the LiNC project , 1998 .

[18]  S. Kennedy,et al.  Using video in the BNR usability lab , 1989, SGCH.

[19]  Robert C. Williges,et al.  A Critical Incident Evaluation Tool For Software Documentation , 1987 .

[20]  N. Denzin.  The research act in sociology: A theoretical introduction to sociological methods , 1970 .

[21]  L. Cohen,et al.  Research Methods in Education , 1980 .

[22]  A. Strauss,et al.  Grounded theory methodology: An overview. , 1994 .

[23]  Mary Beth Rosson,et al.  Participatory analysis: shared development of requirements from scenarios , 1997, CHI.

[24]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[25]  Penelope M. Sanderson,et al.  Exploratory sequential data analysis: foundations , 1994 .

[26]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[27]  Matthew B. Miles,et al.  Qualitative Data Analysis: An Expanded Sourcebook , 1994 .

[28]  F. Lau,et al.  A review on the use of action research in information systems studies , 1997 .

[29]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[30]  David Meister,et al.  Behavioral Analysis and Measurement Methods , 1985 .