Drawing on Design to Improve Evaluation of Computer Supported Collaborative Learning: Two Complementary Views

This chapter addresses theoretical frameworks for the evaluation of computer-supported learning environments. It outlines the characteristics and obstacles this evaluation must face with regard to projects that design learning experiences, stressing the notion that human-computer interaction is imbedded in social context that is complex and dynamic. The authors examine how scenario-based design and program theory can contribute to 701 E. Chocolate Avenue, Suite 200, Hershey PA 17033-1240, USA Tel: 717/533-8845; Fax 717/533-8661; URL-http://www.idea-group.com ITB9553 IDEA GROUP PUBLISHING This chapter appears in the book, Online Collaborative Learning: Theory and Practice, edited by Tim S. Roberts. Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 282 Nash, Richter and Allert Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. the design and evaluation of computer-supported collaborative learning (CSCL) and present a case study in which both approaches are applied. Based on the revealed complementary frameworks, a compelling approach is drafted that combines both of them. Our goal is to make CSCL designers more aware of the benefits of evaluative thinking in their work and to introduce two tangible approaches to evaluation that, when implemented as a design step, can strengthen CSCL initiatives. PROGRAM EVALUATION IN CONTEXT OF COMPUTER-SUPPORTED COLLABORATIVE LEARNING This chapter addresses theoretical frameworks of evaluating computersupported learning environments. We describe two approaches that facilitate the design for evaluation of computer-supported collaborative learning (CSCL) and then proffer ways in which the two approaches can be used together. Our perspectives on these matters are based, in part, on experiences as directors of evaluation in laboratories in the United States and Germany within the Wallenberg Global Learning Network (WGLN). Our collective experience includes work on technology and teaching reform in higher education, program evaluation theory, evaluation of CSCL and learning design. It is increasingly common for the academic community at large (and grantgiving organizations, in particular) to call for some kind of program evaluation as an integral part of proposals for the development of computer-supported learning environments. Within this call, we see a growing need for formative evaluation approaches that address the specific requirements and characteristics of computer-supported learning environments (Keil-Slawik, 1999). Keil-Slawik (1999) points out that one of the main problems with evaluating computer-supported learning environments is that some goals and opportunities just arise in the course of the development process and cannot be specified in advance. Therefore an evaluation that aims to test if the specified goals were met is not helpful. We believe that this is due to the fact that design in this context addresses illstructured and situated problems. Therefore it requires evolutionary and cyclic processes. As Weiss (1998) notes, “much evaluation is done by investigating outcomes without much attention to the paths by which they were produced” (p. 55). We argue that the evaluation processes in CSCL projects must draw out “the reality of the program rather than its illusion” (Weiss, 1998, p. 49). For investigators designing and carrying out projects at the intersection of information and communication technology (ICT) and the learning sciences, evaluation is difficult. Evaluation efforts are often subverted by a myriad of 28 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the product's webpage: www.igi-global.com/chapter/drawing-design-improveevaluation-computer/27727?camid=4v1 This title is available in InfoSci-Books, InfoSci-Educational Technologies, Virtual Learning Environments, Library Science, Information Studies, and Education, InfoSciEducation, InfoSci-Select, InfoSci-Select. Recommend this product to your librarian: www.igi-global.com/e-resources/libraryrecommendation/?id=1