This chapter addresses theoretical frameworks for the evaluation of computer-supported learning environments. It outlines the characteristics and obstacles this evaluation must face with regard to projects that design learning experiences, stressing the notion that human-computer interaction is imbedded in social context that is complex and dynamic. The authors examine how scenario-based design and program theory can contribute to 701 E. Chocolate Avenue, Suite 200, Hershey PA 17033-1240, USA Tel: 717/533-8845; Fax 717/533-8661; URL-http://www.idea-group.com ITB9553 IDEA GROUP PUBLISHING This chapter appears in the book, Online Collaborative Learning: Theory and Practice, edited by Tim S. Roberts. Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 282 Nash, Richter and Allert Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. the design and evaluation of computer-supported collaborative learning (CSCL) and present a case study in which both approaches are applied. Based on the revealed complementary frameworks, a compelling approach is drafted that combines both of them. Our goal is to make CSCL designers more aware of the benefits of evaluative thinking in their work and to introduce two tangible approaches to evaluation that, when implemented as a design step, can strengthen CSCL initiatives. PROGRAM EVALUATION IN CONTEXT OF COMPUTER-SUPPORTED COLLABORATIVE LEARNING This chapter addresses theoretical frameworks of evaluating computersupported learning environments. We describe two approaches that facilitate the design for evaluation of computer-supported collaborative learning (CSCL) and then proffer ways in which the two approaches can be used together. Our perspectives on these matters are based, in part, on experiences as directors of evaluation in laboratories in the United States and Germany within the Wallenberg Global Learning Network (WGLN). Our collective experience includes work on technology and teaching reform in higher education, program evaluation theory, evaluation of CSCL and learning design. It is increasingly common for the academic community at large (and grantgiving organizations, in particular) to call for some kind of program evaluation as an integral part of proposals for the development of computer-supported learning environments. Within this call, we see a growing need for formative evaluation approaches that address the specific requirements and characteristics of computer-supported learning environments (Keil-Slawik, 1999). Keil-Slawik (1999) points out that one of the main problems with evaluating computer-supported learning environments is that some goals and opportunities just arise in the course of the development process and cannot be specified in advance. Therefore an evaluation that aims to test if the specified goals were met is not helpful. We believe that this is due to the fact that design in this context addresses illstructured and situated problems. Therefore it requires evolutionary and cyclic processes. As Weiss (1998) notes, “much evaluation is done by investigating outcomes without much attention to the paths by which they were produced” (p. 55). We argue that the evaluation processes in CSCL projects must draw out “the reality of the program rather than its illusion” (Weiss, 1998, p. 49). For investigators designing and carrying out projects at the intersection of information and communication technology (ICT) and the learning sciences, evaluation is difficult. Evaluation efforts are often subverted by a myriad of 28 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the product's webpage: www.igi-global.com/chapter/drawing-design-improveevaluation-computer/27727?camid=4v1 This title is available in InfoSci-Books, InfoSci-Educational Technologies, Virtual Learning Environments, Library Science, Information Studies, and Education, InfoSciEducation, InfoSci-Select, InfoSci-Select. Recommend this product to your librarian: www.igi-global.com/e-resources/libraryrecommendation/?id=1
[1]
Ellen Tove Christiansen,et al.
A conceptual toolbox for designing CSCW applications
,
1994
.
[2]
Melvin M. Mark,et al.
Impact analysis for program evaluation: L. B. Mohr. Newbury Park: Sage, (1992)
,
1995
.
[3]
Peter Baumgartner.
Didaktische Anforderungen an (multimediale) Lernsoftware
,
1995
.
[4]
Ivar Jacobson,et al.
The unified modeling language reference manual
,
2010
.
[5]
A Min Tjoa,et al.
Software-Ergonomie : Gestaltung von EDV-Systemen ; Kriterien, Methoden und Werkzeuge
,
1991
.
[6]
M. Jarke,et al.
A PROPOSAL FOR A SCENARIO CLASSIFICATION FRAMEWORK 1
,
2001
.
[7]
Lawrence B. Mohr.
Impact analysis for program evaluation
,
1988
.
[8]
S. Bødker,et al.
Scenarios as Springboards in Design
,
1993
.
[9]
Martin S. Feather,et al.
Utilizing Scenarios in the Software Development Process
,
1993,
Information System Development Process.
[10]
James P. Connell and Anne C. Kubisch.
Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects, and Problems
,
1998
.
[11]
Julio Cesar Sampaio do Prado Leite,et al.
A Scenario Construction Process
,
2000,
Requirements Engineering.
[12]
Mary Beth Rosson,et al.
Usability Engineering: Scenario-based Development of Human-Computer Interaction
,
2001
.
[13]
Peter H. Rossi,et al.
Issues in the theory-driven perspective
,
1989
.
[14]
John M. Carroll,et al.
Making Use: Scenario-Based Design of Human-Computer Interactions
,
2000
.
[15]
Tore Risch,et al.
EDUTELLA: a P2P networking infrastructure based on RDF
,
2002,
WWW.
[16]
Carol H. Weiss.
Evaluation Research: Methods of Assessing Program Effectiveness
,
1972
.
[17]
M.M. Lehman,et al.
Programs, life cycles, and laws of software evolution
,
1980,
Proceedings of the IEEE.
[18]
Wolfgang Nejdl.
Personalized Access to Distributed Learning Repositories - PADLR - Final Proposal
,
2001
.
[19]
Kristen Nygaard,et al.
Program development as a social activity
,
1990
.