Learning-centred evaluation of Computer-facilitated learning projects in higher education: Outcomes of CUTSD staff development grant

This staff development project, funded by the Australian Government Committee for University Teaching and Staff Development (CUTSD) was managed by the Teaching and Learning Centre at Murdoch University, on behalf of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE) and a consortium of 10 other universities. The project sought to address widely-acknowledged shortcomings in the evaluation of Computer-facilitated Learning projects (CFL), identified in a CUTSD report by Alexander & McKenzie, in 1998. It involved staff development in evaluation of CFL projects, proceeding from the premise that academics in most discipline areas generally have neither the skills nor expertise to carry out scholarly evaluations of student learning. 41 university staff were supported by 11 mentors in evaluating 18 CFL projects, and 17 of these are reported in this publication. Background material about the conduct of the project is also available. An evaluation of the overall project is also reported here. The evaluation had two purposes: to examine how well the project was implemented (formative approach) and to analyse the extent to which the stated outcomes of the project were achieved (summative approach). The formative component of the evaluation was designed to collect data throughout the term of the project and to encourage critical reflection by the various project participants including mentees, mentors, writers and the project coordinator team. In the summative component of the evaluation, information was collected to assess the extent to which the following intended outcomes, identified in the original funding application to CUTSD, were achieved: participants will have gained transferable skills in conducting meaningful evaluation of CFL projects using high standards of practice; participants will have conducted a well-grounded evaluation of a CFL development project from the viewpoint of student learning outcomes; participants will have experienced the action inquiry process; and participants will have published a scholarly paper on the results of the evaluation study. the project will result in the production of a staff development guide. the project will deliver a volume of scholarly papers which details the results of the evaluation studies carried out as part of this project. Overall, the implementation of the project was found to be largely as planned, with some changes made to address changed circumstances or in response to lessons learned during implementation. The project was a complex undertaking, which was well-planned and monitored. The six outcomes of the project also were found to be largely achieved. Most mentees had not previously conducted a formal evaluation study and many had not been exposed to action inquiry processes. It was a very challenging experience for a number of them to take on both of these new activities on an innovation in one of their own courses. However, to their considerable credit, nearly all persevered and completed their evaluation studies. Judging from their feedback, and that of their mentors, as well as the high quality of the evaluation plans and reports, the mentees found the experience very rewarding and gained a range of skills in evaluation which they will be able to transfer to future situations. Every evaluation study identified areas of improvement in the way the innovation was used. In three cases, project teams also identified improvements in the way that the subject was taught or in the way that the innovation was integrated into the curriculum. Two studies identified that the CFL was not used by students in the way intended. However, six projects identified clear evidence of student learning, either in terms of learning processes or learning outcomes. The Staff Development Guide has yet to be produced but is scheduled for publication in 2002. In addition, a special issue of the Australian Journal of Educational Technology will be published comprising articles about several of the evaluation studies conducted in this project.