Evaluating the effectiveness of learning interventions: an information systems case study

Currently, there is no standard instrument for evaluating learning effectiveness. While final examinations and end-of-semester course evaluation surveys can be used to do this, they are not designed for this purpose, and there are inherent problems using them in this way. This paper describes a survey instrument, called the Learning Effectiveness Survey, which can be used to evaluate and improve the effectiveness of learning interventions. Learning effectiveness is evaluated in the context of the learning goals of the course (short term learning), and in the context of the overall educational programme and future working life (long term learning). The instrument also provides feedback on the intervention and how it could be improved. A case study is described in which the instrument is used to evaluate the use of peer reviews as a learning activity in a requirements analysis course. The instrument was found to have relatively high validity, but reliability was below acceptable levels. Some interesting results were also found on the determinants of learning. In particular, attitude was found to have no effect on short term learning, but was found to be the primary determinant of long term learning.

[1]  B. Bloom.  Taxonomy of educational objectives , 1956 .

[2]  William E. Cashin.  Student Ratings of Teaching: The Research Revisited. IDEA Paper No. 32. , 1995 .

[3]  H. Russell Bernard,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2000 .

[4]  William E. Cashin.  Student Ratings of Teaching: The Research Revisited , 1982 .

[5]  Hugh Hinton.  Reliability and Validity of Student Evaluations: Testing Models Versus Survey Research Models , 1993 .

[6]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[7]  Kenneth A. Feldman,et al.  The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies , 1989 .

[8]  Hans van Vliet,et al.  Software engineering - principles and practice , 1993 .

[9]  Robin Wilson,et al.  New Research Casts Doubt on Value of Student Evaluations of Professors. , 1998 .

[10]  Hans van Vliet.  Software Engineering: Principles and Practice, 2nd Edition , 2000 .

[11]  Leslie J. Briggs,et al.  Principles of Instructional Design , 1974 .

[12]  Charles E. Snare,et al.  An Alternative End-of-Semester Questionnaire , 2000, PS: Political Science & Politics.

[13]  P. Abrami,et al.  Students' Evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Future Research , 1987 .

[14]  P. A. Cohen.  Student Ratings of Instruction and Student Achievement: A Meta-analysis of Multisection Validity Studies , 1981 .

[15]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[16]  Laura Langbein,et al.  The Validity of Student Evaluations of Teaching , 1994, PS: Political Science & Politics.