Identifying Consistent Variables in a Heterogeneous Data Set: Evaluation of a Web-Based Pre-Course in Mathematics.

E-learning has made course evaluation easier in many ways, as a multitude of learner data can be collected and related to student performance. At the same time, open learning environments can be a difficult field for evaluation, with a large variance in participants’ knowledge level, learner behaviour, and commitment. In this study the effectiveness of a mathematics pre-course administered to four cohorts of prospective students at a technical faculty in Germany was evaluated. Deficits in basic mathematics knowledge are considered one risk factor regarding graduation in STEM-related subjects, thus the overall goal was to investigate if the pre-course enabled “at risk” students to improve their starting position. A data analysis was performed, relating students’ preconditions when entering university, their attitude towards mathematics, and their use of learning strategies with further study success. The strongest determinant of first year performance were results in a diagnostic pretest, confirming both the importance of basic mathematics knowledge for academic achievement in engineering and the reliability of the chosen pre-posttest design. Other outcomes were quite unexpected and demanded deeper analyses. Students who had participated in additional face-toface courses, for example, showed less learning gains than students who had participated in an e-tutoring version. It also could be observed that meta-cognitive variables failed to explain successful course participation. Reasons for these outcomes are discussed, suggesting reliability threats and interactions between students’ preconditions and their learner behaviour. A significant and unmoderated impact on students’ learning gains in the pre-course was found for the number of online test attempts, making this variable a reliable indicator of student engagement. The evaluations show that open learning designs with heterogeneous learner groups can deliver meaningful information, provided that limitations are considered and that external references, like academic grades, are available in order to establish consistency.