Using Bayesian networks to improve knowledge assessment

In this paper, we describe the integration and evaluation of an existing generic Bayesian student model (GBSM) into an existing computerized testing system within the Mathematics Education Project (PmatE - Projecto Matematica Ensino) of the University of Aveiro. This generic Bayesian student model had been previously evaluated with simulated students, but a real application was still missing. In the work presented here, we have used the GBSM to define Bayesian Student Models (BSMs) for a concrete domain: first degree equations. In order to test the diagnosis capabilities of such BSMs, an evaluation with 152 students has been performed. Each of the 152 students took both a computerized test within PMatE and a written exam, both of them designed to measure students' knowledge in 12 concepts related to first degree equations. The written exam was graded by three experts. Then two BSMs were developed, one for the computer test and another one for the written exam. These BSMs were used to obtain estimations of student's knowledge on the same 12 concepts, and the inter-rater agreement among the different measures was computed. Results show a high degree of agreement among the scores given by the experts and also among the diagnosis provided by the BSM in the written exam and expert's average, but a low degree of agreement among the diagnosis provided by the BSM in the computer test and expert's average.

[1]  José-Luis Pérez-de-la-Cruz,et al.  Bayesian networks for student model engineering , 2010, Comput. Educ..

[2]  Gladys Castillo,et al.  Computerized Evaluation and Diagnosis of Student's Knowledge Based on Bayesian Networks , 2010, EC-TEL.

[3]  J. S. Pinto,et al.  TDmat–mathematics diagnosis evaluation test for engineering sciences students , 2007 .

[4]  Cristina Conati,et al.  On-Line Student Modeling for Coached Problem Solving Using Bayesian Networks , 1997 .

[5]  Kurt VanLehn,et al.  Student Modeling from Conversational Test Data: A Bayesian Approach Without Priors , 1998, Intelligent Tutoring Systems.

[6]  José-Luis Pérez-de-la-Cruz,et al.  Dynamic versus Static Student Models Based on Bayesian Networks: An Empirical Study , 2003, KES.

[7]  Jim E. Greer,et al.  Adaptive Assessment Using Granularity Hierarchies and Bayesian Nets , 1996, Intelligent Tutoring Systems.

[8]  James Stamey,et al.  Using Bland−Altman to Assess Agreement between Two Medical Devices – Don’t Forget the Confidence Intervals! , 2007, Journal of Clinical Monitoring and Computing.

[9]  Ricardo Conejo,et al.  Introducing Prerequisite Relations in a Multi-layered Bayesian Student Model , 2005, User Modeling.

[10]  Anthony Jameson,et al.  Numerical uncertainty management in user and student modeling: An overview of systems and issues , 2005, User Modeling and User-Adapted Interaction.

[11]  Douglas G. Altman,et al.  Measurement in Medicine: The Analysis of Method Comparison Studies , 1983 .

[12]  Kurt VanLehn,et al.  Applications of simulated students: an exploration , 1994 .

[13]  Jirí Vomlel,et al.  Bayesian Networks In Educational Testing , 2002, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[14]  José-Luis Pérez-de-la-Cruz,et al.  A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation , 2002, User Modeling and User-Adapted Interaction.