Data Efficient Educational Assessment via Multi-Dimensional Pairwise Comparisons

The assessment of students based on their tasks is important in Education, and many advanced methods are applied to the field to solve this problem. Many recent neural network approaches involve heavy modeling of the contents and students. However, it is shown that using pairwise comparisons without the direct usage of instance features can show better assessments in the aspect of consistency, and speed. These ideas have been examined in various perspectives since Thurstone proposed the idea of Comparative Judgement(CJ). Whereas CJ requires direct comparisons of instances to obtain the final fit of the label, we give a generalization by proposing a label prediction model which uses the multi-dimensional features of pairwise comparisons. By reducing the cost in label inference, an Education service can provide visualizations of multi-dimensional skill levels for better meta-cognition of the users. Experimental results on the open dataset EdNet KT1 show that our method gives higher accuracy even without using the actual responses for the model input.

[1]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[2]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[3]  Niall Seery,et al.  The validity and value of peer assessment using adaptive comparative judgement in design driven practical education , 2012 .

[4]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[5]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[6]  Forrest W. Young,et al.  Nonmetric individual differences multidimensional scaling: An alternating least squares method with optimal scaling features , 1977 .

[7]  Enhong Chen,et al.  QuesNet: A Unified Representation for Heterogeneous Test Questions , 2019, KDD.

[8]  Leonidas J. Guibas,et al.  Deep Knowledge Tracing , 2015, NIPS.

[9]  A. Culyer.  Thurstone’s Law of Comparative Judgment , 2014 .

[10]  L. Thurstone.  A law of comparative judgment. , 1994 .

[11]  Jean-Claude Falmagne,et al.  Spaces for the Assessment of Knowledge , 1985, Int. J. Man Mach. Stud..

[12]  Joel D. Martin,et al.  Student assessment using Bayesian nets , 1995, Int. J. Hum. Comput. Stud..

[13]  A. Pollitt.  The method of Adaptive Comparative Judgement , 2012 .

[14]  Jozef Hvorecký.  Can E-learning break the Digital Divide? , 2004 .

[15]  A. Pollitt.  Let ’ s stop marking exams , 2004 .

[16]  Kenneth R. Koedinger,et al.  Individualized Bayesian Knowledge Tracing Models , 2013, AIED.

[17]  Alastair Pollitt,et al.  Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment , 2012 .

[18]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[19]  Masashi Sugiyama,et al.  Uncoupled Regression from Pairwise Comparison Data , 2019, NeurIPS.

[20]  Weisi Lin,et al.  Pairwise comparison and rank learning for image quality assessment , 2016, Displays.

[21]  Byungsoo Kim,et al.  Assessment Modeling: Fundamental Pre-training Tasks for Interactive Educational Systems , 2020, ArXiv.

[22]  William F. Strout.  A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation , 1990 .

[23]  Kangwook Lee,et al.  Learning analytics: Collaborative filtering or regression with experts? , 2016, NIPS 2016.

[24]  Ian Jones,et al.  ASSESSING MATHEMATICAL PROBLEM SOLVING USING COMPARATIVE JUDGEMENT , 2015 .

[25]  Byungsoo Kim,et al.  EdNet: A Large-Scale Hierarchical Dataset in Education , 2020, AIED.

[26]  Mehrbakhsh Nilashi,et al.  Collaborative filtering recommender systems , 2013 .

[27]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.