Optimally Discriminative Choice Sets in Discrete Choice Models: Application to Data-Driven Test Design

Difficult multiple-choice (MC) questions can be made easy by providing a set of answer options of which most are obviously wrong. In the education literature, a plethora of instructional guides exist for crafting a suitable set of wrong choices (distractors) that enable the assessment of the students' understanding. The art of MC question design thus hinges on the question-maker's experience and knowledge of the potential misconceptions. In contrast, we advocate a data-driven approach, where correct and incorrect options are assembled directly from the students' own past submissions. Large-scale online classroom settings, such as massively open online courses (MOOCs), provide an opportunity to design optimal and adaptive multiple-choice questions that are maximally informative about the students' level of understanding of the material. In this work, we (i) develop a multinomial-logit discrete choice model for the setting of MC testing, (ii) derive an optimization objective for selecting optimally discriminative option sets, (iii) propose an algorithm for finding a globally-optimal solution, and (iv) demonstrate the effectiveness of our approach via synthetic experiments and a user study. We finally showcase an application of our approach to crowd-sourcing tests from technical online forums.

[1]  C H COOMBS,et al.  Psychological scaling without a unit of measurement. , 1950, Psychological review.

[2]  R. Fortet.  L’algebre de Boole et ses applications en recherche operationnelle , 1960 .

[3]  P. Schönemann.  ON METRIC MULTIDIMENSIONAL UNFOLDING , 1970 .

[4]  Paramesh Ray.  Independence of Irrelevant Alternatives , 1973 .

[5]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[6]  David J. Weiss,et al.  Improving Measurement Quality and Efficiency with Adaptive Testing , 1982 .

[7]  Isaac I. Bejar,et al.  A sentence-based automated approach to the assessment of writing: a feasibility study , 1987 .

[8]  T. Haladyna,et al.  Validity of a Taxonomy of Multiple-Choice Item-Writing Rules. , 1989 .

[9]  D. Thissen,et al.  Multiple-Choice Models: The Distractors Are also Part of the Item. , 1989 .

[10]  T. Haladyna,et al.  How Many Options is Enough for a Multiple-Choice Test Item? , 1993 .

[11]  I. W. Molenaar,et al.  Rasch models: foundations, recent developments and applications , 1995 .

[12]  T. Haladyna.  Writing Test Items to Evaluate Higher Order Thinking , 1996 .

[13]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[14]  Michael C. Rodriguez,et al.  A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment , 2002 .

[15]  Maxine Eskénazi,et al.  Automatic Question Generation for Vocabulary Assessment , 2005, HLT.

[16]  Michael C. Rodriguez.  Three Options Are Optimal for Multiple‐Choice Items: A Meta‐Analysis of 80 Years of Research , 2005 .

[17]  Le An Ha,et al.  A computer-aided environment for generating multiple-choice test items , 2006, Natural Language Engineering.

[18]  Andrew S. Lan,et al.  Learning Analytics via Sparse Factor Analysis , 2012 .

[19]  Tom Minka,et al.  How To Grade a Test Without Knowing the Answers - A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing , 2012, ICML.

[20]  Richard G. Baraniuk,et al.  Test-size Reduction for Concept Estimation , 2013, EDM.