Automatic Item Generation

Changes to the design and development of educational tests are resulting in the unprecedented demand for a large supply of content-specific test items. One way to address this growing demand is with automatic item generation. Automatic item generation is the process of using models to create test items with the aid of computer technology. The purpose of this chapter is to describe and illustrate a method for generating test items. The method is also illustrated using an example from the medical health sciences.

[1]  David M. Williamson,et al.  Calibrating Item Families and Summarizing the Results Using Family Expected Response Functions , 2003 .

[2]  Mark J. Gierl,et al.  Instructional Topics in Educational Measurement (ITEMS) Module: Using Automated Processes to Generate Test Items , 2013 .

[3]  Mark J. Gierl,et al.  The Role of Cognitive Models in Automatic Item Generation , 2016 .

[4]  Markus Sommer,et al.  Using Automatic Item Generation to Simultaneously Construct German and English Versions of a Word Fluency Test , 2012 .

[5]  Maxine Eskénazi,et al.  Automatic Question Generation for Vocabulary Assessment , 2005, HLT.

[6]  Cornelis A.W. Glas,et al.  Modeling Rule-Based Item Generation , 2011 .

[7]  Wells HivelyII,et al.  A “UNIVERSE‐DEFINED” SYSTEM OF ARITHMETIC ACHIEVEMENT TESTS1 , 1968 .

[8]  Paul Deane,et al.  Automatic Item Generation via Frame Semantics: Natural Language Generation of Math Word Problems. , 2003 .

[9]  James W Pellegrino,et al.  Technology and Testing , 2009, Science.

[10]  Donovan R. Hare,et al.  Assembling an inventory of multistage adaptive testing systems , 2009 .

[11]  Vikram Sheel Kumar,et al.  A test in development. , 2013, Clinical chemistry.

[12]  Mark J. Gierl,et al.  Evaluating the quality of medical multiple‐choice items created with automated processes , 2013, Medical education.

[13]  Lutz F. Hornke.  Item Generation Models for Higher Order Cognitive Functions , 1999 .

[14]  Mark J. Gierl,et al.  Generating Items Under the Assessment Engineering Framework , 2012 .

[15]  Michal Beller,et al.  Technologies in Large-Scale Assessments: New Directions, Challenges, and Opportunities , 2013 .

[16]  Markus Sommer,et al.  Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items , 2007 .

[17]  A. Laduca,et al.  Item modelling procedure for constructing content‐equivalent multiple choice questions , 1986, Medical education.

[18]  Mark J. Gierl,et al.  Automatic item generation : theory and practice , 2012 .

[19]  Montse Maritxalar,et al.  Automatic Distractor Generation for Domain Specific Texts , 2010, IceTAL.

[20]  Thomas M. Haladyna,et al.  Using Weak and Strong Theory to Create Item Models for Automatic Item Generation: Some Practical Guidelines with Examples , 2012 .

[21]  M. Oliveri,et al.  The Learning Sciences in Educational Assessment: The Role of Cognitive Models , 2011, Alberta Journal of Educational Research.

[22]  Martin Arendasy,et al.  Automatic Generation of Rasch-Calibrated Items: Figural Matrices Test GEOM and Endless-Loops Test EC , 2005 .

[23]  Tomoko Kojiri,et al.  Automatic Generation System of Multiple-Choice Cloze Questions and its Evaluation , 2010 .

[24]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[25]  Le An Ha,et al.  Generating Multiple-Choice Test Items from Medical Text: A Pilot Study , 2006, INLG.

[26]  Michael C. Rodriguez,et al.  Developing and Validating Test Items , 2013 .

[27]  Kevin W. Eva,et al.  The Cambridge Handbook of Expertise and Expert Performance: Expertise in Medicine and Surgery , 2006 .

[28]  Le An Ha,et al.  A computer-aided environment for generating multiple-choice test items , 2006, Natural Language Engineering.

[29]  Sandip Sinharay,et al.  Use of Item Models in a Large-Scale Admissions Test: A Case Study , 2008 .

[30]  Richard M. Luecht.  An Introduction to Assessment Engineering for Automatic Item Generation , 2012 .

[31]  Issac I. Bejar.  A Generative Analysis of a Three-Dimensional Spatial Task , 1990 .

[32]  Randy Elliot Bennett,et al.  How the Internet Will Help Large-Scale Assessment Reinvent Itself , 2001 .

[33]  Mark J. Gierl,et al.  Developing a Taxonomy of Item Model Types to Promote Assessment Engineering , 2008 .

[34]  Randy Elliot Bennett,et al.  Item generation and beyond: Applications of schema theory to mathematics assessment. , 2002 .

[35]  张来春（编译）.  美国《教育周刊》Technology Counts三年追踪 , 2007 .

[36]  Wim J. van der Linden,et al.  Computerized Adaptive Testing With Item Cloning , 2003 .

[37]  Susan E. Embretson,et al.  23 Automatic Item Generation and Cognitive Psychology , 2006 .

[38]  S. Embretson.  CONSTRUCT VALIDITY: CONSTRUCT REPRESENTATION VERSUS NOMOTHETIC SPAN , 1983 .

[39]  Roberto J. Bayardo,et al.  Scaling up all pairs similarity search , 2007, WWW '07.

[40]  Robert J. Mislevy,et al.  Evidence-Centered Assessment Design , 2017 .

[41]  Morris De Beer,et al.  Technology and Testing , 2013 .

[42]  Isaac I. Bejar.  GENERATIVE RESPONSE MODELING: LEVERAGING THE COMPUTER AS A TEST DELIVERY MEDIUM , 1996 .

[43]  Mehran Sahami,et al.  Evaluating similarity measures: a large-scale study in the orkut social network , 2005, KDD '05.

[44]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[45]  John R. Bormuth,et al.  On the theory of achievement test items , 1970 .

[46]  Mary Webb,et al.  Challenges for information technology supporting educational assessment , 2013, J. Comput. Assist. Learn..

[47]  Steven M. Downing,et al.  Handbook of test development , 2006 .

[48]  Mark J. Gierl,et al.  Using automatic item generation to create multiple‐choice test items , 2012, Medical education.

[49]  Isaac I. Bejar,et al.  A FEASIBILITY STUDY OF ON‐THE‐FLY ITEM GENERATION IN ADAPTIVE TESTING , 2002 .

[50]  Anne Wendt,et al.  Developing Item Variants: An Empirical Study , 2009 .

[51]  Christian Gütl,et al.  Refined Distractor Generation with LSA and Stylometry for Automated Multiple Choice Question Generation , 2012, Australasian Conference on Artificial Intelligence.

[52]  Lawrence M. Rudner,et al.  Implementing the Graduate Management Admission Test Computerized Adaptive Test , 2009 .

[53]  M. Sommer,et al.  Evaluating the contribution of different item features to the effect size of the gender difference in three-dimensional mental rotation using automatic item generation , 2010 .

[54]  Marija Cubric,et al.  Towards automatic generation of e-assessment using semantic web technologies , 2011 .

[55]  Andreas Papasalouros,et al.  Automated Transformation of SWRL Rules into Multiple-Choice Questions , 2011, FLAIRS.

[56]  Christian Gütl,et al.  Enhanced Automatic Question Creator--EAQC: Concept, Development and Evaluation of an Automatic Test Item Creation Tool to Foster Modern e-Education , 2011 .

[57]  P. Black,et al.  Inside the Black Box: Raising Standards through Classroom Assessment , 2010 .

[58]  J. Herman,et al.  Defining Twenty-First Century Skills , 2012 .

[59]  J. H. McMillan.  Annual Meeting of the American Educational Research , 2001 .

[60]  Susan E. Embretson,et al.  Generating items during testing: Psychometric issues and models , 1999 .

[61]  Bijan Parsia,et al.  Automatic generation of analogy questions for student assessment: an Ontology-based approach , 2012 .

[62]  Mark J. Gierl,et al.  Using Automated Item Generation to Promote Principled Test Design and Development , 2010 .

[63]  Susan E. Embretson,et al.  Understanding and Quantifying Cognitive Complexity Level in Mathematical Problem Solving Items , 2008 .

[64]  Paul Deane,et al.  MULTILINGUAL GENERALIZATION OF THE MODELCREATOR SOFTWARE FOR MATH ITEM GENERATION , 2005 .

[65]  Derrick Higgins.  Item Distiller : Text Retrieval for Computer-Assisted Test Item Creation , 2007 .