An empirical study on the quantitative notion of task difficulty

Most Adaptive and Intelligent Web-based Educational Systems (AIWBES) use tasks in order to collect evidence for inferring knowledge states and adapt the learning process appropriately. To this end, it is important to determine the difficulty of tasks posed to the student. In most situations, difficulty values are directly provided by one or more persons. In this paper we explore the relationship between task difficulty estimations made by two different types of individuals, teachers and students, and compare these values with those estimated from experimental data. We have performed three different experiments with three different real student samples. All these experiments have been done using the SIETTE web-based assessment system. We conclude that heuristic estimation is not always the best solution and claim that automatic estimation should improve the performance of AIWBES.

[1]  Eric G. Hansen,et al.  DESIGNING ADAPTIVE, DIAGNOSTIC MATH ASSESSMENTS FOR SIGHTED AND VISUALLY DISABLED STUDENTS , 2005 .

[2]  Wim J. van der Linden,et al.  Optimal Test Design With Rule-Based Item Generation , 2013 .

[3]  Wim Jansen,et al.  Multilog: Multiple, Categorical Item Analysis and Test Scoring Using Item Response Theory , 1994 .

[4]  Peter Brusilovsky,et al.  ELM-ART: An Adaptive Versatile System for Web-based Instruction , 2001 .

[5]  B. Plake,et al.  Ability of Panelists to Estimate Item Performance for a Target Group of Candidates: An Issue in Judgmental Standard Setting , 2001 .

[6]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[7]  Mária Bieliková,et al.  On the impact of adaptive test question selection for learning efficiency , 2010, Comput. Educ..

[8]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[9]  Wim van den Noortgate,et al.  Item difficulty estimation: An auspicious collaboration between data and judgment , 2012, Comput. Educ..

[10]  Paul Libbrecht,et al.  ActiveMath: A Generic and Adaptive Web-Based Learning Environment , 2001 .

[11]  Christoph Peylo,et al.  W2 - Adaptive and Intelligent Web-Based Education Systems , 2003, Intelligent Tutoring Systems.

[12]  Joseph E. Beck,et al.  Using the Student Model to Control Problem Difficulty , 1997 .

[13]  Eduardo Guzmán,et al.  Improving Student Performance Using Self-Assessment Tests , 2007, IEEE Intelligent Systems.

[14]  Hahn-Ming Lee,et al.  Personalized e-learning system using Item Response Theory , 2005, Comput. Educ..

[15]  R. Hambleton,et al.  An NCME Instructional Module on Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development. , 2005 .

[16]  Zachary A. Pardos,et al.  KT-IDEM: introducing item difficulty to the knowledge tracing model , 2011, UMAP'11.

[17]  Barbara S. Plake,et al.  Teachers' Ability to Estimate Item Difficulty: A Test of the Assumptions in the Angoff Standard Setting Method , 1998 .

[18]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[19]  Luisa M. Regueras,et al.  A genetic fuzzy expert system for automatic question classification in a competitive learning environment , 2012, Expert Syst. Appl..

[20]  Konstantina Chrysafiadi,et al.  Evaluating the integration of fuzzy logic into the student model of a web-based learning environment , 2012, Expert Syst. Appl..

[21]  Antonija Mitrovic,et al.  Using a Probabilistic Student Model to Control Problem Difficulty , 2000, Intelligent Tutoring Systems.

[22]  Neil T. Heffernan,et al.  Addressing the assessment challenge with an online system that tutors as it assesses , 2009, User Modeling and User-Adapted Interaction.

[23]  Ricardo Conejo,et al.  SIETTE: A Web-Based Tool for Adaptive Testing , 2004, Int. J. Artif. Intell. Educ..

[24]  Neil J. Dorans,et al.  REVIEW OF COMPUTERIZED ADAPTIVE TESTING: A PRIMER (Second Edition) Computerized Adaptive Testing: A Primer (Second Edition) , 2001 .

[25]  Peter Brusilovsky,et al.  Adaptive Hypermedia , 2001, User Modeling and User-Adapted Interaction.

[26]  Ricardo Conejo,et al.  Adaptive testing for hierarchical student models , 2007, User Modeling and User-Adapted Interaction.

[27]  Sebastián Ventura,et al.  Educational Data Mining: A Review of the State of the Art , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[28]  Gladys Castillo,et al.  Using Bayesian networks to improve knowledge assessment , 2013, Comput. Educ..

[29]  Joel D. Martin,et al.  J. Evaluation on an assessment system based on Bayesian student modeling , 1997 .

[30]  Janine van der Rijt,et al.  Teachers’ and students’ perceptions of assessments: A review and a study into the ability and accuracy of estimating the difficulty levels of assessment items , 2006 .

[31]  Jonathan P. Weeks.  plink: An R Package for Linking Mixed-Format Tests Using IRT-Based Methods , 2010 .

[32]  Ramón Zataraín-Cabada,et al.  EDUCA: A web 2.0 authoring tool for developing adaptive and intelligent tutoring systems using a Kohonen network , 2011, Expert Syst. Appl..

[33]  Dragan Gasevic,et al.  Student modeling and assessment in intelligent tutoring of software patterns , 2012, Expert Syst. Appl..

[34]  Chih-Ming Chen,et al.  Personalized web-based tutoring system based on fuzzy item response theory , 2008, Expert Syst. Appl..

[35]  Vladimir Zadorozhny,et al.  Re-assessing the Value of Adaptive Navigation Support in E-Learning Context , 2008, AH.

[36]  P. Black,et al.  Developing the theory of formative assessment , 2009 .

[37]  George D. Magoulas,et al.  Fuzzy Inference for Student Diagnosis in Adaptive Educational Hypermedia , 2002, SETN.

[38]  Eduardo Guzmán,et al.  An Educational Component Based Framework for Web ITS Development , 2003, ICWE.

[39]  David Huard,et al.  PyMC: Bayesian Stochastic Modelling in Python. , 2010, Journal of statistical software.

[40]  S. Klinkenberg,et al.  Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation , 2011, Comput. Educ..