The SIETTE Automatic Assessment Environment

This article describes the evolution and current state of the domain-independent Siette assessment environment. Siette supports different assessment methods—including classical test theory, item response theory, and computer adaptive testing—and integrates them with multidimensional student models used by intelligent educational systems. Teachers can use an authoring tool to create large item pools of different types of questions, including multiple choice, open answer, generative questions, and complex tasks. Siette can be used for formative and summative assessment and incorporates different learning elements, including scaffolding features, such as hints, feedback, and misconceptions. It includes numerous other features covering different educational needs and techniques, such as spaced repetition, collaborative testing, or pervasive learning. Siette is designed as a web-based assessment component that can be semantically integrated with intelligent systems or with large LMSs, such as Moodle. This article reviews the evolution of the Siette system, presents information on its use, and analyses this information from a broader and critical perspective on the use of intelligent systems in education.

[1]  Neil J. Dorans,et al.  REVIEW OF COMPUTERIZED ADAPTIVE TESTING: A PRIMER (Second Edition) Computerized Adaptive Testing: A Primer (Second Edition) , 2001 .

[2]  Antonija Mitrovic,et al.  A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling , 2003, User Modeling.

[3]  Eduardo Guzmán,et al.  Exploring Quality of Constraints for Assessment in Problem Solving Environments , 2012, ITS.

[4]  Davinia Hernández Leo,et al.  QuesTInSitu: From tests to routes for assessment in situ activities , 2011, Comput. Educ..

[5]  Ricardo Conejo,et al.  An Empirical Approach to On-Line Learning in SIETTE , 2000, Intelligent Tutoring Systems.

[6]  Ricardo Conejo,et al.  Introducing adaptive assistance in adaptive testing , 2005, AIED.

[7]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[8]  Sebastián Ventura,et al.  Using mobile and web‐based computerized tests to evaluate university students , 2009, Comput. Appl. Eng. Educ..

[9]  Moffat Mathews,et al.  Semantic Integration of Adaptive Educational Systems , 2009, Advances in Ubiquitous User Modelling.

[10]  Paul Libbrecht,et al.  ActiveMath: A Generic and Adaptive Web-Based Learning Environment , 2001 .

[11]  Eduardo Guzmán,et al.  Simultaneous Evaluation of Multiple Topics in SIETTE , 2002, Intelligent Tutoring Systems.

[12]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[13]  Ricardo Conejo,et al.  An Empirical Study About Calibration of Adaptive Hints in Web-Based Adaptive Testing Environments , 2006, AH.

[14]  Eduardo Guzmán,et al.  Measuring Procedural Knowledge in Problem Solving Environments with Item Response Theory , 2013, AIED.

[15]  Jaime Gálvez A Probabilistic Model for Student Knowledge Diagnosis in Learning Environments , 2009, AIED.

[16]  Eduardo Guzmán,et al.  Towards Efficient Item Calibration in Adaptive Testing , 2005, User Modeling.

[17]  R. Conejo,et al.  A library of templates for exercise construction in an adaptive assessment system , 2007 .

[18]  Zachary A. Pardos,et al.  KT-IDEM: introducing item difficulty to the knowledge tracing model , 2011, UMAP'11.

[19]  Eduardo Guzmán,et al.  Measuring the Effect of Collaboration in an Assessment Environment , 2007, AIED.

[20]  A. Rios,et al.  SIETTE: Intelligent Evaluation System using Tests for TeleEducation , 2000 .

[21]  Eduardo Guzmán,et al.  Qualitative and Quantitative Student Models , 2005, AIED.

[22]  Ricardo Conejo,et al.  SIETTE: A Web-Based Tool for Adaptive Testing , 2004, Int. J. Artif. Intell. Educ..

[23]  Eduardo Guzmán,et al.  Data-Driven Student Knowledge Assessment through Ill-Defined Procedural Tasks , 2009, CAEPIA.

[24]  Manuel Hernando Student procedural knowledge inference through item response theory , 2011, UMAP'11.

[25]  Philip I. Pavlik,et al.  An ACT-R Model of the Spacing Effect , 2003 .

[26]  Eduardo Guzmán,et al.  An experiment to measure learning in a collaborative assessment environment , 2009, AIED.

[27]  Eduardo Guzmán,et al.  Self-assessment in a feasible, adaptive web-based testing system , 2005, IEEE Transactions on Education.

[28]  Eduardo Guzmán,et al.  A blended E-learning experience in a course of object oriented programming fundamentals , 2009, Knowl. Based Syst..

[29]  Jaime Galvez Cordero,et al.  Exploring quality of constraints for assessment in problem solving environments , 2012, ITS 2012.

[30]  Rafael García,et al.  INGRID: A Web Service Tool for Hierarchical Open Learner Model Visualization , 2011, UMAP Workshops.

[31]  Kenneth R. Koedinger,et al.  Performance Factors Analysis - A New Alternative to Knowledge Tracing , 2009, AIED.

[32]  Eduardo Guzmán,et al.  Student Knowledge Diagnosis Using Item Response Theory and Constraint-Based Modeling , 2009, AIED.

[33]  Ricardo Conejo,et al.  Technology-Enhanced Formative Assessment of Plant Identification , 2016 .

[34]  Eduardo Guzmán,et al.  A Model for Student Knowledge Diagnosis Through Adaptive Testing , 2004, Intelligent Tutoring Systems.

[35]  Eduardo Guzmán,et al.  A Brief Introduction to the New Architecture of SIETTE , 2004, AH.

[36]  Gordon I. McCalla,et al.  Granularity-Based Reasoning and Belief Revision in Student Models , 1994 .

[37]  Eduardo Guzmán,et al.  Validating Item Response Theory Models in Simulated Environments , 2013, AIED Workshops.

[38]  Martha C. Polson,et al.  Foundations of intelligent tutoring systems , 1988 .

[39]  Ricardo Conejo,et al.  Adaptive testing for hierarchical student models , 2007, User Modeling and User-Adapted Interaction.

[40]  Eduardo Guzmán,et al.  An empirical study on the quantitative notion of task difficulty , 2014, Expert Syst. Appl..

[41]  Jean-Claude Falmagne,et al.  Learning Spaces: Interdisciplinary Applied Mathematics , 2010 .

[42]  J. Ramsay Kernel smoothing approaches to nonparametric item characteristic curve estimation , 1991 .

[43]  Eduardo Guzmán,et al.  A web based collaborative testing environment , 2013, Comput. Educ..

[44]  José-Luis Pérez-de-la-Cruz,et al.  Internet based Evaluation System , 2000 .

[45]  Eduardo Guzmán,et al.  An Authoring Environment for Adaptive Testing , 2005, J. Educ. Technol. Soc..

[46]  Robert J. Mislevy,et al.  Evidence-Centered Assessment Design , 2017 .

[47]  Gordon I. McCalla,et al.  AIED 2013 Simulated Learners Workshop , 2013, AIED.

[48]  Mária Bieliková,et al.  On the impact of adaptive test question selection for learning efficiency , 2010, Comput. Educ..

[49]  Antonija Mitrovic,et al.  Fifteen years of constraint-based tutors: what we have achieved and where we are going , 2011, User Modeling and User-Adapted Interaction.

[50]  Moni Naor,et al.  Adaptive Hypermedia and Adaptive Web-Based Systems , 2004, Lecture Notes in Computer Science.

[51]  Eduardo Guzmán,et al.  A Data-Driven Technique for Misconception Elicitation , 2010, UMAP.

[52]  Bert Bredeweg,et al.  Student Modelling: The Key to Individualized Knowledge-Based Instruction , 2010, NATO ASI Series.

[53]  Eduardo Guzmán,et al.  Measuring Misconceptions Through Item Response Theory , 2015, AIED.

[54]  Michel C. Desmarais,et al.  Learned student models with item to item knowledge structures , 2006, User Modeling and User-Adapted Interaction.

[55]  R. Conejo,et al.  Formative Evaluation of the SIETTE collaborative Testing Environment , 2008 .

[56]  Kenneth R. Koedinger,et al.  Comparing Two IRT Models for Conjunctive Skills , 2008, Intelligent Tutoring Systems.

[57]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[58]  Eduardo Guzmán,et al.  Collaborative assessment with SIETTE , 2009, AIED.

[59]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[60]  Eduardo Guzmán,et al.  Improving Student Performance Using Self-Assessment Tests , 2007, IEEE Intelligent Systems.

[61]  Russell G. Almond,et al.  Evaluating ACED: The Impact of Feedback and Adaptivity on Learning , 2007, AIED.

[62]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[63]  I. Arroyo,et al.  2 An Adaptive Web-based Component for Cognitive Abilit y Estimation , 2000 .

[64]  Eduardo Guzmán,et al.  A Web-Based Socratic Tutor for Trees Recognition , 2000, AH.

[65]  Peter Brusilovsky,et al.  Guiding students to the right questions: adaptive navigation support in an E-Learning system for Java programming , 2010, J. Comput. Assist. Learn..

[66]  Jaime Gálvez Cordero Modelado probabilístico del alumno en entornos inteligentes de resolución de problemas educativos , 2012 .

[67]  Eric G. Hansen,et al.  Evidence-based Approach to Interacting with Open Student Models , 2007, Int. J. Artif. Intell. Educ..

[68]  Eduardo Guzmán,et al.  Statistical Techniques to Explore the Quality of Constraints in Constraint-Based Modeling Environments , 2013, International Journal of Artificial Intelligence in Education.

[69]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[70]  Eduardo Guzmán de los Riscos Un modelo de evaluación cognitiva basado en tests adaptativos informatizados para el diagnóstico en sistemas tutores inteligentes , 2005 .