An Item Response Theory Analysis of Problem-Solving Processes in Scenario-Based Tasks

(ProQuest: ... denotes formulae omitted.)1. IntroductionAdvances in both technology and cognitively-based assessment design are the drivers towards a radically new vision of assessment, which holds the promise of increasing validity, reliability, and generalizability of the test scores (e.g., Zenisky, & Sireci, 2002). For example, the National Assessment of Educational Progress (NAEP) has embarked on including Scenario-Based Tasks (SBTs) in its Technology and Engineering Literacy (TEL) assessment. SBTs are interactive tasks in which students need to solve problems within realistic scenarios.Although advances in technology allow for new opportunities for educational learning and measurement, and influence the task design, delivery, and data collection, these new technology-enhanced SBTs bring with them new challenges for the analysis and modeling of the data. These challenges are due to the myriad of possibilities of responses and the ill-defined unit of measurement in this complex solution space (Levy, 2012).Unlike multiple-choice items, SBTs provide students with a relatively open workspace to solve a problem, that is, students are allowed to exercise a greater freedom in how they approach problems posed by the tasks. As a result, different students may use different processes for resolving the problems in the tasks. The term process data is used to refer to all of the tracked steps that a student takes to solve a problem in a SBT. Task analysis and scoring, which normally focus exclusively on outcomes of the problem-solving activity, cannot address the question of whether meaningful differences exist among students' different approaches/processes to solving the problem. For instance, what features in the tracked steps are characteristic of successful approaches to a problem? How can unsuccessful strategies be described and distinguished from one another? Progress on broad questions like these depends on having reliable and valid quantitative approaches for identifying and describing students' response processes for new types of items. In this paper, we address two interwoven research questions: (1) how to characterize the process data, so that the key features of students' processes can be captured, and thus, the differences among processes can be distinguished, and (2) how to use the identified features of students' response processes to make inferences about target constructs.In the educational testing field, there is a strong interest in inferring the individual students' abilities based on their response processes. Recent work focused on scoring and characterizing the process data; see for example, a set of papers focused on analyzing the NAEP TEL process data, such as Hao et al (2015) where a measure borrowed from the text analysis called "the editing distance" was introduced to describe score students' processes, Bergner et al (2014) where clustering analysis was proposed for characterizing the process data, and Zhu et al (2016) where the social network analysis was applied to the steps and sequences of the students' processes.In this paper, we propose an approach inspired by the classic Markov models and Item Response Theory (IRT) models to model the process of solving problems in SBTs. We start with a more general theoretical description in order to introduce the method. Then, we narrow it down for this analysis of the data example. Like for the classic Markov models, we first assume that a student's response process has a Markov property, that is, the next state of a stochastic process only depends on the present state. Like the IRT models, the proposed approach utilizes individual-level latent variables to characterize the features of each individual student's response process. Markov-IRT model hereafter is used to refer to this proposed approach.In the rest of the paper, we first present the task, called the Wells task, from the NAEP TEL assessment, and then we present and discuss the proposed Markov-IRT model using the task described previously. …