Psychometric Challenges in Modeling Scientific Problem-Solving Competency: An Item Response Theory Approach

The ability to solve complex problems is one of the key competencies in science. In previous research, modeling scientific problem solving has mainly focused on the dimensionality of the construct, but rarely addressed psychometric test characteristics such as local item dependencies which could occur, especially in computer-based assessments. The present study consequently aims to model scientific problem solving by taking into account four components of the construct and dependencies among items within these components. Based on a data set of 1,487 German high-school students of different grade levels, who worked on computer-based assessments of problem solving, local item dependencies were quantified by using testlet models and Q 3 statistics. The results revealed that a model differentiating testlets of cognitive processes and virtual systems fitted the data best and remained invariant across grades.

[1]  Joachim Funke,et al.  Complex problem solving: a case for complex cognition? , 2010, Cognitive Processing.

[2]  Christine E. DeMars.  Confirming Testlet Effects , 2012 .

[3]  J. Lucke.  “Rassling the Hog”: The Influence of Correlated Item Error on Internal Consistency, Classical Reliability, and Congeneric Reliability , 2005 .

[4]  Samuel Greiff,et al.  Complex problem solving : More than reasoning ? , 2017 .

[5]  Steffen Brandt.  Robustness of multidimensional analyses against local item dependence , 2012 .

[6]  Marco Ragni,et al.  Complex problem solving: another test case? , 2009, Cognitive Processing.

[7]  Ronny Scherer,et al.  Evidence on the effects of task interactivity and grade level on thinking skills involved in complex problem solving , 2014 .

[8]  Edward Haksing Ip,et al.  Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models. , 2010, The British journal of mathematical and statistical psychology.

[9]  R. Brennan,et al.  Test Equating, Scaling, and Linking , 2004 .

[10]  Wendy M. Yen,et al.  Scaling Performance Assessments: Strategies for Managing Local Item Dependence , 1993 .

[11]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[12]  Sun-Joo Cho,et al.  Explanatory Item Response Models , 2004 .

[13]  D. Andrich,et al.  Formalizing dimension and response violations of local independence in the unidimensional Rasch model. , 2008, Journal of applied measurement.

[14]  C. Artelt,et al.  Die Bedeutung der Itemauswahl und der Modellwahl für die längsschnittliche Erfassung von Kompetenzen , 2011 .

[15]  Joachim Wirth,et al.  Computer-based Assessment of Problem Solving Competence , 2003 .