Validating process variables of sourcing in an assessment of multiple document comprehension

Background With digital technologies, competence assessments can provide process data, such as mouse clicks with corresponding timestamps, as additional information about the skills and strategies of test takers. However, in order to use variables generated from process data sensibly for educational purposes, their interpretation needs to be validated with regard to their intended meaning. Aims This study seeks to demonstrate how process data from an assessment of multiple document comprehension can be used to represent sourcing, which summarizes activities for the consideration of the origin and intention of documents. The investigated process variables were created according to theoretical assumptions about sourcing, and systematically tested for differences between persons, units (i.e., documents and items), and properties of the test administration. Sample The sample included 310 German university students (79.4% female), enrolled in several bachelor's or master's programmes of the social sciences and humanities. Methods Regarding the hierarchical data structure, the hypotheses were analysed with generalized linear mixed models (GLMM). Results The results mostly revealed expected differences between individuals and units. However, unexpected effects of the administered order of units and documents were detected. Conclusions The study demonstrates the theory‐informed construction of process variables from log‐files and an approach for empirical validation of their interpretation. The results suggest that students apply sourcing for different reasons, but also stress the need of further validation studies and refinements in the operationalization of the indicators investigated.

[1]  Ivan K. Ash,et al.  Source Evaluation, Comprehension, and Learning in Internet Science Inquiry Tasks , 2009 .

[2]  Jason L. G. Braasch,et al.  Processing Inaccurate Information: Theoretical and Applied Perspectives from Cognitive Science and the Educational Sciences , 2014 .

[3]  Nicolas Vibert,et al.  Readers’ use of source information in text comprehension , 2011, Memory & Cognition.

[4]  M. Anne Britt,et al.  Enhancing the Quality of Learning: Learning with Multiple Documents , 2012 .

[5]  T. A. Warm.  Weighted likelihood estimation of ability in item response theory , 1989 .

[6]  Steven M. Downing,et al.  Handbook of test development , 2006 .

[7]  C. Artelt,et al.  Entwicklung und Skalierung eines Tests zur Erfassung des Verständnisses multipler Dokumente von Studierenden , 2020 .

[8]  Jason L. G. Braasch,et al.  The Discrepancy-Induced Source Comprehension (D-ISC) Model: Basic Assumptions and Preliminary Evidence , 2017 .

[9]  Jukka Hyönä,et al.  Perspective-driven text comprehension , 2008 .

[10]  Frank Goldhammer,et al.  How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items , 2018, Behaviormetrika.

[11]  S. Wineburg.  Historical Problem Solving: A Study of the Cognitive Processes Used in the Evaluation of Documentary and Pictorial Evidence , 1991 .

[12]  Ivar Bråten,et al.  The Role of Sourcing in Discourse Comprehension , 2017 .

[13]  G. Jacobs,et al.  Alternative Assessment , 2019, Student Centered Cooperative Learning.

[14]  M. Kane.  Validating the Interpretations and Uses of Test Scores , 2013 .

[15]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[16]  Daniel M. McNeish,et al.  Psychological Methods Thanks Coefficient Alpha , We ’ ll Take It From Here , 2022 .

[17]  M. A. Britt,et al.  Improving Students' Ability to Identify and Use Source Information , 2002 .

[18]  Jason L. G. Braasch,et al.  Content integration across multiple documents reduces memory for sources , 2016 .

[19]  Bert P. M. Creemers,et al.  International encyclopedia of education (3rd ed.) , 2010 .

[20]  Gerd Gigerenzer,et al.  Heuristic decision making. , 2011, Annual review of psychology.

[21]  M. Anne Britt,et al.  Reading multiple texts about climate change: The relationship between memory for sources and text comprehension , 2010 .

[22]  S. Wineburg.  Historical Problem Solving: A Study of the Cognitive Processes Used in the Evaluation of Documentary and Pictorial Evidence , 1991 .

[23]  Fabian Zehner,et al.  What to Make Of and How to Interpret Process Data , 2017 .