Analysing Log File Data from PIAAC

The OECD Programme for the International Assessment of Adult Competencies (PIAAC) was the first computer-based large-scale assessment to provide anonymised log file data from the cognitive assessment together with extensive online documentation and a data analysis support tool. The goal of the chapter is to familiarise researchers with how to access, understand, and analyse PIAAC log file data for their research purposes. After providing some conceptual background on the multiple uses of log file data and how to infer states of information processing from log file data, previous research using PIAAC log file data is reviewed. Then, the accessibility, structure, and documentation of the PIAAC log file data are described in detail, as well as how to use the PIAAC LogDataAnalyzer to extract predefined process indicators and how to create new process indicators based on the raw log data export.

[1]  Oliver Lüdtke,et al.  Conditioning factors of test-taking engagement in PIAAC: an exploratory IRT modelling approach considering person and item characteristics , 2017 .

[2]  S. Wise,et al.  A General Approach to Measuring Test-Taking Effort on Computer-Based Tests , 2017 .

[3]  Beatrice Rammstedt,et al.  Personality, competencies, and life outcomes: results from the German PIAAC longitudinal study , 2017, Large-scale Assessments in Education.

[4]  R. H. Klein Entink,et al.  A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers , 2008, Psychometrika.

[5]  Frank Goldhammer,et al.  Validating Test Score Interpretations Using Time Information , 2019, Front. Psychol..

[6]  Annette Stelter,et al.  Erfolgreiches Problemlösen in technologiebasierten Umgebungen: Wechselwirkungen zwischen Interaktionsschritten und Aufgabenanforderungen , 2014 .

[7]  Oliver Lüdtke,et al.  Test-taking engagement in PIAAC , 2016 .

[8]  Steven L. Wise,et al.  The Effects of Effort Monitoring With Proctor Notification on Test-Taking Engagement, Test Performance, and Validity , 2019, Applied Measurement in Education.

[9]  Johannes Naumann,et al.  Relating Product Data to Process Data from Computer-Based Competency Assessment , 2017 .

[10]  Frank Goldhammer,et al.  Speed of reasoning and its relation to reasoning ability , 2011 .

[11]  Bryan Maddox,et al.  Observing response processes with eye tracking in international large-scale assessments: evidence from the OECD PIAAC assessment , 2018, European Journal of Psychology of Education.

[12]  Arthur C. Graesser,et al.  PIAAC Problem Solving in Technology-Rich Environments: A Conceptual Framework. OECD Education Working Papers, No. 36. , 2009 .

[13]  Fabian Zehner,et al.  What to Make Of and How to Interpret Process Data , 2017 .

[14]  Frank Goldhammer,et al.  How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items , 2018, Behaviormetrika.

[15]  Matthias von Davier,et al.  Analyzing Process Data from Problem-Solving Items with N-Grams: Insights from a Computer-Based Large-Scale Assessment , 2016 .

[16]  Frank Goldhammer,et al.  Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading , 2019, Front. Appl. Math. Stat..

[17]  Matthias von Davier,et al.  Identifying Feature Sequences from Process Data in Problem-Solving Items with N -Grams , 2015 .

[18]  Willem J. van der Linden,et al.  Using Response Times for Item Selection in Adaptive Testing , 2008 .

[19]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[20]  Frank Goldhammer,et al.  What makes the difference? The impact of item properties on mode effects in reading assessments , 2019, Studies in Educational Evaluation.

[21]  H. Wainer,et al.  Differential item functioning , 1995 .

[22]  Heiko Rölke,et al.  The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. , 2014 .

[23]  A. Daffertshofer,et al.  Directed Flow of Information in Chimera States , 2019, Front. Appl. Math. Stat..

[24]  Yigal Rosen,et al.  Handbook of Research on Technology Tools for Real-World Skill Development , 2016 .

[25]  Samuel Greiff,et al.  Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files , 2016, Comput. Hum. Behav..

[26]  S. Khapova,et al.  Collective Traumas and the Development of Leader Values: A Currently Omitted, but Increasingly Urgent, Research Area , 2019, Front. Psychol..

[27]  Jonathan P. Weeks,et al.  Using Response Time Data to Inform the Coding of Omitted Responses , 2016 .

[28]  R. Almond,et al.  A BRIEF INTRODUCTION TO EVIDENCE-CENTERED DESIGN , 2003 .

[29]  Jean-François Rouet,et al.  Laypersons' digital problem solving: Relationships between strategy and performance in a large-scale international survey , 2016, Comput. Hum. Behav..

[30]  Willem J. van der Linden,et al.  Linear Models for Optimal Test Design , 2005 .

[31]  Johannes Naumann,et al.  The role of planning in complex problem solving , 2019, Comput. Educ..

[32]  J. Fox,et al.  Bayesian Covariance Structure Modeling of Responses and Process Data , 2019, Front. Psychol..

[33]  Steven L. Wise,et al.  An Application of Item Response Time: The Effort‐Moderated IRT Model , 2006 .

[34]  Qiwei He,et al.  Mapping Background Variables With Sequential Patterns in Problem-Solving Environments: An Investigation of United States Adults’ Employment Status in PIAAC , 2019, Front. Psychol..

[35]  H. Wainer,et al.  Differential Item Functioning. , 1994 .

[36]  Maria Bolsinova,et al.  Improving precision of ability estimation: Getting more from response times. , 2018, The British journal of mathematical and statistical psychology.

[37]  Wim J. van der Linden,et al.  Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing , 2008 .

[38]  Ingo Barkow,et al.  Educational process mining: New possibilities for understanding students' problem-solving skills , 2017 .

[39]  Steffi Pohl,et al.  Using Response Times to Model Not-Reached Items due to Time Limits , 2019, Psychometrika.

[40]  Frank Goldhammer,et al.  Test-Taking Engagement in PIAAC. OECD Education Working Papers, No. 133. , 2016 .