The Beast of Aggregating Cognitive Load Measures in Technology-Based Learning

An increasing part of cognitive load research in technology-based learning includes a component of repeated measurements, that is: participants are measured two or more times on the same performance, mental effort or other variable of interest. In many cases, researchers aggregate scores obtained from repeated measurements to one single sum or average score per participant and use these aggregated scores in subsequent analysis. This paper demonstrates some dangers of this commonly encountered aggregation approach and presents two comprehensive alternatives: Split-plot analysis of variance (ANOVA) and more flexible two-level regression analysis. The core message of this paper is that the application of the aggregation approach can seriously distort our view of effects and relations of interest and should therefore not be used in cognitive load research. Multilevel analysis of repeated measurements data can account for various features of the data and constitutes a best practice.

[1]  T. Gog,et al.  Training self-assessment and task-selection skills: A cognitive approach to improving self-regulated learning , 2012 .

[2]  G. Salomon.  Television is "easy" and print is "tough": The differential investment of mental effort in learning as a function of perceptions and attributions. , 1984 .

[3]  T. Gog,et al.  Effects of pairs of problems and examples on task performance and different types of cognitive load , 2014 .

[4]  J. M. Cortina,et al.  What Is Coefficient Alpha? An Examination of Theory and Applications , 1993 .

[5]  A. Koriat,et al.  Judgments of learning depend on how learners interpret study effort. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[6]  Kerri L. Johnson,et al.  Why People Fail to Recognize Their Own Incompetence , 2003 .

[7]  Katharina Scheiter,et al.  Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? , 2009, Comput. Hum. Behav..

[8]  Andy P. Field,et al.  Discovering Statistics Using Ibm Spss Statistics , 2017 .

[9]  E. Erdfelder,et al.  Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses , 2009, Behavior research methods.

[10]  Fred Paas,et al.  Self-assessment and task selection in learner-controlled instruction: Differences between effective and ineffective learners , 2010, Comput. Educ..

[11]  T. Gog,et al.  Timing and Frequency of Mental Effort Measurement: Evidence in Favour of Repeated Measures , 2012 .

[12]  John Dunlosky,et al.  Toward a general model of self-regulated study: An analysis of selection of items for study and self-paced study time. , 1999 .

[13]  Daniel C. Moos,et al.  Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? , 2008 .

[14]  Jimmie Leppink,et al.  Prior knowledge moderates instructional effects on conceptual understanding of statistics , 2012 .

[15]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[16]  D. C. Howell.  Statistical methods for psychology, 3rd ed. , 1992 .

[17]  P. Chandler,et al.  Cognitive load as a factor in the structuring of technical material. , 1990 .

[18]  Donald Hedeker,et al.  Longitudinal Data Analysis , 2006 .

[19]  C. McCulloch,et al.  Misspecifying the Shape of a Random Effects Distribution: Why Getting It Wrong May Not Matter , 2011, 1201.1980.

[20]  Jimmie Leppink.  Data analysis in medical education research: a multilevel perspective , 2015, Perspectives on medical education.

[21]  J. Sweller.  Element Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load , 2010 .

[22]  T. O. Nelson.  Metamemory: A Theoretical Framework and New Findings , 1990 .

[23]  Slava Kalyuga.  Cognitive Load Theory: How Many Types of Load Does It Really Need? , 2011 .

[24]  R. Azevedo,et al.  Does training on self-regulated learning facilitate students' learning with hypermedia? , 2004 .

[25]  F. Paas,et al.  Instructional control of cognitive load in the training of complex cognitive tasks , 1994 .

[26]  D. Weakliem.  A Critique of the Bayesian Information Criterion for Model Selection , 1999 .

[27]  J. Hox.  Multilevel analysis: Techniques and applications, 2nd ed. , 2010 .

[28]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[29]  G. A. Marcoulides.  Multilevel Analysis Techniques and Applications , 2002 .

[30]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[31]  P. Chandler,et al.  Why Some Material Is Difficult to Learn , 1994 .

[32]  J. Sweller.  COGNITIVE LOAD THEORY, LEARNING DIFFICULTY, AND INSTRUCTIONAL DESIGN , 1994 .

[33]  Frans E. S. Tan,et al.  Best Practices in Analysis of Longitudinal Data: A Multilevel Approach , 2008 .

[34]  J. Flavell.  Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry. , 1979 .

[35]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[36]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[37]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[38]  F. Paas,et al.  Variability of Worked Examples and Transfer of Geometrical Problem-Solving Skills: A Cognitive-Load Approach , 1994 .

[39]  J. Metcalfe,et al.  CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE Metacognitive Judgments and Control of Study , 2022 .

[40]  Bradley E. Huitema,et al.  The Analysis of Covariance and Alternatives: Statistical Methods for Experiments, Quasi-Experiments, and Single-Case Studies , 2011 .

[41]  G. Molenberghs,et al.  Models for Discrete Longitudinal Data , 2005 .

[42]  Robert S. Stawski,et al.  Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling (2nd Edition) , 2013 .

[43]  F. Paas.  Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. , 1992 .

[44]  Bobby Hoffman,et al.  Cognitive efficiency: A conceptual and methodological comparison , 2012 .

[45]  William J. Browne,et al.  Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models , 2000, Comput. Stat..

[46]  G. Molenberghs.  Applied Longitudinal Analysis , 2005 .

[47]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[48]  Ana Ivelisse Avilés,et al.  Linear Mixed Models for Longitudinal Data , 2001, Technometrics.

[49]  T. Gog,et al.  Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings , 2015 .

[50]  B. Zimmerman,et al.  Self-regulated learning and academic achievement : theoretical perspectives , 2001 .

[51]  E. Bareiss.  Numerical solution of linear equations with Toeplitz and Vector Toeplitz matrices , 1969 .

[52]  Fred Paas,et al.  Personalised adaptive task selection in air traffic control: Effects on training efficiency and transfer , 2006 .

[53]  T. Gog,et al.  Development of an instrument for measuring different types of cognitive load , 2013, Behavior Research Methods.

[54]  John Sweller,et al.  Cognitive Load Theory , 2020, Encyclopedia of Education and Information Technologies.

[55]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[56]  Tamara van Gog,et al.  The effect of delayed-JOLs and sentence generation on children’s monitoring accuracy and regulation of idiom study , 2013 .

[57]  J. Merriënboer,et al.  Selecting learning tasks: Effects of adaptation and shared control on learning efficiency and task involvement☆ , 2008 .

[58]  D. C. Howell.  Statistical Methods for Psychology , 1987 .