Do Instrumentation Tools Capture Self-Regulated Learning?

Researchers have been struggling with the measurement of Self-Regulated Learning (SRL) for decades. Instrumentation tools have been proposed to help capture SRL processes that are difficult to capture. The aim of the present study was to improve measurement of SRL by embedding instrumentation tools in a learning environment and validating the measurement of SRL with these instrumentation tools using think aloud. Synchronizing log data and concurrent think aloud data helped identify which SRL processes were captured by particular instrumentation tools. One tool was associated with a single SRL process: the timer co-occurred with monitoring. Other tools co-occurred with a number of SRL processes, i.e., the highlighter and note taker captured superficial writing down, organizing, and monitoring, whereas the search and planner tools revealed planning and monitoring. When specific learner actions with the tool were analyzed, a clearer picture emerged of the relation between the highlighter and note taker and SRL processes. By aligning log data with think aloud data, we showed that instrumentation tool use indeed reflects SRL processes. The main contribution is that this paper is the first to show that SRL processes that are difficult to measure by trace data can indeed be captured by instrumentation tools such as high cognition and metacognition. Future challenges are to collect and process log data real time with learning analytic techniques to measure ongoing SRL processes and support learners during learning with personalized SRL scaffolds.

[1]  Marit S. Samuelstuen,et al.  Measuring strategic processing: comparing task-specific self-reports to traces , 2007 .

[2]  Marcel V. J. Veenman,et al.  Assessing Metacognitive Skills in Computerized Learning Environments , 2013 .

[3]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[4]  B. Zimmerman,et al.  Handbook of Self-Regulation of Learning and Performance , 2011 .

[5]  Philip H. Winne,et al.  Theorizing and researching levels of processing in self‐regulated learning , 2018, The British journal of educational psychology.

[6]  Philip H. Winne,et al.  A METACOGNITIVE VIEW OF INDIVIDUAL DIFFERENCES IN SELF-REGULATED LEARNING , 1996 .

[7]  Henning Holle,et al.  EasyDIAg: A tool for easy determination of interrater agreement , 2014, Behavior Research Methods.

[8]  Daniel C. Moos,et al.  Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? , 2008 .

[9]  Philip H. Winne,et al.  Construct and consequential validity for learning analytics based on trace data , 2020, Comput. Hum. Behav..

[10]  Daniel C. Moos,et al.  Measuring Cognitive and Metacognitive Regulatory Processes During Hypermedia Learning: Issues and Challenges , 2010 .

[11]  R. Mayer.  Learning strategies for making sense out of expository text: The SOI model for guiding three cognitive processes in knowledge construction , 1996 .

[12]  Michelle Taub,et al.  How Does Prior Knowledge Influence Eye Fixations and Sequences of Cognitive and Metacognitive SRL Processes during Learning with an Intelligent Tutoring System? , 2018, International Journal of Artificial Intelligence in Education.

[13]  Geraldine Clarebout,et al.  Tool use in computer-based learning environments: towards a research framework , 2006, Comput. Hum. Behav..

[14]  I. Molenaar,et al.  Metacognitive scaffolding in an innovative learning arrangement , 2011 .

[15]  Dragan Gasevic,et al.  Trace-SRL: A Framework for Analysis of Microlevel Processes of Self-Regulated Learning From Trace Data , 2020, IEEE Transactions on Learning Technologies.

[16]  Philip H. Winne,et al.  Cognition and Metacognition within Self-Regulated Learning , 2017 .

[17]  Ido Roll,et al.  Understanding, evaluating, and supporting self-regulated learning using learning analytics , 2015, J. Learn. Anal..

[18]  Michelle Taub,et al.  Understanding and Reasoning about Real-Time Cognitive, Affective, and Metacognitive Processes to Foster Self-Regulation with Advanced Learning Technologies , 2017 .

[19]  Ernesto Panadero,et al.  A Review of Self-regulated Learning: Six Models and Four Directions for Research , 2017, Front. Psychol..

[20]  I. Molenaar,et al.  Effects of Sequences of Cognitions on Group Performance Over Time , 2017, Small group research.

[21]  I. Molenaar,et al.  Dissecting sequences of regulation and cognition: statistical discourse analysis of primary school children’s collaborative learning , 2014 .

[22]  Anastasios A. Economides,et al.  Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence , 2014, J. Educ. Technol. Soc..

[23]  A. Hadwin,et al.  Using Cognitive Tools in Gstudy to Investigate How Study Activities Covary with Achievement Goals , 2006 .

[24]  F. di Vesta,et al.  Listening and note taking. , 1972, Journal of educational psychology.

[25]  Roger Azevedo,et al.  A macro-level analysis of SRL processes and their relations to the acquisition of a sophisticated mental model of a complex system☆☆☆ , 2009 .

[26]  Philip H. Winne,et al.  Experimenting to Bootstrap Self-Regulated Learning , 1997 .

[27]  Roger Azevedo,et al.  Does adaptive scaffolding facilitate students ability to regulate their learning with hypermedia? q , 2004 .

[28]  Inge Molenaar,et al.  The effects of scaffolding metacognitive activities in small groups , 2010, Comput. Hum. Behav..

[29]  Jovita M. Vytasek,et al.  What if Learning Analytics Were Based on Learning Science , 2016 .

[30]  Michelle Taub,et al.  Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners' levels of prior knowledge in hypermedia-learning environments? , 2014, Comput. Hum. Behav..

[31]  P. Reimann,et al.  Process mining techniques for analysing patterns and strategies in students’ self-regulated learning , 2013, Metacognition and Learning.

[32]  Marek Hatala,et al.  Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes , 2016, J. Learn. Anal..

[33]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[34]  Philip H. Winne,et al.  Learning Analytics for Self-Regulated Learning , 2017 .

[35]  Jeffrey A. Greene,et al.  Historical, Contemporary, and Future Perspectives on Self-Regulated Learning and Performance , 2017 .

[36]  Matthew L. Bernacki.  Examining the cyclical, loosely sequenced, and contingent features of self-regulated learning: Trace data and their analysis. , 2017 .

[37]  Bennett L. Schwartz,et al.  Motivation and Affect in Self-Regulated Learning : Does Metacognition Play a Role? , 2017 .

[38]  Philip H. Winne,et al.  How Software Technologies Can Improve Research on Learning and Bolster School Reform , 2006 .

[39]  Matthew L. Bernacki,et al.  The Effects of Achievement Goals and Self-Regulated Learning Behaviors on Reading Comprehension in Technology-Enhanced Learning Environments. , 2012 .

[40]  Dragan Gasevic,et al.  Analyzing Multimodal Multichannel Data about Self-Regulated Learning with Advanced Learning Technologies: Issues and Challenges , 2019, Comput. Hum. Behav..