The effect of cues for calibration on learners' self-regulated learning through changes in learners' learning behaviour and outcomes

Abstract Literature on blended learning emphasizes the importance of self-regulation for learning in blended learning environments and the role of learners' calibration. Although literature on calibration is clear on its importance for self-regulated learning, it provides inconclusive insight into the effect of support for calibration on learners' self-regulated learning. One under-investigated avenue might be learners' ability to act on the cues provided. In order to establish a more accurate picture of the effect of support for calibration on self-regulated learning, our study investigates whether providing cues for calibration affects learners' self-regulated learning, and whether this effect is different for learners with different metacognitive abilities. We investigate this effect by examining changes in learners' learning behaviour and outcomes. A pre-post design with one control and two experimental conditions was applied in a blended learning environment. Learners in the experimental conditions received either functional validity feedback (F condition) or functional and cognitive validity feedback (FC condition). Learners in the control condition did not receive any cues. Learners' behaviour was analysed using event sequence analysis. Learners' post-test learning scores were subjected to multivariate analysis of covariance, with condition and learners' metacognitive ability as independent variables. The results show a significant and unexpected impact of condition and learners' metacognitive abilities on learners' learning behaviour and outcomes. This manuscript discusses the unexpected results in terms of their theoretical and practical implications and provides recommendations for future research. We conclude that if cues for calibration are provided through functional and cognitive validity feedback, learners’ calibration capabilities will increase. Yet, we hypothesize that for this to result in goal-directed self-regulated learning, learners might need to be guided in how to apply the cognitive and metacognitive strategies needed.

[1]  Shuchi Grover,et al.  A Framework for Using Hypothesis-Driven Approaches to Support Data-Driven Learning Analytics in Measuring Computational Thinking in Block-Based Programming Environments , 2017, ACM Trans. Comput. Educ..

[2]  Geraldine Clarebout,et al.  The complexity of tool use in computer-based learning environments , 2009 .

[3]  M. Veenman,et al.  The on-line assessment of metacognitive skills in a computerized learning environment , 2014 .

[4]  Jeffrey D. Karpicke,et al.  Improving Self-Regulated Learning With a Retrieval Practice Intervention , 2018, Journal of experimental psychology. Applied.

[5]  P. Evans.  Government action, social capital and development: Reviewing the evidence on synergy , 1996 .

[6]  Elizabeth Popelka,et al.  Improving the accuracy of middle school students' self-assessment, peer assessment, and mathematics achievement. , 2015 .

[7]  Dirk T. Tempelaar,et al.  In search for the most informative data for feedback generation: Learning analytics in a data-rich context , 2015, Comput. Hum. Behav..

[8]  Gilbert Ritschard,et al.  Analyzing and Visualizing State Sequences in R with TraMineR , 2011 .

[9]  Kathy A. Paulson Gjerde,et al.  The Impact of Process vs. Outcome Feedback on Student Performance and Perceptions. , 2017 .

[10]  M. Doherty,et al.  Effects of cognitive feedback on performance. , 1989 .

[11]  Tung Bui,et al.  Supporting cognitive feedback using system dynamics: A demand model of the global system of mobile telecommunication , 1996, Decis. Support Syst..

[12]  P. Alexander,et al.  Looking down the road: Future directions for research on depth and regulation of strategic processing. , 2018, The British journal of educational psychology.

[13]  E. Mory.  Feedback research revisited. , 2004 .

[14]  Bernard P. Veldkamp,et al.  The effect of regulation feedback in a computer-based formative assessment on information problem solving , 2015, Comput. Educ..

[15]  Karen M. Zabrucky,et al.  Contributions of metacognitive and self-regulated learning theories to investigations of calibration of comprehension , 2009 .

[16]  J. Broadbent,et al.  Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review , 2015, Internet High. Educ..

[17]  George Siemens,et al.  Let’s not forget: Learning analytics are about learning , 2015 .

[18]  Olusola Adesope,et al.  Exploring Effectiveness and Moderators of Language Learning Strategy Instruction on Second Language and Self-Regulated Learning Outcomes , 2017 .

[19]  Daniel L. Dinsmore,et al.  Focusing the Conceptual Lens on Metacognition, Self-regulation, and Self-regulated Learning , 2008 .

[20]  Nancy Eisenberg,et al.  Self-Regulation and Reading Achievement , 2016 .

[21]  Jan Elen,et al.  Adults’ Self-Regulatory Behaviour Profiles in Blended Learning Environments and Their Implications for Design , 2018, Technology, Knowledge and Learning.

[22]  Linda Bol,et al.  Explaining calibration accuracy in classroom contexts: the effects of incentives, reflection, and explanatory style , 2008 .

[23]  Nickolas J. G. Falkner,et al.  Understanding the Effects of Lecturer Intervention on Computer Science Student Behaviour , 2018, ITiCSE-WGR.

[24]  B. Zimmerman,et al.  Self-regulated learning and academic achievement : theoretical perspectives , 2001 .

[25]  Gregory Schraw.  A conceptual analysis of five measures of metacognitive monitoring , 2009 .

[26]  Josef Frysak,et al.  Feedback Mechanisms for Decision Support Systems: A Literature Review , 2017, WorldCIST.

[27]  Li Cao,et al.  A comparison of high and low achieving students on self-regulated learning variables , 2016 .

[28]  J. Beishuizen,et al.  The relation between intellectual and metacognitive skills from a developmental perspective , 2004 .

[29]  B. Zimmerman,et al.  Self-regulated learning and academic achievement: Theory, research, and practice. , 1989 .

[30]  Karen D. Könings,et al.  Barriers to the uptake and use of feedback in the context of summative assessment , 2015, Advances in health sciences education : theory and practice.

[31]  Nancy J. Stone,et al.  Exploring the Relationship between Calibration and Self-Regulated Learning , 2000 .

[32]  Geraldine Clarebout,et al.  Tool-use in a blended undergraduate course: In Search of user profiles , 2011, Comput. Educ..

[33]  Aimee A. Callender,et al.  Improving metacognition in the classroom through instruction, training, and feedback , 2016 .

[34]  Marek Hatala,et al.  Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes , 2016, J. Learn. Anal..

[35]  Ruth H. Maki,et al.  Individual Differences in Absolute and Relative Metacomprehension Accuracy. , 2005 .

[36]  James R. Segedy,et al.  Adaptive Scaffolds in Open-Ended Computer-Based Learning Environments , 2014 .

[37]  R. Azevedo,et al.  A Theoretical Review of Winne and Hadwin’s Model of Self-Regulated Learning: New Perspectives and Directions , 2007 .

[38]  Erin D. Besser.  Exploring the role of feedback and its impact within a Digital Badge system from multiple perspectives: A case study of preservice teachers , 2016 .

[39]  B. Zimmerman,et al.  Assessing Self-Regulation as a Cyclical, Context-Specific Phenomenon: Overview and Analysis of SRL Microanalytic Protocols , 2012 .

[40]  Lorin W. Anderson,et al.  OBJECTIVES, EVALUATION, AND THE IMPROVEMENT OF EDUCATION , 2005 .

[41]  Bram De Wever,et al.  Blended learning in adult education: towards a definition of blended learning , 2015 .

[42]  Patricia A. Alexander,et al.  Calibration: What Is It and Why It Matters? An Introduction to the Special Issue on Calibrating Calibration. , 2013 .

[43]  Woon Chia Liu,et al.  Academic self-concept: A cross-sectional study of grade and gender differences in a Singapore secondary school , 2005 .

[44]  S. Järvelä,et al.  Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand , 2016 .

[45]  David Fonseca,et al.  Relationship between student profile, tool use, participation, and academic performance with the use of Augmented Reality technology for visualized architecture models , 2014, Comput. Hum. Behav..

[46]  R. Mayer.  Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. , 2004, The American psychologist.

[47]  Maria Bannert,et al.  Short- and long-term effects of students' self-directed metacognitive prompts on navigation behavior and learning performance , 2015, Comput. Hum. Behav..

[48]  Dragan Gasevic,et al.  Relevance of learning analytics to measure and support students' learning in adaptive educational technologies , 2017, LAK.

[49]  Johannes Hartig,et al.  Understanding and Using Feedback--A Review of Empirical Studies Concerning Feedback from External Evaluations to Teachers. , 2013 .

[50]  Paul A. Kirschner,et al.  Changing learning behaviour: Self-efficacy and goal orientation in PBL groups in higher education , 2016 .

[51]  Wim Westera,et al.  Effects of isolated versus combined learning enactments in an online course , 2017 .

[52]  J. Eccles,et al.  Motivational beliefs, values, and goals. , 2002, Annual review of psychology.

[53]  P. Winne,et al.  Feedback and Self-Regulated Learning: A Theoretical Synthesis , 1995 .

[54]  Griet Lust.  Opening the Black Box. Students' Tool-use within a Technology-Enhanced learning environment: An Ecological-Valid Approach , 2012 .

[55]  Wolfgang Lutz,et al.  Improving the efficiency of psychological treatment using outcome feedback technology. , 2017, Behaviour research and therapy.

[56]  P. Pintrich.  The Role of Metacognitive Knowledge in Learning, Teaching, and Assessing , 2002 .

[57]  Philip H. Winne,et al.  The Role of Calibration Bias and Performance Feedback in Achievement Goal Regulation , 2016 .

[58]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[59]  Seung Youn Chyung,et al.  The effects of conceptual tempo and computer-assisted instructional environment on college students' self-regulated learning and academic achievement , 1996 .

[60]  A. Hartley,et al.  Age differences and similarities in the effects of cues and prompts. , 1990, Journal of experimental psychology. Human perception and performance.

[61]  P. Christopher Earley,et al.  IMPACT OF PROCESS AND OUTCOME FEEDBACK ON THE RELATION OF GOAL SETTING TO TASK PERFORMANCE , 1990 .

[62]  W. C. Liu,et al.  A longitudinal study of students' academic self-concept in a streamed setting: the Singapore context. , 2005, The British journal of educational psychology.

[63]  Steven Vos,et al.  Authentic learning tasks within physical education teacher education : redesigning the curriculum , 2016 .

[64]  A. Elliot,et al.  A 2 X 2 achievement goal framework. , 2001, Journal of personality and social psychology.

[65]  David J. Therriault,et al.  Accuracy of metacognitive monitoring affects learning of texts. , 2003 .

[66]  A. Elliot,et al.  A HIERARCHICAL MODEL OF APPROACH AND AVOIDANCE ACHIEVEMENT MOTIVATION , 1997 .

[67]  Popa Daniela,et al.  The Relationship Between Self-Regulation, Motivation And Performance At Secondary School Students , 2015 .

[68]  M. Steinhauser,et al.  Top-down control over feedback processing: The probability of valid feedback affects feedback-related brain activity , 2017, Brain and Cognition.

[69]  R. Azevedo,et al.  Scaffolding Self-regulated Learning and Metacognition – Implications for the Design of Computer-based Scaffolds , 2005 .

[70]  Philip H. Winne,et al.  Studying as self-regulated learning. , 1998 .

[71]  K. Eva,et al.  Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes , 2011, Advances in health sciences education : theory and practice.

[72]  Gilbert Ritschard,et al.  Analyse de séquences d'événements avec TraMineR , 2010, EGC.

[73]  Fabienne M. Van der Kleij,et al.  Effects of Feedback in a Computer-Based Learning Environment on Students’ Learning Outcomes , 2013 .

[74]  Bart Rienties,et al.  The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules , 2016, Comput. Hum. Behav..

[75]  Philip H. Winne,et al.  Exploring students’ calibration of self reports about study tactics and achievement , 2002 .

[76]  Daniel L Dinsmore,et al.  The intersection between depth and the regulation of strategy use. , 2018, The British journal of educational psychology.

[77]  Eduardo E. Veas,et al.  Finding traces of self-regulated learning in activity streams , 2018, LAK.

[78]  John Dunlosky,et al.  Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention , 2012 .

[79]  M. Puustinen,et al.  Models of Self-regulated Learning: A review , 2001 .

[80]  J. Archer,et al.  State of the science in health professional education: effective feedback , 2010, Medical education.

[81]  Bart Rienties,et al.  "Scaling up" learning design: impact of learning design activities on LMS behavior and performance , 2015, LAK.

[82]  Gilbert Ritschard,et al.  Classer, discriminer et visualiser des séquences d'événements , 2010, EGC.

[83]  W. McGaghie,et al.  Variables that affect the process and outcome of feedback, relevant for medical training: a meta‐review , 2015, Medical education.

[84]  Monique Snoeck,et al.  Cognitive Feedback and Behavioral Feedforward Automation Perspectives for Modeling and Validation in a Learning Context , 2016, MODELSWARD.

[85]  A. Elliot,et al.  Handbook of Competence and Motivation , 2013 .

[86]  Dragan Gasevic,et al.  Learning analytics to unveil learning strategies in a flipped classroom , 2017, Internet High. Educ..

[87]  Meagan C. Arrastia-Chisholm,et al.  Using Reflection to Increase Self-Regulation Among Pre-Service Teachers , 2018 .

[88]  Chunliang Yang,et al.  The Forward Testing Effect on Self-Regulated Study Time Allocation and Metamemory Monitoring , 2017, Journal of experimental psychology. Applied.

[89]  Douglas J. Hacker,et al.  A Comparison of the Effects of Practice Tests and Traditional Review on Performance and Calibration , 2001 .

[90]  John Dunlosky,et al.  Four Cornerstones of Calibration Research: Why Understanding Students' Judgments Can Improve Their Achievement , 2013 .

[91]  Robert M. Klassen,et al.  A Question of Calibration: A Review of the Self-Efficacy Beliefs of Students with Learning Disabilities , 2002 .

[92]  J. Sweller,et al.  Domain-Specific Knowledge and Why Teaching Generic Skills Does Not Work , 2014 .

[93]  C. Wolters,et al.  Examining the relations of time management and procrastination within a model of self-regulated learning , 2017 .

[94]  B. Zimmerman.  From Cognitive Modeling to Self-Regulation: A Social Cognitive Career Path , 2013 .

[95]  T. Levine,et al.  Computer use, confidence, attitudes, and knowledge : A causal analysis , 1998 .

[96]  J. Elshout,et al.  The generality vs domain-specificity of metacognitive skills in novice learning across domains. , 1997 .

[97]  Valerie J. Shute,et al.  The interplay of game elements with psychometric qualities, learning, and enjoyment in game-based assessment , 2015, Comput. Educ..

[98]  J. Hattie,et al.  The Power of Feedback , 2007 .

[99]  Mieke Brekelmans,et al.  Individual differences in student teachers' self-regulated learning: An examination of regulation configurations in relation to conceptions of learning to teach , 2014 .

[100]  P. Pintrich.  The role of goal orientation in self-regulated learning. , 2000 .

[101]  Zdenek Zdráhal,et al.  Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment , 2013, LAK '13.

[102]  Anique B. H. de Bruin,et al.  Bridging Cognitive Load and Self-Regulated Learning Research: A complementary approach to contemporary issues in educational research , 2017 .

[103]  Tabassum Rashid,et al.  Technology use, self-directed learning, student engagement and academic performance: Examining the interrelations , 2016, Comput. Hum. Behav..