Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation

Abstract Technological advancements have generated a strong interest in exploring learner behavior data through learning analytics to provide both learner and instructor with process-oriented feedback in the form of dashboards. However, little is known about the typology of dashboard feedback relevant for different learning goals, learners and teachers. While most dashboards and the feedback that they give are based only on learner performance indicators, research shows that effective feedback needs also to be grounded in the regulatory mechanisms underlying learning processes and an awareness of the learner's learning goals. The design artefact presented in this article uses a conceptual model that visualizes the relationships between dashboard design and the learning sciences to provide cognitive and behavioral process-oriented feedback to learners and teachers to support regulation of learning. A practical case example is given that demonstrates how the ideas presented in the paper can be deployed in the context of a learning dashboard. The case example uses several analytics/visualization techniques based on empirical evidence from earlier research that successfully tested these techniques in various learning contexts.

[1]  Alexander Mikroyannidis,et al.  weSPOT: A Personal and Social Approach to Inquiry-Based Learning , 2013, J. Univers. Comput. Sci..

[2]  Dennis Zielke,et al.  Design and Implementation of a Learning Analytics Toolkit for Teachers , 2012, J. Educ. Technol. Soc..

[3]  Monique Snoeck,et al.  Process-mining enabled feedback: "Tell me what I did wrong" vs. "tell me how to do it right" , 2016, Comput. Hum. Behav..

[4]  John Mylopoulos,et al.  Enterprise, Business-Process and Information Systems Modeling , 2012, Lecture Notes in Business Information Processing.

[5]  Rebecca Ferguson,et al.  Research Evidence on the Use of Learning Analytics: Implications for Education Policy , 2016 .

[6]  Erik Duval,et al.  Learning Analytics Dashboard Applications , 2013 .

[7]  Philip H. Winne,et al.  Metacognition and Computer-Supported Collaborative Learning , 2013 .

[8]  María Jesús Rodríguez-Triana,et al.  Understanding learning at a glance: an overview of learning dashboard studies , 2016, LAK.

[9]  A. Carr-Chellman,et al.  Instructional Systems, Learning Sciences, Design, Technology: A More Complete Pathway , 2004 .

[10]  D. Sadler Formative assessment and the design of instructional systems , 1989 .

[11]  Hanna Järvenoja,et al.  Socially shared regulation of learning and participation in social interaction in collaborative learning , 2017 .

[12]  Kennon M. Sheldon,et al.  Avoidance achievement motivation: a personal goals analysis. , 1997, Journal of personality and social psychology.

[13]  Matthew L. Bernacki,et al.  The Effects of Achievement Goals and Self-Regulated Learning Behaviors on Reading Comprehension in Technology-Enhanced Learning Environments. , 2012 .

[14]  Maria Bannert,et al.  Exploring regulatory processes during a computer-supported collaborative learning task using process mining , 2012, Comput. Hum. Behav..

[15]  Anatoliy A. Gruzd,et al.  Connected scholars: Examining the role of social media in research practices of faculty using the UTAUT model , 2012, Comput. Hum. Behav..

[16]  Erik Duval,et al.  Learning dashboards: an overview and future research opportunities , 2013, Personal and Ubiquitous Computing.

[17]  Steven Lonn,et al.  Investigating student motivation in the context of a learning analytics intervention during a summer bridge program , 2015, Comput. Hum. Behav..

[18]  Gijsbert Erkens,et al.  Group awareness of social and cognitive performance in a CSCL environment: Effects of a peer feedback and reflection tool , 2011, Comput. Hum. Behav..

[19]  P. Pintrich Multiple Goals, Multiple Pathways: The Role of Goal Orientation in Learning and Achievement. , 2000 .

[20]  Paul A. Kirschner,et al.  Effects of feedback on collaborative writing in an online learning environment , 2013 .

[21]  Monique Snoeck,et al.  Design and evaluation of feedback automation architecture: Enriching model execution with feedback to support testing of semantic conformance between models and requirements , 2016, International Conference on Model-Driven Engineering and Software Development.

[22]  Oana Velcu,et al.  A review of dashboards in performance management: Implications for design and research , 2012, Int. J. Account. Inf. Syst..

[23]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[24]  Erik Duval,et al.  Towards balanced discussions in the classroom using ambient information visualisations , 2017 .

[25]  Carlos Delgado Kloos,et al.  Evaluating Student-Facing Learning Dashboards of Affective States , 2017, EC-TEL.

[26]  A. Hamilton Church,et al.  The Principles of Management , 2020 .

[27]  Erik Duval,et al.  Learning Dashboards & Learnscapes , 2012, CHI 2012.

[28]  J. Hattie,et al.  The Power of Feedback , 2007 .

[29]  Chia-Lun Lo,et al.  Developing early warning systems to predict students' online learning performance , 2014, Comput. Hum. Behav..

[30]  C. Dweck,et al.  A social-cognitive approach to motivation and personality , 1988 .

[31]  P. Blumenfeld Classroom Learning and Motivation: Clarifying and Expanding Goal Theory , 1992 .

[32]  Paul A. Kirschner,et al.  Awareness of group performance in a CSCL-environment: Effects of peer feedback and reflection , 2010, Comput. Hum. Behav..

[33]  Monique Snoeck,et al.  Effects of Simulation on Novices' Understanding of the Concept of Inheritance in Conceptual Modeling , 2015, ER Workshops.

[34]  Erik Mannens,et al.  Data Harvesting, Curation and Fusion Model to Support Public Service Recommendations for e-Governments , 2018, MODELSWARD.

[35]  Monique Snoeck,et al.  Cognitive Feedback and Behavioral Feedforward Automation Perspectives for Modeling and Validation in a Learning Context , 2016, MODELSWARD.

[36]  Paul A. Kirschner,et al.  Ten Steps to Complex Learning: A Systematic Approach to Four-Component Instructional Design , 2007 .

[37]  B. Zimmerman Motivational Sources and Outcomes of Self-Regulated Learning and Performance , 2011 .

[38]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[39]  A. Elliot,et al.  A 2 X 2 achievement goal framework. , 2001, Journal of personality and social psychology.

[40]  Monique Snoeck,et al.  Assessing the influence of feedback-inclusive rapid prototyping on understanding the semantics of parallel UML statecharts by novice modellers , 2017, Inf. Softw. Technol..

[41]  E. A. Locke,et al.  Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. , 2002, The American psychologist.

[42]  F. Kerlinger Behavioral Research: A Conceptual Approach , 1978 .

[43]  Erik Duval,et al.  WESPOT: Inquiry based learning meets learning analytics , 2012 .

[44]  Ruben Verborgh,et al.  Supporting Sustainable Publishing and Consuming of Live Linked Time Series Streams , 2018, ESWC.

[45]  Monique Snoeck,et al.  Process mining analysis of conceptual modeling behavior of novices - empirical study using JMermaid modeling and experimental logging environment , 2014, Comput. Hum. Behav..

[46]  Armin Grunwald,et al.  Technology Assessment for Responsible Innovation , 2014 .

[47]  Alphonse Chapanis Evaluating usability , 1991 .

[48]  Nian-Shing Chen,et al.  Use of Dashboards and Visualization Techniques to Support Teacher Decision Making , 2015 .

[49]  A. Irons,et al.  Enhancing Learning through Formative Assessment and Feedback , 2007 .

[50]  Sally Sieloff Magnan,et al.  Research Design: Qualitative and Quantitative Approaches , 1997 .

[51]  S. Järvelä,et al.  New Frontiers: Regulating Learning in CSCL , 2013 .

[52]  Sanna Järvelä,et al.  Self-Regulation, Co-Regulation, and Shared Regulation in Collaborative Learning Environments , 2017 .

[53]  Dragan Gasevic,et al.  Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success , 2016, Internet High. Educ..

[54]  Katrien Verbert,et al.  Review of Research on Student-Facing Learning Analytics Dashboards and Educational Recommender Systems , 2017, IEEE Transactions on Learning Technologies.

[55]  Fred D. Davis Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[56]  P. Winne,et al.  Feedback and Self-Regulated Learning: A Theoretical Synthesis , 1995 .

[57]  Marilyn E. Gist,et al.  EFFECTS OF SELF‐EFFICACY and GOAL‐ORIENTATION TRAINING ON NEGOTIATION SKILL MAINTENANCE: WHAT ARE THE MECHANISMS? , 1997 .

[58]  Mihail C. Roco,et al.  Societal implications of nanoscience and nanotechnology: Maximizing human benefit , 2005 .

[59]  Monique Snoeck,et al.  Assessing the effectiveness of feedback enabled simulation in teaching conceptual modeling , 2014, Comput. Educ..

[60]  Monique Snoeck,et al.  Technology-Enhanced Support for Learning Conceptual Modeling , 2012, BMMDS/EMMSAD.

[61]  Mohamed Jemni,et al.  Role of personality in computer based learning , 2016, Comput. Hum. Behav..

[62]  Welko Tomic BEHAVIORISM AND COGNITIVISM IN EDUCATION , 1993 .

[63]  Ibis M. Álvarez,et al.  The value of feedback in improving collaborative writing assignments in an online learning environment , 2012 .

[64]  Carlos Delgado Kloos,et al.  Evaluating usability of affective state visualizations using AffectVis, an affect-aware dashboard for students , 2017 .

[65]  Monique Snoeck,et al.  Do we need to teach testing skills in courses on requirements engineering and modelling , 2014 .

[66]  Arie Rip,et al.  Constructive Technology Assessment and Socio-Technical Scenarios. , 2008 .

[67]  Arie Rip,et al.  Constructive technology assessment and the methodology of insertion , 2019, Nanotechnology and Its Governance.

[68]  Sanna Järvelä,et al.  Conceptual framework for feedback automation and personalization for designing learning analytics dashboards , 2016 .

[69]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[70]  E. A. Locke,et al.  New developments in and directions for goal-setting research. , 2007 .

[71]  Judy Kay,et al.  Narcissus: Group and Individual Models to Support Small Group Work , 2009, UMAP.

[72]  Frederik Anseel,et al.  The influence of mastery‐avoidance goals on performance improvement , 2009 .

[73]  A. Elliot Approach and avoidance motivation and achievement goals , 1999 .

[74]  Gayane Sedrakyan Process-oriented feedback perspectives based on feedback-enabled simulation and learning process data analytics. , 2016 .

[75]  Yeonjeong Park,et al.  Development of the Learning Analytics Dashboard to Support Students' Learning Performance , 2015, J. Univers. Comput. Sci..

[76]  Vive Kumar,et al.  Supporting Self-Regulated Learning with gStudy Software: The Learning Kit Project ** , 2006 .

[77]  B. Zimmerman Self-Regulated Learning and Academic Achievement: An Overview , 1990 .

[78]  Philip H. Winne,et al.  Studying as self-regulated learning. , 1998 .

[79]  Paul A. Kirschner,et al.  How does feedback and peer feedback affect collaborative writing in a virtual learning environment , 2011 .