Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-wild

Multimodal Learning Analytics (MMLA) innovations make use of rapidly evolving sensing and artificial intelligence algorithms to collect rich data about learning activities that unfold in physical spaces. The analysis of these data is opening exciting new avenues for both studying and supporting learning. Yet, practical and logistical challenges commonly appear while deploying MMLA innovations ”in-the-wild”. These can span from technical issues related to enhancing the learning space with sensing capabilities, to the increased complexity of teachers’ tasks. These practicalities have been rarely investigated. This paper addresses this gap by presenting a set of lessons learnt from a 2-year human-centred MMLA in-the-wild study conducted with 399 students and 17 educators in the context of nursing education. The lessons learnt were synthesised into topics related to i) technological/physical aspects of the deployment; ii) multimodal data and interfaces; iii) the design process; iv) participation, ethics and privacy; and v) sustainability of the deployment.

[1]  D. Gašević,et al.  METS: Multimodal Learning Analytics of Embodied Teamwork Learning , 2023, LAK.

[2]  D. Gašević,et al.  How Do Teachers Use Dashboards Enhanced with Data Storytelling Elements According to their Data Visualisation Literacy Skills? , 2023, LAK.

[3]  Roberto Martínez Maldonado,et al.  "That Student Should be a Lion Tamer!" StressViz: Designing a Stress Analytics Dashboard for Teachers , 2023, LAK.

[4]  D. Gašević,et al.  HuCETA: A Framework for Human-Centered Embodied Teamwork Analytics , 2023, IEEE Pervasive Computing.

[5]  Olga Viberg,et al.  Designing Culturally Aware Learning Analytics: A Value Sensitive Perspective , 2022, ArXiv.

[6]  Eleanor O'Rourke,et al.  Using Electrodermal Activity Measurements to Understand Student Emotions While Programming , 2022, ICER.

[7]  Roberto Martínez Maldonado,et al.  The role of indoor positioning analytics in assessment of simulation-based learning , 2022, Br. J. Educ. Technol..

[8]  Jiun-Yu Wu,et al.  Deploying multimodal learning analytics models to explore the impact of digital distraction and peer learning on student performance , 2022, Comput. Educ..

[9]  M. Cukurova,et al.  Teachers' trust in AI-powered educational technology and a professional development program to improve it , 2022, Br. J. Educ. Technol..

[10]  Roberto Martínez Maldonado,et al.  Explainable Artificial Intelligence in education , 2022, Comput. Educ. Artif. Intell..

[11]  Angelika Strohmayer,et al.  Forgetting Practices in the Data Sciences , 2022, CHI.

[12]  S. Kahai,et al.  Who needs explanation and when? Juggling explainable AI and user epistemic uncertainty , 2022, Int. J. Hum. Comput. Stud..

[13]  K. Boyer,et al.  Detecting Impasse During Collaborative Problem Solving with Multimodal Learning Analytics , 2022, LAK.

[14]  Roberto Martínez Maldonado,et al.  Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics , 2022, LAK.

[15]  Roberto Martínez Maldonado,et al.  Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling , 2022, LAK.

[16]  S. Uzzo,et al.  Exploring the Utility of Social-Network-Derived Collaborative Opportunity Temperature Readings for Informing Design and Research of Large-Group Immersive Learning Environments , 2022, J. Learn. Anal..

[17]  A. Wise,et al.  What Makes Learning Analytics Research Matter , 2021, J. Learn. Anal..

[18]  M. Cukurova,et al.  An Instrument for Measuring Teachers’ Trust in AI-Based Educational Technology , 2021, LAK.

[19]  H. Drachsler,et al.  Keep Me in the Loop: Real-Time Feedback with Multimodal Data , 2021, International Journal of Artificial Intelligence in Education.

[20]  Cynthia A. D'Angelo,et al.  A New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow MMLA , 2021, J. Learn. Anal..

[21]  D. Gašević,et al.  Exploring students' cognitive and affective states during problem solving through multimodal data: Lessons learned from a programming activity , 2021, J. Comput. Assist. Learn..

[22]  Rachel Dzombak,et al.  Human-Centered AI , 2021, IEEE Pervasive Comput..

[23]  M. Giannakos,et al.  The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review , 2021 .

[24]  Gina Neff,et al.  Bringing Human-Centredness to Technologies for Buildings: An agenda for linking new types of data to the challenge of sustainability , 2021, CHItaly.

[25]  H. Drachsler,et al.  Literature Review on Co-Located Collaboration Modeling Using Multimodal Learning Analytics—Can We Go the Whole Nine Yards? , 2021, IEEE Transactions on Learning Technologies.

[26]  Carolina Guzmán-Valenzuela,et al.  Learning analytics in higher education: a preponderance of analytics but very little learning? , 2021, International Journal of Educational Technology in Higher Education.

[27]  Tobias Ley,et al.  Do Teachers Find Dashboards Trustworthy, Actionable and Useful? A Vignette Study Using a Logs and Audio Dashboard , 2021, Technol. Knowl. Learn..

[28]  Kirsty Kitto,et al.  What Can Analytics for Teamwork Proxemics Reveal About Positioning Dynamics In Clinical Simulations? , 2021, Proc. ACM Hum. Comput. Interact..

[29]  Roberto Martínez Maldonado,et al.  Footprints at School: Modelling In-class Social Dynamics from Students’ Physical Positioning Traces , 2021, LAK.

[30]  Dragan Gasevic,et al.  Challenges and opportunities of multimodal data in human learning: The computer science students' perspective , 2021, J. Comput. Assist. Learn..

[31]  Jesús Ureña,et al.  LOCATE-US: Indoor Positioning for Mobile Devices Using Encoded Ultrasonic Signals, Inertial Sensors and Graph-Matching , 2021, Sensors.

[32]  Anders I. Mørch,et al.  Overcoming Challenges to the Adoption of Learning Analytics at the Practitioner Level: A Critical Analysis of 18 Learning Analytics Frameworks , 2021, Scandinavian Journal of Educational Research.

[33]  Peter Gerjets,et al.  Multimodal Engagement Analysis From Facial Videos in the Classroom , 2021, IEEE Transactions on Affective Computing.

[34]  Xiaodi Huang,et al.  Multimodal Data Fusion in Learning Analytics: A Systematic Review , 2020, Sensors.

[35]  Michail N. Giannakos,et al.  Utilizing Multimodal Data Through fsQCA to Explain Engagement in Adaptive Learning , 2020, IEEE Transactions on Learning Technologies.

[36]  Lucrezia Crescenzi‐Lanna,et al.  Multimodal Learning Analytics research with young children: A systematic review , 2020, Br. J. Educ. Technol..

[37]  Muhterem Dindar,et al.  What does physiological synchrony reveal about metacognitive experiences and group performance? , 2020, Br. J. Educ. Technol..

[38]  Michail N. Giannakos,et al.  The promise and challenges of multimodal learning analytics , 2020, Br. J. Educ. Technol..

[39]  Fabián Riquelme,et al.  Where are You? Exploring Micro-Location in Indoor Learning Environments , 2020, IEEE Access.

[40]  Davinia Hernández Leo,et al.  Enhancing consent forms to support participant decision making in multimodal learning data research , 2020, Br. J. Educ. Technol..

[41]  Kshitij Sharma,et al.  Multimodal data capabilities for learning: What can multimodal data tell us about learning? , 2020, Br. J. Educ. Technol..

[42]  X. Ochoa,et al.  Controlled evaluation of a multimodal system to improve oral presentation skills in a real learning setting , 2020, Br. J. Educ. Technol..

[43]  Omid Noroozi,et al.  Multimodal data indicators for capturing cognitive, motivational, and emotional learning processes: A systematic literature review , 2020, Education and Information Technologies.

[44]  Roberto Martínez Maldonado,et al.  From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics , 2020, CHI.

[45]  Brian Magerko,et al.  What is AI Literacy? Competencies and Design Considerations , 2020, CHI.

[46]  Sidney K. D'Mello,et al.  Focused or stuck together: multimodal patterns reveal triads' performance in collaborative problem solving , 2020, LAK.

[47]  Neil Selwyn,et al.  What's the Problem with Learning Analytics? , 2019, J. Learn. Anal..

[48]  Britta Hofmann,et al.  Beyond HCI and CSCW: Challenges and Useful Practices Towards a Human-Centred Vision of AI and IA , 2019, HTTF.

[49]  ForteAndrea,et al.  Reliability and Inter-rater Reliability in Qualitative Research , 2019 .

[50]  Andrea Forte,et al.  Reliability and Inter-rater Reliability in Qualitative Research , 2019, Proc. ACM Hum. Comput. Interact..

[51]  Omid Noroozi,et al.  Multimodal data to design visual learning analytics for understanding regulation of learning , 2019, Comput. Hum. Behav..

[52]  Amy Ogan,et al.  Reframing classroom sensing , 2019, Interactions.

[53]  Michail N. Giannakos,et al.  Learning Analytics for Learning Design: A Systematic Literature Review of Analytics-Driven Design to Enhance Learning , 2019, IEEE Transactions on Learning Technologies.

[54]  Franceska Xhakaj,et al.  EduSense: Practical Classroom Sensing at Scale , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[55]  Rodolfo Villarroel,et al.  Introducing Low-Cost Sensors into the Classroom Settings: Improving the Assessment in Agile Practices with Multimodal Learning Analytics , 2019, Sensors.

[56]  June Ahn,et al.  Designing in Context: Reaching Beyond Usability in Learning Analytics Dashboard Design , 2019, J. Learn. Anal..

[57]  Rebecca Ferguson,et al.  Human-Centred Learning Analytics , 2019, J. Learn. Anal..

[58]  Aaron Marcus,et al.  Seven HCI Grand Challenges , 2019, Int. J. Hum. Comput. Interact..

[59]  Dragan Gasevic,et al.  Analyzing Multimodal Multichannel Data about Self-Regulated Learning with Advanced Learning Technologies: Issues and Challenges , 2019, Comput. Hum. Behav..

[60]  Roberto Martínez Maldonado,et al.  Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data , 2019, CHI.

[61]  Steve Benford,et al.  Seeing with New Eyes: Designing for In-the-Wild Museum Gifting , 2019, CHI.

[62]  Justin Dauwels,et al.  Technologies for automated analysis of co-located, real-life, physical learning spaces: Where are we now? , 2019, LAK.

[63]  Jeffrey Heer Agency plus automation: Designing artificial intelligence into interactive systems , 2019, Proceedings of the National Academy of Sciences.

[64]  Sharon L. Oviatt,et al.  Ten Opportunities and Challenges for Advancing Student-Centered Multimodal Learning Analytics , 2018, ICMI.

[65]  Bertrand Schneider,et al.  Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups , 2018, Int. J. Comput. Support. Collab. Learn..

[66]  María Jesús Rodríguez-Triana,et al.  A Review of Multimodal Learning Analytics Architectures , 2018, 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT).

[67]  Xiaodong Liu,et al.  Towards Modelling and Reasoning About Uncertain Data of Sensor Measurements for Decision Support in Smart Spaces , 2018, 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC).

[68]  Natsuki Nakayama,et al.  Heart rate variability can clarify students’ level of stress during nursing simulation , 2018, PloS one.

[69]  Kshitij Sharma,et al.  Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data , 2018, J. Comput. Assist. Learn..

[70]  Simon Buckingham Shum,et al.  Embracing imperfection in learning analytics , 2018, LAK.

[71]  Xavier Ochoa,et al.  The RAP system: automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors , 2018, LAK.

[72]  Nazmus Saquib,et al.  Sensei , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[73]  N. Selwyn,et al.  Doing data differently? Developing personal data tactics and strategies amongst young mobile media users , 2018, Big Data Soc..

[74]  R. Schreiber,et al.  Teaching nurses teamwork: Integrative review of competency-based team training in nursing education. , 2017, Nurse education in practice.

[75]  Kasper Hornbæk,et al.  Technology Acceptance and User Experience , 2017, ACM Trans. Comput. Hum. Interact..

[76]  Deborah E. White,et al.  Thematic Analysis , 2017 .

[77]  Mutlu Cukurova,et al.  Estimation of Success in Collaborative Learning Based on Multimodal Learning Analytics Features , 2017, 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT).

[78]  Bertrand Schneider,et al.  Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning , 2016, ACM Trans. Comput. Hum. Interact..

[79]  Xavier Ochoa,et al.  Editorial: Augmenting Learning Analytics with Multimodal Sensory Data​ , 2016, J. Learn. Anal..

[80]  Noshir Contractor,et al.  Evaluating the Paper-to-Screen Translation of Participant-Aided Sociograms with High-Risk Participants , 2016, CHI.

[81]  Rebecca Ferguson,et al.  Guest Editorial: Ethics and Privacy in Learning Analytics , 2016, J. Learn. Anal..

[82]  Andrew Olney,et al.  Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms , 2015, ICMI.

[83]  Steve Benford,et al.  The Ethical Implications of HCI's Turn to the Cultural , 2015, TCHI.

[84]  Marcelo Worsley,et al.  Leveraging multimodal learning analytics to differentiate student learning strategies , 2015, LAK.

[85]  D. Nestel,et al.  Expert practice of video-assisted debriefing: an Australian qualitative study , 2015 .

[86]  Xavier Ochoa,et al.  Estimation of Presentations Skills Based on Slides and Audio Features , 2014, MLA@ICMI.

[87]  Nadir Weibel,et al.  ICMI 2013 grand challenge workshop on multimodal learning analytics , 2013, ICMI '13.

[88]  M. Megel,et al.  High-Fidelity Simulation: How Are We Using the Videos? , 2013 .

[89]  John Rooksby,et al.  Wild in the Laboratory: A Discussion of Plans and Situated Actions , 2013, TCHI.

[90]  Yvonne Rogers,et al.  Introduction to the Special Issue of “The Turn to The Wild” , 2013, TCHI.

[91]  Patrick Olivier,et al.  Leaving the wild: lessons from community technology handovers , 2013, CHI.

[92]  Paulo Blikstein,et al.  Multimodal learning analytics , 2013, LAK '13.

[93]  Marek Hatala,et al.  Factors influencing beliefs for adoption of a learning analytics tool: An empirical study , 2013, Comput. Educ..

[94]  Ivan Marsic,et al.  Teamwork Errors in Trauma Resuscitation , 2012, TCHI.

[95]  George Siemens,et al.  Learning analytics: envisioning a research discipline and a domain of practice , 2012, LAK.

[96]  A. Graesser,et al.  Dynamics of affective states during complex learning , 2012 .

[97]  B. Penders,et al.  Research in the wild , 2007, Nature.

[98]  J. Fereday,et al.  Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development , 2006 .

[99]  Gregory D. Abowd,et al.  Lessons learned from eClass: Assessing automated capture and access in the classroom , 2004, TCHI.

[100]  R. Glavin,et al.  Low‐ to high‐fidelity simulation – a continuum of medical education? , 2003, Medical education.

[101]  Gregory D. Abowd,et al.  Charting past, present, and future research in ubiquitous computing , 2000, TCHI.

[102]  Michael Boyle,et al.  PDAs and shared public displays: Making personal information public, and public information personal , 1999, Personal Technologies.

[103]  Ben Shneiderman,et al.  Direct manipulation vs. interface agents , 1997, INTR.

[104]  Fred D. Davis Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[105]  D. Gašević,et al.  Physiological Synchrony and Arousal as Indicators of Stress and Learning Performance in Embodied Collaborative Learning , 2023, AIED.

[106]  Shashi Kant Shankar,et al.  Multimodal Learning Analytics Research in the Wild: Challenges and Their Potential Solutions , 2023, CrossMMLA@LAK.

[107]  Olga C. Santos,et al.  Towards Exploring Stress Reactions in Teamwork using Multimodal Physiological Data , 2021, MAIED@AIED.

[108]  Y. Rogers,et al.  Moving HCI Outdoors: Lessons Learned from Conducting Research in the Wild , 2020 .

[109]  E. Churchill,et al.  Introduction to Human-Building Interaction (HBI): Interfacing HCI with Architecture and Urban Design , 2020 .

[110]  Silvia Miksch,et al.  Characterizing Guidance in Visual Analytics , 2017, IEEE Transactions on Visualization and Computer Graphics.

[111]  Zhan Zhang,et al.  Constructing Awareness Through Speech, Gesture, Gaze and Movement During a Time-Critical Medical Task , 2015, ECSCW.

[112]  Kirk Goldsberry,et al.  CourtVision : New Visual and Spatial Analytics for the NBA , 2012 .

[113]  BMC Medical Research Methodology , 2007 .