Technologies for automated analysis of co-located, real-life, physical learning spaces: Where are we now?

The motivation for this paper is derived from the fact that there has been increasing interest among researchers and practitioners in developing technologies that capture, model and analyze learning and teaching experiences that take place beyond computer-based learning environments. In this paper, we review case studies of tools and technologies developed to collect and analyze data in educational settings, quantify learning and teaching processes and support assessment of learning and teaching in an automated fashion. We focus on pipelines that leverage information and data harnessed from physical spaces and/or integrates collected data across physical and digital spaces. Our review reveals a promising field of physical classroom analysis. We describe some trends and suggest potential future directions. Specifically, more research should be geared towards a) deployable and sustainable data collection set-ups in physical learning environments, b) teacher assessment, c) developing feedback and visualization systems and d) promoting inclusivity and generalizability of models across populations.

[1]  Ibrahima Faye,et al.  A physiological index for situational interest in classroom settings using EEG (preliminary result) , 2017, TENCON 2017 - 2017 IEEE Region 10 Conference.

[2]  Sharon L. Oviatt Problem solving, domain expertise and learning: ground-truth performance results for math data corpus , 2013, ICMI '13.

[3]  Paul Salvador Inventado,et al.  Educational Data Mining and Learning Analytics Potentials and Possibilities for Online Education , 2019 .

[4]  Andrew Olney,et al.  Identifying Teacher Questions Using Automatic Speech Recognition in Classrooms , 2016, SIGDIAL Conference.

[5]  Tzyy-Ping Jung,et al.  Sustained Attention in Real Classroom Settings: An EEG Study , 2017, Front. Hum. Neurosci..

[6]  Pierre Dillenbourg,et al.  Studying Teacher Cognitive Load in Multi-tabletop Classrooms Using Mobile Eye-tracking , 2014, ITS '14.

[7]  Hendrik Drachsler,et al.  From signals to knowledge: A conceptual model for multimodal learning analytics , 2018, J. Comput. Assist. Learn..

[8]  Dinesh Babu Jayagopi,et al.  Predicting student engagement in classrooms using facial behavioral cues , 2017, MIE@ICMI.

[9]  Bertrand Schneider,et al.  Unraveling Students' Interaction Around a Tangible Interface Using Gesture Recognition , 2014, EDM.

[10]  Alejandro Andrade,et al.  Understanding student learning trajectories using multimodal learning analytics within an embodied-interaction learning environment , 2017, LAK.

[11]  Lei Chen,et al.  Using Multimodal Cues to Analyze MLA'14 Oral Presentation Quality Corpus: Presentation Delivery and Slides Quality , 2014, MLA@ICMI.

[12]  Sidney D'Mello,et al.  Data mining and education. , 2015, Wiley interdisciplinary reviews. Cognitive science.

[13]  Pablo Munguia,et al.  Classroom size, activity and attendance: scaling up drivers of learning space occupation , 2018, LAK.

[14]  Fang Chen,et al.  Spoken Interruptions Signal Productive Problem Solving and Domain Expertise in Mathematics , 2015, ICMI.

[15]  Andrew Olney,et al.  Assessing the Dialogic Properties of Classroom Discourse: Proportion Models for Imbalanced Classes , 2017, EDM.

[16]  Andrew Olney,et al.  Automatic Classification of Question & Answer Discourse Segments from Teacher's Speech in Classrooms. , 2015, EDM 2015.

[17]  Arthur C. Graesser,et al.  Modeling Classroom Discourse: Do Models That Predict Dialogic Instruction Properties Generalize across Populations?. , 2015, EDM 2015.

[18]  Joshua A. Danish,et al.  Using Multimodal Learning Analytics to Model Student Behavior: A Systematic Analysis of Epistemological Framing , 2016, J. Learn. Anal..

[19]  Catherine Creech,et al.  Classroom sound can be used to classify teaching practices in college science courses , 2017, Proceedings of the National Academy of Sciences.

[20]  Louis-Philippe Morency,et al.  Multimodal prediction of expertise and leadership in learning groups , 2012, MLA '12.

[21]  Diana Gonzalez,et al.  Using Multimodal Learning Analytics to Identify Patterns of Interactions in a Body-Based Mathematics Activity , 2016 .

[22]  Takashi Ozeki,et al.  Analysis of interactions between lecturers and students , 2018, LAK.

[23]  Xavier Ochoa,et al.  Editorial: Augmenting Learning Analytics with Multimodal Sensory Data​ , 2016, J. Learn. Anal..

[24]  Roberto Martínez Maldonado,et al.  Analytics meet patient manikins: challenges in an authentic small-group healthcare simulation classroom , 2017, LAK.

[25]  Marcelo Worsley,et al.  Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks , 2016, J. Learn. Anal..

[26]  Jian Han Lim,et al.  Automated classroom monitoring with connected visioning system , 2017, 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[27]  Erik Duval,et al.  Learning Analytics for Natural User Interfaces: A Framework, Case Studies and a Maturity Analysis , 2017 .

[28]  David Poeppel,et al.  Brain-to-Brain Synchrony and Learning Outcomes Vary by Student–Teacher Dynamics: Evidence from a Real-world Classroom Electroencephalography Study , 2019, Journal of Cognitive Neuroscience.

[29]  Kate Thompson Using micro-patterns of speech to predict the correctness of answers to mathematics problems: an exercise in multimodal learning analytics , 2013, ICMI '13.

[30]  Ryan Shaun Joazeiro de Baker,et al.  Educational Data Mining and Learning Analytics: Applications to Constructionist Research , 2014, Technology, Knowledge and Learning.

[31]  Justin Dauwels,et al.  Automated Classification of Classroom Climate by Audio Analysis , 2018, International Workshop on Spoken Dialogue Systems Technology.

[32]  Andrew Olney,et al.  Words matter: automatic detection of teacher questions in live classroom discourse using linguistics, acoustics, and context , 2017, LAK.

[33]  Andrew Olney,et al.  Multi-sensor modeling of teacher instructional segments in live classrooms , 2016, ICMI.

[34]  Haohan Li,et al.  Who missed the class? — Unifying multi-face detection, tracking and recognition in videos , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[35]  Saturnino Luz,et al.  Automatic identification of experts and performance prediction in the multimodal math data corpus through analysis of speech interaction , 2013, ICMI '13.

[36]  Sharon L. Oviatt,et al.  Written and multimodal representations as predictors of expertise and problem-solving success in mathematics , 2013, ICMI '13.

[37]  Andrew Olney,et al.  A Study of Automatic Speech Recognition in Noisy Classroom Environments for Automated Dialog Analysis , 2015, AIED.

[38]  Andrew Olney,et al.  Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms , 2015, ICMI.

[39]  Bertrand Schneider,et al.  Detecting Collaborative Dynamics Using Mobile Eye-Trackers , 2016, ICLS.

[40]  Pierre Dillenbourg,et al.  Student motion and it's potential as a classroom performance metric , 2013 .

[41]  Andrew Olney,et al.  Automatic Teacher Modeling from Live Classroom Audio , 2016, UMAP.

[42]  Lucas C. Parra,et al.  EEG in the classroom: Synchronised neural recordings during video presentation , 2016, Scientific Reports.

[43]  Marcelo Worsley,et al.  Using learning analytics to study cognitive disequilibrium in a complex learning environment , 2015, LAK.

[44]  Ismar Silveira,et al.  Deep Learning applied to Learning Analytics and Educational Data Mining: A Systematic Literature Review , 2017 .

[45]  D. Poeppel,et al.  Brain-to-Brain Synchrony Tracks Real-World Dynamic Group Interactions in the Classroom , 2017, Current Biology.

[46]  Alex Pappachen James,et al.  Automated rating of recorded classroom presentations using speech analysis in kazakh , 2017, 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[47]  Lei Chen,et al.  Designing An Automated Assessment of Public Speaking Skills Using Multimodal Cues , 2016, J. Learn. Anal..

[48]  Marcelo Worsley,et al.  (Dis)engagement matters: identifying efficacious learning practices with multimodal learning analytics , 2018, LAK.

[49]  K. Miller,et al.  Where Low and High Inference Data Converge: Validation of CLASS Assessment of Mathematics Instruction Using Mobile Eye Tracking with Expert and Novice Teachers , 2015, International Journal of Science and Mathematics Education.

[50]  Patrick Donnelly,et al.  Automatically Measuring Question Authenticity in Real-World Classrooms , 2018 .

[51]  Mutlu Cukurova,et al.  Supervised machine learning in multimodal learning analytics for estimating success in project-based learning , 2018, J. Comput. Assist. Learn..

[52]  Marcelo Worsley,et al.  What's an Expert? Using Learning Analytics to Identify Emergent Markers of Expertise through Automated Speech, Sentiment and Sketch Analysis , 2011, EDM.

[53]  Xingyu Pan,et al.  Automatic classification of activities in classroom discourse , 2014, Comput. Educ..

[54]  Roberto Martínez Maldonado,et al.  Where Is the Nurse? Towards Automatically Visualising Meaningful Team Movement in Healthcare Education , 2018, AIED.

[55]  Mutlu Cukurova,et al.  Estimation of Success in Collaborative Learning Based on Multimodal Learning Analytics Features , 2017, 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT).

[56]  Jeremy N. Bailenson,et al.  Automatic Detection of Nonverbal Behavior Predicts Learning in Dyadic Interactions , 2014, IEEE Transactions on Affective Computing.

[57]  Xavier Ochoa,et al.  Estimation of Presentations Skills Based on Slides and Audio Features , 2014, MLA@ICMI.

[58]  Arthur C. Graesser,et al.  Domain Independent Assessment of Dialogic Properties of Classroom Discourse , 2014, EDM.

[59]  Marcelo Worsley,et al.  Exploring Behavior Representation for Learning Analytics , 2015, ICMI.

[60]  Pierre Dillenbourg,et al.  Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors , 2016, LAK.

[61]  Doug Clow,et al.  The learning analytics cycle: closing the loop effectively , 2012, LAK.

[62]  Pierre Dillenbourg,et al.  Sleepers' lag - study on motion and attention , 2014, LAK.

[63]  Sharon L. Oviatt,et al.  Written Activity, Representations and Fluency as Predictors of Domain Expertise in Mathematics , 2014, ICMI.

[64]  Xavier Ochoa,et al.  Expertise estimation based on simple multimodal features , 2013, ICMI '13.

[65]  Alejandro Peña-Ayala Review: Educational data mining: A survey and a data mining-based analysis of recent works , 2014 .

[66]  Xavier Ochoa,et al.  Presentation Skills Estimation Based on Video and Kinect Data Analysis , 2014, MLA@ICMI.

[67]  Heather Pon-Barry,et al.  Acoustic-Prosodic Entrainment and Rapport in Collaborative Learning Dialogues , 2014, MLA@ICMI.

[68]  Hong Lu,et al.  Teaching Video Analytics Based on Student Spatial and Temporal Behavior Mining , 2015, ICMR.

[69]  Kshitij Sharma,et al.  Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data , 2018, J. Comput. Assist. Learn..

[70]  Pierre Dillenbourg,et al.  Translating Head Motion into Attention - Towards Processing of Student's Body-Language , 2015, EDM.

[71]  Andrew Olney,et al.  Semi-Automatic Detection of Teacher Questions from Human-Transcripts of Audio in Live Classrooms , 2016, EDM.

[72]  Sanna Järvelä,et al.  Investigating collaborative learning success with physiological coupling indices based on electrodermal activity , 2016, LAK.

[73]  Olga C. Santos,et al.  Physical learning analytics: a multimodal perspective , 2018, LAK.

[74]  Xavier Ochoa,et al.  The RAP system: automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors , 2018, LAK.

[75]  Kevin F. Miller,et al.  Using the LENA in teacher training: Promoting student involvement through automated feedback , 2013 .

[76]  Shuchi Grover,et al.  Multimodal analytics to study collaborative problem solving in pair programming , 2016, LAK.

[77]  Pierre Dillenbourg,et al.  Classroom Social Signal Analysis. , 2014 .