Automated Analysis of Middle School Students’ Written Reflections During Game-Based Learning

Game-based learning environments enable students to engage in authentic, inquiry-based learning. Reflective thinking serves a critical role in inquiry-based learning by encouraging students to think critically about their knowledge and experiences in order to foster deeper learning processes. Free-response reflection prompts can be embedded in game-based learning environments to encourage students to engage in reflection and externalize their reflection processes, but automatically assessing student reflection presents significant challenges. In this paper, we present a framework for automatically assessing students’ written reflection responses during inquiry-based learning in Crystal Island, a game-based learning environment for middle school microbiology. Using data from a classroom study involving 153 middle school students, we compare the effectiveness of several computational representations of students’ natural language responses to reflection prompts—GloVe, ELMo, tf-idf, unigrams—across several machine learning-based regression techniques (i.e., random forest, support vector machine, multi-layer perceptron) to assess the depth of student reflection responses. Results demonstrate that assessment models based on ELMo deep contextualized word representations yield more accurate predictions of students’ written reflection depth than competing techniques. These findings point toward the potential of leveraging automated assessment of student reflection to inform real-time adaptive support for inquiry-based learning in game-based learning environments.

[1]  Elaine H. J. Yew,et al.  Problem-Based Learning: An Overview of its Process and Impact on Learning , 2016 .

[2]  M. Manen Linking Ways of Knowing with Ways of Being Practical , 1977 .

[3]  F. Schmidt Meta-Analysis , 2008 .

[4]  Sinthia Bosnic-Anticevich,et al.  A Model for Assessing Reflective Practices in Pharmacy Education , 2015, American Journal of Pharmaceutical Education.

[5]  Yue Zhang,et al.  Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring , 2017, CoNLL.

[6]  Thomas Daniel Ullmann,et al.  Automated detection of reflection in texts : a machine learning based approach , 2015 .

[7]  P. Brickman,et al.  Effects of Inquiry-based Learning on Students’ Science Literacy Skills and Confidence , 2009 .

[8]  Michelle Taub,et al.  Self-Regulation in Computer-Assisted Learning Systems , 2019, The Cambridge Handbook of Cognition and Education.

[9]  Beverley Bailey Turning experience into learning , 2016 .

[10]  Arthur C. Graesser,et al.  Linguistic characteristics of reflective states in video annotations under different instructional conditions , 2019, Comput. Hum. Behav..

[11]  C. Hmelo‐Silver Problem-Based Learning: What and How Do Students Learn? , 2004 .

[12]  B. Zimmerman,et al.  Handbook of Self-Regulation of Learning and Performance , 2011 .

[13]  Ruth Harmsen,et al.  Meta-Analysis of Inquiry-Based Learning , 2016 .

[14]  Alyssa Friend Wise,et al.  Developing reflection analytics for health professions education: A multi-dimensional framework to align critical concepts with data features , 2019, Comput. Hum. Behav..

[15]  Christopher. Simons,et al.  Machine learning with Python , 2017 .

[16]  A. Kitsantas,et al.  Motivation and Self-Regulated Learning Influences on Middle School Mathematics Achievement , 2017 .

[17]  Philip H. Winne,et al.  Cognition and Metacognition within Self-Regulated Learning , 2017 .

[18]  D. Boud,et al.  Reflection, turning experience into learning , 1985 .

[19]  George Siemens,et al.  Understand students' self-reflections through learning analytics , 2018, LAK.

[20]  Michael Flor,et al.  How to account for mispellings: Quantifying the benefit of character representations in neural content scoring models , 2019, BEA@ACL.

[21]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[22]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[23]  Michelle Taub,et al.  The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment , 2020, Comput. Educ..

[24]  J. Mezirow Transformative Dimensions of Adult Learning , 1991 .

[25]  Brian R. Belland,et al.  Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education , 2016, Review of educational research.

[26]  A. Kitsantas,et al.  Motivation and Self-Regulated Learning Influences on Middle School Mathematics Achievement , 2017 .

[27]  D. Kember,et al.  Assessing the level of student reflection from reflective journals. , 1995, Journal of advanced nursing.

[28]  Jennifer A. Moon A Handbook of Reflective and Experiential Learning: Theory and Practice , 2004 .

[29]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[30]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[31]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[32]  Barry J. Zimmerman,et al.  Developing Self-Regulated Learners: Beyond Achievement to Self-Efficacy , 1996 .

[33]  Richard E. Mayer,et al.  Handbook of Game-Based Learning. , 2020 .

[34]  Tamar Frankel [The theory and the practice...]. , 2001, Tijdschrift voor diergeneeskunde.

[35]  P. Robert-Jan Simons,et al.  Assessing student teachers’ reflective writing through quantitative content analysis , 2014 .

[36]  Steven Bethard,et al.  DLS@CU: Sentence Similarity from Word Alignment and Semantic Vector Composition , 2015, *SEMEVAL.

[37]  Lucia Specia,et al.  Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) , 2017, CoNLL.

[38]  Thomas Daniel Ullmann,et al.  Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches , 2019, International Journal of Artificial Intelligence in Education.

[39]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.