Linking students' emotions to engagement and writing performance when learning Japanese letters with a pen-based tablet: An investigation based on individual pen pressure parameters

Abstract In this paper, two studies were conducted to understand the link between students’ emotions and their engagement and performance when learning to write Japanese letters with a learning technology involving a pen-based tablet. The incorporation of this haptic sensory modality in the form of a pen tablet is seen to have much potential for continuous emotional detection. In Study 1 (136 students, 84 females, Mage = 23.55, SD = 3.89), the learning effectiveness in terms of students’ writing performance was assessed. We further examined the relationship between discrete learning-centered emotions such as enjoyment, frustration and boredom on the one hand, and pen pressure parameters on the other. The results demonstrated a significant gain in writing. Generalized estimating equation (GEE) models showed that minimum, maximum and average pen pressures all serve as indicators of enjoyment and frustration; on the other hand, for boredom, no significant relationships were found. Compared to enjoyment, the results demonstrate that, the higher a student's frustration when learning, the more pressure they apply to the pen when writing. In Study 2 (90 students, 60 females, Mage = 22.94, SD = 3.82), the findings from Study 1 were used to understand the practical implications of the results, investigating the relationship between emotion-based pen pressure, student engagement and writing performance over the course of the learning process. As in Study 1, the results, again, showed a significant gain in learning how to write Japanese letters. The partial least squares path model (PLS-PM) analyses revealed that higher pen pressure was a negative predictor of engagement, whereas higher engagement was a positive predictor of writing performance. Furthermore, the relationship between pen pressure and performance was mediated by engagement. Overall, the findings can contribute to our understanding of the interaction between students’ emotions and different learning variables in an attempt to support the development of adaptive learning technologies.

[1]  Takafumi Matsumaru,et al.  Calligraphy-Stroke Learning Support System Using Projector and Motion Sensor , 2017, J. Adv. Comput. Intell. Intell. Informatics.

[2]  P. Niemi,et al.  Inducing affective states with success-failure manipulations: a meta-analysis. , 2004, Emotion.

[3]  Reinhard Pekrun,et al.  Reciprocal relations between students' academic enjoyment, boredom, and achievement over time , 2017 .

[4]  Arthur C. Graesser,et al.  Multimethod assessment of affective experience and expression during deep learning , 2009, Int. J. Learn. Technol..

[5]  Li Kai,et al.  The Effects of Handwritten Feedback on Paper and Tablet PC in Learning Japanese Writing , 2007 .

[6]  J. Russell,et al.  Independence and bipolarity in the structure of current affect. , 1998 .

[7]  Matthias Rauterberg,et al.  Incongruity-Based Adaptive Game Balancing , 2009, ACG.

[8]  J. Hayes,et al.  A Cognitive Process Theory of Writing , 1981, College Composition & Communication.

[9]  Abdulmotaleb El-Saddik,et al.  A haptic multimedia handwriting learning system , 2007, Emme '07.

[10]  Friedrich Leisch,et al.  semPLS: Structural Equation Modeling Using Partial Least Squares , 2012 .

[11]  Antoine Bossard Japanese Characters Cartography for Efficient Memorization , 2014, Int. J. Comput. Their Appl..

[12]  Indira Thouvenin,et al.  CalliSmart: An Adaptive Informed Environment for Intelligent Calligraphy Training , 2016, ACHI 2016.

[13]  Richard Gevirtz,et al.  Heart Rate Variability as a Marker of Self-Regulation , 2011, Applied psychophysiology and biofeedback.

[14]  Uta Hinrichs,et al.  Children's Creativity Lab: Creating a 'Pen of the Future' , 2014, ACE '14 Workshops.

[15]  Matthew K. Miller,et al.  Differentiating in-Game Frustration from at-Game Frustration using Touch Pressure , 2016, ISS.

[16]  Marcos Faúndez-Zanuy,et al.  EMOTHAW: A Novel Database for Emotional State Recognition From Handwriting and Drawing , 2017, IEEE Transactions on Human-Machine Systems.

[17]  A. Graesser,et al.  Dynamics of affective states during complex learning , 2012 .

[18]  Nobuko Chikamatsu,et al.  The Effects of Computer Use on L2 Japanese Writing. , 2003 .

[19]  S. D’Mello A selective meta-analysis on the relative incidence of discrete affective states during learning with technology , 2013 .

[20]  M. Macaulay,et al.  The speed of mouse-click as a measure of anxiety during human-computer interaction , 2004, Behav. Inf. Technol..

[21]  Anne C. Frenzel,et al.  The Control-Value Theory of Achievement Emotions: An Integrative Approach to Emotions in Education , 2007 .

[22]  Martha C. Pennington,et al.  Comparing writing process and product across two languages: A study of 6 Singaporean university student writers , 1993 .

[23]  Elizabeth O. Hayward,et al.  Emotional design in multimedia learning: Effects of shape and color on affect and learning , 2014 .

[24]  C. Fornell,et al.  Evaluating structural equation models with unobservable variables and measurement error. , 1981 .

[25]  R. Pekrun The Control-Value Theory of Achievement Emotions: Assumptions, Corollaries, and Implications for Educational Research and Practice , 2006 .

[26]  Elizabeth A. Linnenbrink The Role of Affect in Student Learning. A Multi-Dimensional Approach to Considering the Interaction of Affect, Motivation, and Engagement. , 2007 .

[27]  Jeremy N. Bailenson,et al.  Virtual Interpersonal Touch: Expressing and Recognizing Emotions Through Haptic Devices , 2007, Hum. Comput. Interact..

[28]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[29]  Reinhard Pekrun,et al.  Boredom in achievement settings: Exploring control–value antecedents and performance outcomes of a neglected emotion. , 2010 .

[30]  Mary Ainley,et al.  Students, tasks and emotions: Identifying the contribution of emotions to students' reading of popular culture and popular science texts , 2005 .

[31]  J. P. Wanous,et al.  Overall job satisfaction: how good are single-item measures? , 1997, The Journal of applied psychology.

[32]  Thomas Goetz,et al.  Testing the predictors of boredom at school: development and validation of the precursors to boredom scales. , 2011, The British journal of educational psychology.

[33]  Lisa Linnenbrink-Garcia,et al.  Academic Emotions and Student Engagement , 2012 .

[34]  Roland Brünken,et al.  Emotional design and positive emotions in multimedia learning: An eyetracking study on the use of anthropomorphisms , 2015, Comput. Educ..

[35]  Alan J. Dix,et al.  Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me (ACE) , 2005, DiGRA Conference.

[36]  K. Scherer The dynamic architecture of emotion: Evidence for the component process model , 2009 .

[37]  N. Hall,et al.  Emotion and achievement in the classroom , 2019, Visible Learning Guide to Student Achievement.

[38]  Stanley E. Jones,et al.  A naturalistic study of the meanings of touch , 1985 .

[39]  Julia Moeller,et al.  A word on standardization in longitudinal studies: don't , 2015, Front. Psychol..

[40]  Carol A. Chapelle,et al.  Multimedia CALL: Lessons to be Learned from Research on Instructed SLA , 1998 .

[41]  John G. Lynch,et al.  Reconsidering Baron and Kenny: Myths and Truths about Mediation Analysis , 2010 .

[42]  Anastasia Efklides,et al.  Emotional experiences during learning: Multiple, situated and dynamic , 2005 .

[43]  Julian Frommel,et al.  Identifying Features of Bodily Expression As Indicators of Emotional Experience during Multimedia Learning , 2017, Front. Psychol..

[44]  K. Akita Learning to read and write in Japanese , 1999 .

[45]  Niloy Ganguly,et al.  Emotion detection from touch interactions during text entry on smartphones , 2019, Int. J. Hum. Comput. Stud..

[46]  E. Gentaz,et al.  Haptics in teaching handwriting: the role of perceptual and visuo-motor skills. , 2011, Human movement science.

[47]  Michael K. Filsecker,et al.  Student engagement, context, and adjustment: Addressing definitional, measurement, and methodological issues , 2016 .

[48]  H. Simon,et al.  Motivational and emotional controls of cognition. , 1967, Psychological review.

[49]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[50]  Claudia Schrader,et al.  The perception of control as a predictor of emotional trends during gameplay , 2017 .

[51]  Reinhard Pekrun,et al.  International handbook of emotions in education. , 2014 .

[52]  Jodi Asbell-Clarke,et al.  Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning , 2016, Comput. Hum. Behav..

[53]  Ryan Shaun Joazeiro de Baker,et al.  Off-task behavior in the cognitive tutor classroom: when students "game the system" , 2004, CHI.

[54]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[55]  P. Kleinginna,et al.  A categorized list of emotion definitions, with suggestions for a consensual definition , 1981 .

[56]  R. Moreno,et al.  Does the modality principle hold for different media? A test of the method-affects-learning hypothesis , 2006, J. Comput. Assist. Learn..

[57]  Christothea Herodotou,et al.  Young children and tablets: A systematic review of effects on learning and development , 2018, J. Comput. Assist. Learn..

[58]  S. Zeger,et al.  Longitudinal data analysis using generalized linear models , 1986 .

[59]  Wen-Yuan Wang,et al.  Biologic verification based on pressure sensor keyboards and classifier fusion techniques , 2006, IEEE Transactions on Consumer Electronics.

[60]  Arthur C. Graesser,et al.  Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments , 2010, Int. J. Hum. Comput. Stud..

[61]  Marko Sarstedt,et al.  Partial least squares structural equation modeling (PLS-SEM): An emerging tool in business research , 2014 .

[62]  Mihaly Csikszentmihalyi,et al.  Literacy and Intrinsic Motivation , 2016 .

[63]  Sara Librenjak,et al.  Multimedia assisted learning of Japanese kanji characters , 2012, 2012 Proceedings of the 35th International Convention MIPRO.

[64]  Gary A. Ballinger,et al.  Using Generalized Estimating Equations for Longitudinal Data Analysis , 2004 .

[65]  Andrea Kleinsmith,et al.  Affective Body Expression Perception and Recognition: A Survey , 2013, IEEE Transactions on Affective Computing.

[66]  J. Hayes,et al.  Fluency in Writing , 2001 .

[67]  Rosalind W. Picard,et al.  An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[68]  M. Tulis,et al.  Students' motivational and emotional experiences and their relationship to persistence during academic challenge in mathematics and reading , 2013 .

[69]  Hiroaki Nishino,et al.  A Japanese Calligraphy Trainer Based on Skill Acquisition Through Haptization , 2010, 2010 24th IEEE International Conference on Advanced Information Networking and Applications.

[70]  Slava Kalyuga Cognitive Load Aspects of Text Processing , 2012 .

[71]  Feifei Ye,et al.  The Math and Science Engagement Scales: Scale development, validation, and psychometric properties , 2016 .

[72]  Anne C. Frenzel,et al.  Measuring emotions in students learning and performance: The Achievement Emotions Questionnaire (AE , 2011 .

[73]  Slava Kalyuga,et al.  Cognitive Load Theory: Implications for Affective Computing , 2011, FLAIRS.

[74]  Reinhard Pekrun,et al.  Emotion in Education , 2007 .

[75]  John Sweller,et al.  Cognitive Load Theory , 2020, Encyclopedia of Education and Information Technologies.

[76]  X Zheng,et al.  A Hierarchical Evaluation Approach of Learning Chinese Calligraphy , 2014 .

[77]  Yuan Gao,et al.  What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay? , 2012, TCHI.

[78]  Ryan Shaun Joazeiro de Baker,et al.  Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game , 2015, Comput. Educ..

[79]  Simon Brown,et al.  Affective gaming: measuring emotion through the gamepad , 2003, CHI Extended Abstracts.

[80]  Robert C. Gardner,et al.  Towards a Full Model of Second Language Learning: An Empirical Investigation , 1997 .

[81]  Ryan Shaun Joazeiro de Baker,et al.  Automatic Detection of Learning-Centered Affective States in the Wild , 2015, IUI.

[82]  Ashish Kapoor,et al.  Automatic prediction of frustration , 2007, Int. J. Hum. Comput. Stud..

[83]  Eunjoon Rachel Um,et al.  Emotional design in multimedia learning. , 2012 .

[84]  Roland Brünken,et al.  Emotions and multimedia learning: the moderating role of learner characteristics , 2016, J. Comput. Assist. Learn..

[85]  G. Clore,et al.  Feelings and phenomenal experiences , 1996 .

[86]  M. Sasikumar,et al.  Recognising Emotions from Keyboard Stroke Pattern , 2010 .

[87]  Jan L. Plass,et al.  The Cambridge Handbook of Multimedia Learning: Multimedia Learning in Second Language Acquisition , 2005 .

[88]  Hairong Lv,et al.  Emotion recognition based on pressure sensor keyboards , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[89]  Janet C. Read,et al.  A study of the usability of handwriting recognition for text entry by children , 2007, Interact. Comput..

[90]  Fang Chen,et al.  Dynamic Handwriting Signal Features Predict Domain Expertise , 2018, ACM Trans. Interact. Intell. Syst..

[91]  Roxana Moreno,et al.  Learning from animated classroom exemplars: the case for guiding student teachers' observations with metacognitive prompts , 2009 .

[92]  Sidney D'Mello,et al.  The half-life of cognitive-affective states during complex learning , 2011, Cognition & emotion.

[93]  D. Keltner,et al.  The communication of emotion via touch. , 2009, Emotion.

[94]  Lisa Anthony,et al.  $N-protractor: a fast and accurate multistroke recognizer , 2012, Graphics Interface.