Dynamic Handwriting Signal Features Predict Domain Expertise

As commercial pen-centric systems proliferate, they create a parallel need for analytic techniques based on dynamic writing. Within educational applications, recent empirical research has shown that signal-level features of students’ writing, such as stroke distance, pressure and duration, are adapted to conserve total energy expenditure as they consolidate expertise in a domain. The present research examined how accurately three different machine-learning algorithms could automatically classify users’ domain expertise based on signal features of their writing, without any content analysis. Compared with an unguided machine-learning classification accuracy of 71%, hybrid methods using empirical-statistical guidance correctly classified 79–92% of students by their domain expertise level. In addition to improved accuracy, the hybrid approach contributed a causal understanding of prediction success and generalization to new data. These novel findings open up opportunities to design new automated learning analytic systems and student-adaptive educational technologies for the rapidly expanding sector of commercial pen systems.

[1]  Sharon L. Oviatt,et al.  Written and multimodal representations as predictors of expertise and problem-solving success in mathematics , 2013, ICMI '13.

[2]  Antonio Krüger,et al.  The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition , 2018 .

[3]  Denham L. Phipps The human–computer interaction handbook: fundamentals, evolving technologies and emerging applications (3rd ed) , 2013 .

[4]  Kalyan Veeramachaneni,et al.  Deep feature synthesis: Towards automating data science endeavors , 2015, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[5]  Nadir Weibel,et al.  Multimodal learning analytics: description of math data corpus for ICMI grand challenge workshop , 2013, ICMI '13.

[6]  Sharon Oviatt,et al.  Toward High-Performance Communications Interfaces for Science Problem Solving , 2010 .

[7]  Sharon L. Oviatt,et al.  Expressive pen-based interfaces for math education , 2007, CSCL.

[8]  Peter C.-H. Cheng,et al.  Measuring Mathematical Formula Writing Competence: An Application of Graphical Protocol Analysis , 2007 .

[9]  D. Kahneman Thinking, Fast and Slow , 2011 .

[10]  Peter C.-H. Cheng,et al.  A Graphical Chunk Production Model: Evaluation Using Graphical Protocol Analysis with Artificial Sentences , 2008 .

[11]  Antonio Krüger,et al.  The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition - Volume 2 , 2018 .

[12]  Pedro M. Domingos A few useful things to know about machine learning , 2012, Commun. ACM.

[13]  Sharon L. Oviatt,et al.  Written Activity, Representations and Fluency as Predictors of Domain Expertise in Mathematics , 2014, ICMI.

[14]  Xavier Ochoa,et al.  Expertise estimation based on simple multimodal features , 2013, ICMI '13.

[15]  Ian H. Witten,et al.  Data Mining: Practical Machine Learning Tools and Techniques, 3/E , 2014 .

[16]  Thomas F. Shipley,et al.  Drawing on Experience: How Domain Knowledge Is Reflected in Sketches of Scientific Structures and Processes , 2014, Research in Science Education.

[17]  Sharon L. Oviatt,et al.  User-Centered Modeling for Spoken Language and Multimodal Interfaces , 1996, IEEE Multim..

[18]  Sharon L. Oviatt,et al.  The impact of interface affordances on human ideation, problem solving, and inferential reasoning , 2012, TCHI.

[19]  Ryan S. Baker,et al.  The State of Educational Data Mining in 2009: A Review and Future Visions. , 2009, EDM 2009.

[20]  Björn Schuller,et al.  Multimodal user state and trait recognition: an overview , 2018, The Handbook of Multimodal-Multisensor Interfaces, Volume 2.

[21]  George Siemens,et al.  The Cambridge Handbook of the Learning Sciences: Educational Data Mining and Learning Analytics , 2014 .

[22]  Sharon L. Oviatt,et al.  Quiet interfaces that help students think , 2006, UIST.

[23]  Fang Chen,et al.  Mental Workload Classification via Online Writing Features , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[24]  Benjamin D. Jee,et al.  Drawing on Experience: Use of Sketching to Evaluate Knowledge of Spatial Scientific Concepts , 2009 .

[25]  N. McGlynn Thinking fast and slow. , 2014, Australian veterinary journal.

[26]  Sharon L. Oviatt,et al.  Prototyping novel collaborative multimodal systems: simulation, data collection and analysis tools for the next decade , 2006, ICMI '06.

[27]  Robert J. Crutcher,et al.  The role of deliberate practice in the acquisition of expert performance. , 1993 .

[28]  Sharon Oviatt,et al.  The Design of Future Educational Interfaces , 2013 .

[29]  Fang Chen,et al.  Cognitive load evaluation of handwriting using stroke-level features , 2011, IUI '11.

[30]  George Siemens,et al.  Learning Analytics and Educational Data Mining , 2016 .

[31]  Fang Chen,et al.  Combining empirical and machine learning techniques to predict math expertise using pen signal features , 2014, MLA@ICMI.

[32]  Yang Wang,et al.  Measurable Decision Making with GSR and Pupillary Analysis for Intelligent User Interface , 2015, ACM Trans. Comput. Hum. Interact..

[33]  Alice F. Healy,et al.  Expertise: defined, described, explained , 2014, Front. Psychol..

[34]  Alexander H. Waibel,et al.  Multimodal interfaces , 1996, Artificial Intelligence Review.

[35]  A. Caramazza,et al.  Lexical organization of nouns and verbs in the brain , 1991, Nature.

[36]  Sharon L. Oviatt Problem solving, domain expertise and learning: ground-truth performance results for math data corpus , 2013, ICMI '13.

[37]  Walter Schneider,et al.  Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. , 1977 .

[38]  Daniel L. Schwartz,et al.  Prospective Adaptation in the Use of External Representations , 2009 .

[39]  Xavier Ochoa,et al.  Multimodal learning analytics: assessing learners' mental state during the process of learning , 2018, The Handbook of Multimodal-Multisensor Interfaces, Volume 2.

[40]  Sharon L. Oviatt,et al.  The Paradigm Shift to Multimodality in Contemporary Computer Interfaces , 2015, Synthesis Lectures on Human-Centered Informatics.

[41]  Marcelo Worsley,et al.  Toward the Development of Learning Analytics: Student Speech as an Automatic and Natural Form of Assessment , 2010 .

[42]  Ian Witten,et al.  Data Mining , 2000 .