EMOTHAW: A Novel Database for Emotional State Recognition From Handwriting and Drawing

The detection of negative emotions through daily activities such as writing and drawing is useful for promoting wellbeing. The spread of human–machine interfaces such as tablets makes the collection of handwriting and drawing samples easier. In this context, we present a first publicly available database which relates emotional states to handwriting and drawing, that we call EMOTHAW (EMOTion recognition from HAndWriting and draWing). This database includes samples of 129 participants whose emotional states, namely anxiety, depression, and stress, are assessed by the Depression–Anxiety–Stress Scales (DASS) questionnaire. Seven tasks are recorded through a digitizing tablet: pentagons and house drawing, words copied in handprint, circles and clock drawing, and one sentence copied in cursive writing. Records consist in pen positions, on-paper and in-air, time stamp, pressure, pen azimuth, and altitude. We report our analysis on this database. From collected data, we first compute measurements related to timing and ductus. We compute separate measurements according to the position of the writing device: on paper or in-air. We analyze and classify this set of measurements (referred to as features) using a random forest approach. This latter is a machine learning method [1], based on an ensemble of decision trees, which includes a feature ranking process. We use this ranking process to identify the features which best reveal a targeted emotional state. We then build random forest classifiers associated with each emotional state. We provide accuracy, sensitivity, and specificity evaluation measures obtained from cross-validation experiments. Our results show that anxiety and stress recognition perform better than depression recognition.

[1]  J. Mundt,et al.  Vocal Acoustic Biomarkers of Depression Severity and Treatment Response , 2012, Biological Psychiatry.

[2]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[3]  W. Line,et al.  A Visual Motor Gestalt Test and Its Clinical Use , 1940 .

[4]  Christian O'Reilly,et al.  Design of a neuromuscular disorders diagnostic system using human movement analysis , 2012, 2012 11th International Conference on Information Science, Signal Processing and their Applications (ISSPA).

[5]  Sara Rosenblum,et al.  Detection of Deception Via Handwriting Behaviors Using a Computerized Tool: Toward an Evaluation of Malingering , 2014, Cognitive Computation.

[6]  Sara Rosenblum,et al.  Computerized kinematic analysis of the clock drawing task in elderly people with mild Major Depressive Disorder: an exploratory study , 2009, International Psychogeriatrics.

[7]  Sara Rosenblum,et al.  Handwriting process variables among elderly people with mild Major Depressive Disorder: a preliminary study , 2010, Aging clinical and experimental research.

[8]  Hyungsin Kim,et al.  The ClockMe system: computer-assisted screening tool for dementia , 2013 .

[9]  W. Marsden I and J , 2012 .

[10]  M. Eysenck,et al.  Anxiety and cognitive performance: attentional control theory. , 2007, Emotion.

[11]  K. Mogg,et al.  A cognitive-motivational analysis of anxiety. , 1998, Behaviour research and therapy.

[12]  Javier Garrido Salas,et al.  BiosecurID: a multimodal biometric database , 2009, Pattern Analysis and Applications.

[13]  Réjean Plamondon,et al.  Strokes against stroke - strokes for strides , 2014, Pattern Recognit..

[14]  Réjean Plamondon,et al.  Automatic signature verification and writer identification - the state of the art , 1989, Pattern Recognit..

[15]  Hiroshi Motoda,et al.  Computational Methods of Feature Selection , 2022 .

[16]  Marcos Faúndez-Zanuy,et al.  An Information Analysis of In-Air and On-Surface Trajectories in Online Handwriting , 2011, Cognitive Computation.

[17]  J. M. Tanner,et al.  Journal of Psychiatric Research , 1962, Nature.

[18]  C. Pelachaud,et al.  Emotion-Oriented Systems: The Humaine Handbook , 2011 .

[19]  Jeffrey F. Cohn,et al.  Detecting Depression Severity from Vocal Prosody , 2013, IEEE Transactions on Affective Computing.

[20]  Sargur N. Srihari,et al.  On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Catherine Pelachaud,et al.  Collection and characterization of emotional body behaviors , 2014, MOCO.

[22]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[23]  A. Baum,et al.  Stress, intrusive imagery, and chronic distress. , 1990, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[24]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[25]  Nicu Sebe,et al.  Multimodal approaches for emotion recognition: a survey , 2005, IS&T/SPIE Electronic Imaging.

[26]  Christian O'Reilly,et al.  Recent developments in the study of rapid human movements with the kinematic theory: Applications to handwriting and signature synthesis , 2014, Pattern Recognit. Lett..

[27]  J. M. Digman PERSONALITY STRUCTURE: EMERGENCE OF THE FIVE-FACTOR MODEL , 1990 .

[28]  Marcos Faúndez-Zanuy,et al.  Biometric Applications Related to Human Beings: There Is Life beyond Security , 2012, Cognitive Computation.

[29]  Sheldon Cohen,et al.  Psychological stress and disease. , 2007, JAMA.

[30]  Igor Kononenko,et al.  Machine Learning and Data Mining: Introduction to Principles and Algorithms , 2007 .

[31]  T. Brown,et al.  Psychometric properties of the Depression Anxiety Stress Scales (DASS) in clinical samples. , 1997, Behaviour research and therapy.

[32]  Giora Keinan,et al.  Can Stress be Measured by Handwriting Analysis? The Effectiveness of the Analytic Method , 1993 .

[33]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[34]  Anna Esposito,et al.  On the Significance of Speech Pauses in Depressive Disorders: Results on Read and Spontaneous Narratives , 2016, Recent Advances in Nonlinear Speech Processing.

[35]  Anna Esposito,et al.  On the recognition of emotional vocal expressions: motivations for a holistic approach , 2012, Cognitive Processing.

[36]  Thomas Li-Ping Tang,et al.  Detecting Honest People’s Lies in Handwriting , 2012 .

[37]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[38]  David Eichelberger,et al.  Handbook Of Psychological Testing , 2016 .

[39]  Isabelle Guyon,et al.  UNIPEN project of on-line data exchange and recognizer benchmarks , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[40]  S. Folstein,et al.  "Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. , 1975, Journal of psychiatric research.

[41]  S. Rosenblum,et al.  Age-related changes in executive control and their relationships with activity performance in handwriting. , 2013, Human movement science.

[42]  J. Henry,et al.  The Depression Anxiety Stress Scales (DASS): normative data and latent structure in a large non-clinical sample. , 2003, The British journal of clinical psychology.

[43]  J. Neils-Strunjas,et al.  Perseverative Writing Errors in a Patient with Alzheimer's Disease , 1998, Brain and Language.

[44]  Jun Soo Kwon,et al.  Clinical and empirical applications of the Rey–Osterrieth Complex Figure Test , 2006, Nature Protocols.

[45]  Sara Rosenblum,et al.  The in Air Phenomenon: Temporal and Spatial Correlates of the Handwriting Process , 2003, Perceptual and motor skills.