SoundScript - Supporting the acquisition of character writing by multisensory integration

This work is introducing a new movement sonification method calledSoundScriptto support the acquisition of character writing by children. SoundScript createssound tracesfrom the writing trace in real-time during the process of handwriting. The structural correlation of both - optic and acoustic - traces leads to an integrated audio-visual percep- tion of writing with the expected stimulation of multisensory integration sites of the CNS. Data of a pilot study are intro- duced indicating that the writing kinematics is reproduced more adequately if additional sound traces are available during writing. In the future SoundScript shall be applied to verify if the establishment of internal character representations can be accelerated, if the conciseness of the specific shape of the particular characters can be made stronger and if thereby the efficiency of the handwriting learning process can be enhanced.

[1]  W. Reavley.  The use of biofeedback in the treatment of writer's cramp , 1975 .

[2]  Ellis Bindman,et al.  Writer's Cramp—A Rational Approach to Treatment? , 1977, British Journal of Psychiatry.

[3]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[4]  Richard A. Schmidt,et al.  Frequent Augmented Feedback Can Degrade Learning: Evidence and Interpretations , 1991 .

[5]  F. Bookstein,et al.  Morphometric Tools for Landmark Data: Geometry and Biology , 1999 .

[6]  A. Jacobson,et al.  Morphometric tools for landmark data , 1993 .

[7]  Richard A. Heath,et al.  Space-time invariance in adult handwriting , 1997 .

[8]  C. Adler,et al.  Biofeedback for writer's cramp. , 1997, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[9]  D. Massaro.  Perceiving talking faces: from speech perception to a behavioral principle , 1999 .

[10]  Tsuneo Yoshikawa,et al.  Virtual lesson and its application to virtual calligraphy system , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[11]  Ching,et al.  The State of the Art in On-Line Handwriting Recognition , 2000 .

[12]  Zoubin Ghahramani,et al.  Perspectives and problems in motor learning , 2001, Trends in Cognitive Sciences.

[13]  Claus Bahlmann,et al.  Online handwriting recognition with support vector machines - a kernel approach , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[14]  Etienne Burdet,et al.  A robotic teacher of Chinese handwriting , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[15]  Andy Hunt,et al.  The Importance of Interaction in Sonification , 2004, ICAD.

[16]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[17]  Alfred Effenberg,et al.  Movement sonification: Effects on perception and action , 2005, IEEE MultiMedia.

[18]  Aaron R. Seitz,et al.  Sound Facilitates Visual Learning , 2006, Current Biology.

[19]  Florence Bara,et al.  A Visuo-Haptic Device - Telemaque - Increases Kindergarten Children's Handwriting Acquisition , 2007, Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07).

[20]  Aaron R. Seitz,et al.  Benefits of multisensory learning , 2008, Trends in Cognitive Sciences.

[21]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[22]  Pavel Senin,et al.  Dynamic Time Warping Algorithm Review , 2008 .

[23]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[24]  Sukhvinder S. Obhi,et al.  The Importance of Temporal Structure and Rhythm for the Optimum Performance of Motor Skills: A New Focus for Practitioners of Sport Psychology , 2009 .

[25]  J. Hermsdörfer,et al.  Auditory grip force feedback in the treatment of Writer's cramp. , 2009, Journal of hand therapy : official journal of the American Society of Hand Therapists.

[26]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[27]  J. Krakauer,et al.  Error correction, sensory prediction, and adaptation in motor control. , 2010, Annual review of neuroscience.

[28]  Andreas Weber,et al.  Movement Sonification: Audiovisual benefits on motor learning , 2011 .

[29]  D. Wolpert,et al.  Principles of sensorimotor learning , 2011, Nature Reviews Neuroscience.

[30]  A. Effenberg,et al.  Observation of sonified movements engages a basal ganglia frontocortical network , 2013, BMC Neuroscience.

[31]  A. Effenberg,et al.  Auditory coding of human movement kinematics. , 2013, Multisensory research.

[32]  Roberto Bresin,et al.  A Systematic Review of Mapping Strategies for the Sonification of Physical Quantities , 2013, PloS one.

[33]  R. Kronland-Martinet,et al.  Handwriting Movement Sonification for the Rehabilitation of Dysgraphia , 2013 .

[34]  Gerd Schmitz,et al.  A mobile sonification system for stroke rehabilitation , 2014 .

[35]  Brenda Rapp,et al.  The similarity structure of distributed neural responses reveals the multiple representations of letters , 2014, NeuroImage.

[36]  Dan Tidhar,et al.  Musicians are more consistent: Gestural cross-modal mappings of pitch, loudness and tempo in real-time , 2014, Front. Psychol..

[37]  M. Hasselhorn,et al.  Working Memory in Children With Learning Disabilities in Reading Versus Spelling , 2015, Journal of learning disabilities.

[38]  T. Agostini,et al.  Ecological sounds affect breath duration more than artificial sounds , 2015, Psychological Research.