Feedback Strategies for Embodied Agents to Enhance Sign Language Vocabulary Learning

When learning sign language, feedback on accuracy is critical to vocabulary acquisition. When designing technologies to provide such visual feedback, we need to research effective ways to identify errors and present meaningful and effective feedback to learners. Motion capture technologies provide new opportunities to enhance sign language learning experiences through the presentation of visual feedback that indicates the accuracy of the signs made by learners. We designed, developed, and evaluated an embodied agent-based system for learning the location and gross motor movements of sign language vocabulary. The system presents a sign, tracks the learner's attempts at a sign, and provides visual feedback to the learner on their errors. We compared five different types of visual feedback, and in a study involving 51 participants we established that learners preferred visual feedback where their attempts at a sign were shown concurrently with the movements of the instructor with or without explicit corrections.

[1]  Peter Wolf,et al.  The effect of haptic guidance and visual feedback on learning a complex tennis task , 2013, Experimental Brain Research.

[2]  Kirsten Ellis,et al.  Children, Australian sign language and the web; the possibilities , 2004, WWW 2004.

[3]  R. Riener,et al.  Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review , 2012, Psychonomic Bulletin & Review.

[4]  Louisa Willoughby,et al.  Errors and Feedback in the Beginner Auslan Classroom , 2015 .

[5]  Paul A. Lyddon,et al.  The Efficacy of Corrective Feedback and Textual Enhancement in Promoting the Acquisition of Grammatical Redundancies , 2011 .

[6]  Thad Starner,et al.  American sign language vocabulary: computer aided instruction for non-signers , 2008, Assets '08.

[7]  Kostas Karpouzis,et al.  Educational resources and implementation of a Greek sign language synthesis architecture , 2007, Comput. Educ..

[8]  Thad Starner,et al.  Development of an American Sign Language game for deaf children , 2005, IDC '05.

[9]  Adam Schembri,et al.  Australian Sign Language: Auslan: An Introduction to Sign Language Linguistics , 2007 .

[10]  D A Grushkin,et al.  Lexidactylophobia: The (Irrational) Fear of Fingerspelling , 1998, American annals of the deaf.

[11]  Kirsten Ellis,et al.  MIC, an interactive sign language teaching system , 2018, OZCHI.

[12]  J. D. Bonvillian.  Hand Preference in Young Children's Early Signing. , 1993 .

[13]  Jan Carlo Barca,et al.  Exploring Sensor Gloves for Teaching Children Sign Language , 2012, Adv. Hum. Comput. Interact..

[14]  Matt Huenerfauth,et al.  Evaluation of Language Feedback Methods for Student Videos of American Sign Language , 2017, ACM Trans. Access. Comput..

[15]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[16]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[17]  Houqiang Li,et al.  A Threshold-based HMM-DTW Approach for Continuous Sign Language Recognition , 2014, ICIMCS '14.

[18]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[19]  Julie Fisher,et al.  Taking a User Centred Design Approach for Designing a System to Teach Sign Language , 2014 .

[20]  Noriko Iwashita.  NEGATIVE FEEDBACK AND POSITIVE EVIDENCE IN TASK-BASED INTERACTION , 2003, Studies in Second Language Acquisition.

[21]  Marcel J. T. Reinders,et al.  Sign Language Recognition by Combining Statistical DTW and Independent Classification , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Tadej Bajd,et al.  Virtual environment for lower-extremities training. , 2008, Gait & posture.

[23]  John Bitchener,et al.  Evidence in Support of Written Corrective Feedback. , 2008 .

[24]  Paul Newbold,et al.  Statistics for Business & Economics , 1991 .

[25]  David A Cook,et al.  Feedback for simulation-based procedural skills training: a meta-analysis and critical narrative synthesis , 2013, Advances in Health Sciences Education.

[26]  R. Ellis,et al.  IMPLICIT AND EXPLICIT CORRECTIVE FEEDBACK AND THE ACQUISITION OF L2 GRAMMAR , 2006, Studies in Second Language Acquisition.

[27]  Jiping He,et al.  Interactive Multimodal Biofeedback for Task-Oriented Neural Rehabilitation , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[28]  Bülent Sankur,et al.  SignTutor: An Interactive System for Sign Language Tutoring , 2009, IEEE Multimedia.

[29]  G. Havranek,et al.  FACTORS AFFECTING THE SUCCESS OF CORRECTIVE FEEDBACK , 2001 .

[30]  Marilyn Daniels.  Dancing with Words: Signing for Hearing Children's Literacy , 2000 .

[31]  Julie Fisher,et al.  A design science exploration of a visual-spatial learning system with feedback , 2016, ArXiv.

[32]  H. Pashler,et al.  When does feedback facilitate learning of words? , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[33]  F. Itakura,et al.  Minimum prediction residual principle applied to speech recognition , 1975 .