Towards a Smart Wearable Tool to Enable People with SSPI to Communicate by Sentence Fragments

The ability to communicate with others is of paramount importance for mental well-being. In this paper, we describe an interaction system to reduce communication barriers for people with severe speech and physical impairments (SSPI) such as cerebral palsy. The system consists of two main components: (i) the head-mounted human-computer interaction (HCI) part consisting of smart glasses with gaze trackers and text-to-speech functionality (which implement a communication board and the selection tool), and (ii) a natural language processing pipeline in the backend in order to generate complete sentences from the symbols on the board. We developed the components to provide a smooth interaction between the user and the system thereby including gaze tracking, symbol selection, symbol recognition, and sentence generation. Our results suggest that such systems can dramatically increase communication efficiency of people with SSPI.

[1]  Wolfgang L. Zagler,et al.  Computers Helping People with Special Needs, 12th International Conference, ICCHP 2010, Vienna, Austria, July 14-16, 2010, Proceedings, Part II , 2010, ICCHP.

[2]  Krzysztof Z. Gajos,et al.  Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces , 2008, CHI.

[3]  Laura Farinetti,et al.  Eye Tracking Impact on Quality-of-Life of ALS Patients , 2008, ICCHP.

[4]  András Lörincz,et al.  AGI Architecture Measures Human Parameters and Optimizes Human Performance , 2011, AGI.

[5]  Georgios Kouroupetroglou,et al.  Transforming Spontaneous Telegraphic Language to Well-Formed Greek Sentences for Alternative and Augmentative Communication , 2002, SETN.

[6]  John L. Arnott,et al.  Towards the improvement of Augmentative and Alternative Communication through the modelling of conversation , 2013, Comput. Speech Lang..

[7]  Yoav Goldberg,et al.  A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books , 2013, *SEMEVAL.

[8]  Weixiong Zhang,et al.  State-Space Search , 1999, Springer New York.

[9]  Weixiong Zhang.  State-space search - algorithms, complexity, extensions, and applications , 1999 .

[10]  Ioannis Vlahavas,et al.  Methods and Applications of Artificial Intelligence , 2002, Lecture Notes in Computer Science.

[11]  Rupal Patel,et al.  Non-Syntactic Word Prediction for AAC , 2012, SLPAT@HLT-NAACL.

[12]  Kathleen F. McCoy,et al.  Compansion: From research prototype to practical integration , 1998, Natural Language Engineering.

[13]  Sonja Zillner,et al.  Towards Medical Cyber-Physical Systems: Multimodal Augmented Reality for Doctors and Knowledge Discovery about Patients , 2013, HCI.

[14]  Jörg Tiedemann,et al.  Parallel Data, Tools and Interfaces in OPUS , 2012, LREC.

[15]  Julie Marshall,et al.  Interaction training for conversational partners of children with cerebral palsy: a systematic review. , 2004, International journal of language & communication disorders.

[16]  Rupal Patel,et al.  Beyond Linear Syntax: An Image-Oriented Communication Aid. , 2004 .