Language-Model Assisted and Icon-Based Communication Through a Brain–Computer Interface With Different Presentation Paradigms

Augmentative and alternative communication (AAC) is typically used by people with severe speech and physical disabilities and is one of the main application areas for the brain–computer interface (BCI) technology. The target population includes people with cerebral palsy, amyotrophic lateral sclerosis, and locked-in syndrome. Word-based AAC systems are mainly faster than letter-based counterparts and are usually supplemented by icons to aid the users. Those icon-based AAC systems that use binary signaling methods such as single click can convert into a single-input BCI system such as event-related potential (ERP) detection. Matrix speller paradigm is typically used to help users identify their target icon on the screen; however, it ties screen space to vocabulary size and navigation complexity, which may require users to make repetitive head, neck, or eye movements to visually locate their intended targets on the screen. Rapid serial visual presentation (RSVP) is an alternative interface that minimizes required movement by displaying all icons at a fixed location, one at a time. IconMessenger is an icon-based BCI–AAC system that combines ERP signal detection with a unified framework for different presentation paradigms including RSVP, matrix speller row&column presentation, and matrix speller single character presentation. IconMessenger also takes advantage of a unique sem-gram language model, incorporated tightly in the inference engine. In this paper, we assess the ERP shape, classification accuracy, and typing performance of different presentation paradigms on 10 healthy participants.

[1]  Laura J. Ball,et al.  Acceptance of Augmentative and Alternative Communication Technology by Persons with Amyotrophic Lateral Sclerosis , 2004 .

[2]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[3]  Andrew Jinks,et al.  Consumer response to AAC devices: Acquisition, training, use, and satisfaction , 1994 .

[4]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[5]  David R. Beukelman,et al.  Identifying the nonspeaking population: a demographic study , 1985 .

[6]  Norman Alm,et al.  Computer-aided conversation: A prototype system for nonspeaking people with physical disabilities , 1994, Applied Psycholinguistics.

[7]  Murat Akcakaya,et al.  RSVP IconMessenger: icon-based brain-interfaced alternative and augmentative communication , 2014 .

[8]  Brian Roark,et al.  Offline analysis of context contribution to ERP-based typing BCI performance. , 2013, Journal of neural engineering.

[9]  Brian Roark,et al.  RSVP keyboard: An EEG based typing interface , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  E Donchin,et al.  The mental prosthesis: assessing the speed of a P300-based brain-computer interface. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[11]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[12]  Per Ola Kristensson,et al.  The Imagination of Crowds: Conversational AAC Language Modeling using Crowdsourcing and Large Data Sources , 2011, EMNLP.

[13]  Rupal Patel,et al.  Message formulation, organization, and navigation schemes for icon-based communication aids , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[14]  W. Yule,et al.  Augmentative communication systems taught to cerebral palsied children--a longitudinal study. I. The acquisition of signs and symbols, and syntactic aspects of their use over time. , 1990, The British journal of disorders of communication.

[15]  Murat Akcakaya,et al.  Noninvasive Brain–Computer Interfaces for Augmentative and Alternative Communication , 2014, IEEE Reviews in Biomedical Engineering.

[16]  D. Beukelman,et al.  Augmentative & Alternative Communication: Supporting Children & Adults With Complex Communication Needs , 2006 .

[17]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[18]  Rupal Patel,et al.  Non-Syntactic Word Prediction for AAC , 2012, SLPAT@HLT-NAACL.

[19]  Rupal Patel,et al.  Beyond Linear Syntax: An Image-Oriented Communication Aid. , 2004 .

[20]  Cuntai Guan,et al.  High performance P300 speller for brain-computer interface , 2004, IEEE International Workshop on Biomedical Circuits and Systems, 2004..

[21]  Tonio Wandmacher,et al.  Training Language Models without Appropriate Language Resources: Experiments with an AAC System for Disabled People , 2006, LREC.

[22]  Brian Roark,et al.  Improved accuracy using recursive Bayesian estimation based language model fusion in ERP-based BCI typing systems , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[23]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[24]  Sally Clendon,et al.  Augmentative and Alternative Communication, Language, and Literacy: Fostering the Relationship , 2004 .

[25]  John Todman,et al.  Rate and quality of conversations using a text-storage AAC system: Single-case training study , 2000 .

[26]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[27]  B. Blankertz,et al.  (C)overt attention and visual speller design in an ERP-based brain-computer interface , 2010, Behavioral and Brain Functions.