Towards the automated localisation of targets in rapid image-sifting by collaborative brain-computer interfaces

The N2pc is a lateralised Event-Related Potential (ERP) that signals a shift of attention towards the location of a potential object of interest. We propose a single-trial target-localisation collaborative Brain-Computer Interface (cBCI) that exploits this ERP to automatically approximate the horizontal position of targets in aerial images. Images were presented by means of the rapid serial visual presentation technique at rates of 5, 6 and 10 Hz. We created three different cBCIs and tested a participant selection method in which groups are formed according to the similarity of participants’ performance. The N2pc that is elicited in our experiments contains information about the position of the target along the horizontal axis. Moreover, combining information from multiple participants provides absolute median improvements in the area under the receiver operating characteristic curve of up to 21% (for groups of size 3) with respect to single-user BCIs. These improvements are bigger when groups are formed by participants with similar individual performance, and much of this effect can be explained using simple theoretical models. Our results suggest that BCIs for automated triaging can be improved by integrating two classification systems: one devoted to target detection and another to detect the attentional shifts associated with lateral targets.

[1]  Claus Bahlmann,et al.  In a Blink of an Eye and a Switch of a Transistor: Cortically Coupled Computer Vision , 2010, Proceedings of the IEEE.

[2]  Arthur F. Kramer,et al.  Training and Transfer of Training in Rapid Visual Search for Camouflaged Targets , 2013, PloS one.

[3]  Riccardo Poli,et al.  Towards cooperative brain-computer interfaces for space navigation , 2013, IUI '13.

[4]  Misha Pavel,et al.  Rapid image analysis using neural signals , 2008, CHI Extended Abstracts.

[5]  Kenneth I. Forster,et al.  Visual perception of rapidly presented word sequences of varying complexity , 1970 .

[6]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[7]  Craig K. Abbey,et al.  Neural decoding of collective wisdom with multi-brain computing , 2012, NeuroImage.

[8]  Martin Eimer,et al.  Involuntary Attentional Capture is Determined by Task Set: Evidence from Event-related Brain Potentials , 2008, Journal of Cognitive Neuroscience.

[9]  U. Leonards,et al.  What makes cast shadows hard to see? , 2010, Journal of vision.

[10]  J. Polich Theoretical Overview of P3a and P3b , 2003 .

[11]  Tzyy-Ping Jung,et al.  A Collaborative Brain-Computer Interface for Improving Human Performance , 2011, PloS one.

[12]  Steven J. Luck,et al.  Electrophysiological Correlates of the Focusing of Attention within Complex Visual Scenes: N2pc and Related ERP Components , 2011 .

[13]  K. Shapiro,et al.  Attention to visual pattern information produces the attentional blink in rapid serial visual presentation. , 1994, Journal of experimental psychology. Human perception and performance.

[14]  C. Koch,et al.  The duration of the attentional blink in natural scenes depends on stimulus category , 2007, Vision Research.

[15]  A. Stoica,et al.  Improving decision-making based on visual perception via a collaborative brain-computer interface , 2013, 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA).

[16]  小椋 力,et al.  Recent advances in event-related brain potential research : proceedings of the 11th International Conference on Event-Related Potentials (EPIC), Okinawa, Japan, June 25-30, 1995 , 1996 .

[17]  Martin Eimer,et al.  Item and category-based attentional control during search for real-world objects: Can you find the pants among the pans? , 2014, Journal of experimental psychology. Human perception and performance.

[18]  E Donchin,et al.  The mental prosthesis: assessing the speed of a P300-based brain-computer interface. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[19]  Fei Wang,et al.  Relationships between the resting-state network and the P3: Evidence from a scalp EEG study , 2015, Scientific Reports.

[20]  Riccardo Poli,et al.  Brain–Computer Interfaces for Detection and Localization of Targets in Aerial Images , 2017, IEEE Transactions on Biomedical Engineering.

[21]  Riccardo Poli,et al.  Collaborative Brain-Computer Interface for Aiding Decision-Making , 2014, PloS one.

[22]  N. Kanwisher Repetition blindness: Type recognition without token individuation , 1987, Cognition.

[23]  Riccardo Poli,et al.  A collaborative Brain-Computer Interface to improve human performance in a visual search task , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[24]  P. Sajda,et al.  Cortically coupled computer vision for rapid image search , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[25]  Jon Touryan,et al.  Integration of Automated Neural Processing into an Army-Relevant Multitasking Simulation Environment , 2013, HCI.

[26]  Deborah Penque,et al.  The mechanism of sirtuin 2–mediated exacerbation of alpha-synuclein toxicity in models of Parkinson disease , 2017, PLoS biology.

[27]  Fanglin Chen,et al.  A Speedy Hybrid BCI Spelling Approach Combining P300 and SSVEP , 2014, IEEE Transactions on Biomedical Engineering.

[28]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[29]  Tanja Schultz,et al.  Locating user attention using eye tracking and EEG for spatio-temporal event selection , 2013, IUI '13.

[30]  N. Bigdely-Shamlo,et al.  Brain Activity-Based Image Classification From Rapid Serial Visual Presentation , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[31]  Riccardo Poli,et al.  Collaborative brain-computer interfaces for target localisation in rapid serial visual presentation , 2014, 2014 6th Computer Science and Electronic Engineering Conference (CEEC).

[32]  Martin Eimer,et al.  Attentional capture by task-irrelevant fearful faces is revealed by the N2pc component , 2007, Biological Psychology.

[33]  Martin Eimer,et al.  Activation of New Attentional Templates for Real-world Objects in Visual Search , 2015, Journal of Cognitive Neuroscience.

[34]  Luca Citi,et al.  Possible sources of perceptual errors in P300-based speller paradigm , 2004 .

[35]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[36]  Hubert Cecotti,et al.  Performance estimation of a cooperative brain-computer interface based on the detection of steady-state visual evoked potentials , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[37]  Dongrui Wu,et al.  Improved Neural Signal Classification in a Rapid Serial Visual Presentation Task Using Active Learning , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[38]  P. Latham,et al.  References and Notes Supporting Online Material Materials and Methods Figs. S1 to S11 References Movie S1 Optimally Interacting Minds R�ports , 2022 .

[39]  Misha Pavel,et al.  A framework for rapid visual image search using single-trial brain evoked responses , 2011, Neurocomputing.

[40]  S J Luck,et al.  Spatial filtering during visual search: evidence from human electrophysiology. , 1994, Journal of experimental psychology. Human perception and performance.

[41]  Nicolas Robitaille,et al.  Attentional control and capture in the attentional blink paradigm: Evidence from human electrophysiology , 2006 .

[42]  Peng Yuan,et al.  Study on an online collaborative BCI to accelerate response to visual targets , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[43]  Weidong Chen,et al.  Rapid face recognition based on single-trial event-related potential detection over multiple brains , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[44]  Fanglin Chen,et al.  A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm , 2013, Journal of neural engineering.

[45]  Peter A. Beling,et al.  A multi-instance learning approach to filtering images for presentation to analysts , 2014, Environment Systems and Decisions.

[46]  Christa Neuper,et al.  An asynchronously controlled EEG-based virtual keyboard: improvement of the spelling rate , 2004, IEEE Transactions on Biomedical Engineering.

[47]  Desney S. Tan,et al.  Complementary computing for visual tasks: Meshing computer vision with human visual processing , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[48]  Yang Liu,et al.  A novel task-oriented optimal design for P300-based brain–computer interfaces , 2014, Journal of neural engineering.

[49]  Shih-Fu Chang,et al.  Closing the loop in cortically-coupled computer vision: a brain–computer interface for searching image databases , 2011, Journal of neural engineering.

[50]  Bin Xia,et al.  Brain–Computer Interface–Based Communication in the Completely Locked-In State , 2017, GBCIC.

[51]  Albert B. Kao,et al.  Decision accuracy in complex environments is often maximized by small group sizes , 2014, Proceedings of the Royal Society B: Biological Sciences.

[52]  C. Cinel,et al.  P300-Based BCI Mouse With Genetically-Optimized Analogue Control , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[53]  Hiroshi Nittono,et al.  Electrophysiology of Kansei: Recent Advances in Event-Related Brain Potential Research , 2008 .

[54]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[55]  Desney S. Tan,et al.  Combining brain computer interfaces with vision for object categorization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Don M. Tucker,et al.  A single-trial analytic framework for EEG analysis and its application to target detection and classification , 2008, NeuroImage.

[57]  Nader Pouratian,et al.  Natural language processing with dynamic classification improves P300 speller accuracy and bit rate , 2012, Journal of neural engineering.

[58]  Riccardo Poli,et al.  A collaborative Brain-Computer Interface for improving group detection of visual targets in complex natural environments , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[59]  M. Potter,et al.  Recognition memory for a rapid sequence of pictures. , 1969, Journal of experimental psychology.

[60]  Riccardo Poli,et al.  Collaborative brain-computer interfaces for the automatic classification of images , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[61]  Miguel P. Eckstein,et al.  Multimodal target detection using single trial evoked EEG responses in single and dual-tasks , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[62]  Stephen Umunna,et al.  Towards a brain computer interface based on the N2pc event-related potential , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[63]  M. Eimer The N2pc component as an indicator of attentional selectivity. , 1996, Electroencephalography and clinical neurophysiology.

[64]  Christian Jutten,et al.  Single-trial classification of multi-user P300-based Brain-Computer Interface using riemannian geometry , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[65]  C. Cinel,et al.  Multi-brain fusion and applications to intelligence analysis , 2013, Defense, Security, and Sensing.

[66]  P. Sajda,et al.  Spatiotemporal Linear Decoding of Brain State , 2008, IEEE Signal Processing Magazine.