Assessing driver cortical activity under varying levels of automation with functional near infrared spectroscopy

Information about drivers' mental states can be vital to the design of interfaces for highly automated vehicles. Functional near infrared spectroscopy (fNIRS) is a neuroimaging tool that is fast becoming popular to study the cortical activity of participants in HCI experiments and driving simulator studies in particular. The analysis methods of the fNIRS data create requirements in the experimental design such as repeated measures. In this paper, we present a study of the event related cortical activity of the drivers of manual, partially autonomous, and fully autonomous cars when performing lane changes using functional near infrared spectroscopic measures. We also present the experimental methodology that was adopted to meet the needs of the fNIRS measurement and the subsequent analysis. The study (N=28) was conducted in a driving simulator. Participants drove for approximately 7 minutes and performed 8 lane change maneuvers in each mode of automation. Multiple streams of data including 4 time-synced video recordings, NASA TLX questionnaires and fNIRS data were recorded and analyzed. It was found that the dorsolateral prefrontal cortex activation during lane changes performed in a partially autonomous mode of operation was just as high as that during a manual lane change, showing that drivers of partially automated systems are as cognitively engaged as drivers of manually operated vehicles.

[1]  Yan Wang,et al.  Assessment of cerebral oxygenation during prolonged simulated driving using near infrared spectroscopy: its implications for fatigue development , 2009, European Journal of Applied Physiology.

[2]  Bryan Reimer,et al.  Classifying driver workload using physiological and driving performance data: two field studies , 2014, CHI.

[3]  Wendy Ju,et al.  Distraction Becomes Engagement in Automated Driving , 2015 .

[4]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[5]  Gurpreet Singh,et al.  Drowsiness Detection System for Pilots , 2013 .

[6]  Mark S. Young,et al.  Attention and automation: New perspectives on mental underload and performance , 2002 .

[7]  Martin Steinert,et al.  Capturing emotion reactivity through physiology measurement as a foundation for affective engineering in engineering design science and engineering practices , 2017, J. Intell. Manuf..

[8]  Raja Parasuraman,et al.  Into the Wild: Neuroergonomic Differentiation of Hand-Held and Augmented Reality Wearable Displays during Outdoor Navigation with Functional Near Infrared Spectroscopy , 2016, Front. Hum. Neurosci..

[9]  William L Verplank.  Is there an optimal work-load in manual control? , 1978 .

[10]  Linda Ng Boyle,et al.  Visual Attention in Driving: The Effects of Cognitive Load and Visual Disruption , 2007, Hum. Factors.

[11]  Mark H. Johnson,et al.  Selective prefrontal cortex responses to joint attention in early infancy , 2010, Biology Letters.

[12]  Hideaki Koizumi,et al.  Prefrontal cortical activation associated with working memory in adults and preschool children: an event-related optical topography study. , 2004, Cerebral cortex.

[13]  T. Sejnowski,et al.  Estimating alertness from the EEG power spectrum , 1997, IEEE Transactions on Biomedical Engineering.

[14]  Eiko Hatakeyama,et al.  A comparison of cerebral activity in the prefrontal region between young adults and the elderly while driving. , 2007, Journal of physiological anthropology.

[15]  Britton Chance,et al.  Functional Optical Brain Imaging Using Near-Infrared During Cognitive Tasks , 2004, Int. J. Hum. Comput. Interact..

[16]  Johan Engström,et al.  Effects of visual and cognitive load in real and simulated motorway driving , 2005 .

[17]  M. A. Recarte,et al.  COGNITIVE DEMANDS OF HANDS-FREE-PHONE CONVERSATION WHILE DRIVING , 2002 .

[18]  Myra Blanco,et al.  The impact of secondary task cognitive processing demand on driving performance. , 2006, Accident; analysis and prevention.

[19]  H Summala,et al.  Cognitive load and detection thresholds in car following situations: safety implications for using mobile (cellular) telephones while driving. , 1999, Accident; analysis and prevention.

[20]  C. Vaidya,et al.  Sensitivity of fNIRS to cognitive state and load , 2014, Front. Hum. Neurosci..

[21]  Hitoshi Tsunashima,et al.  Measurement of Brain Function of Car Driver Using Functional Near-Infrared Spectroscopy (fNIRS) , 2009, Comput. Intell. Neurosci..

[22]  A. Reiss,et al.  Sex differences in neural and behavioral signatures of cooperation revealed by fNIRS hyperscanning , 2016, Scientific reports.

[23]  A. Owen,et al.  Anterior prefrontal cortex: insights into function from anatomy and neuroimaging , 2004, Nature Reviews Neuroscience.

[24]  Emery N. Brown,et al.  Motion and Ballistocardiogram Artifact Removal for Interleaved Recording of EEG and EPs during MRI , 2002, NeuroImage.

[25]  S. Bunce,et al.  Detecting cognitive activity related hemodynamic signal for brain computer interface using functional near infrared spectroscopy , 2007, 2007 3rd International IEEE/EMBS Conference on Neural Engineering.

[26]  Toshinori Kato,et al.  Functional brain imaging using near-infrared spectroscopy during actual driving on an expressway , 2013, Front. Hum. Neurosci..

[27]  H. Koizumi,et al.  Development of an Optical Brain-machine Interface , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[28]  G. McArthur,et al.  Validation of the Emotiv EPOC® EEG gaming system for measuring research quality auditory ERPs , 2013, PeerJ.

[29]  Alan C. Evans,et al.  Evidence for a two-stage model of spatial working memory processing within the lateral frontal cortex: a positron emission tomography study. , 1996, Cerebral cortex.

[30]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines , 2009, UIST '09.

[31]  D. E Haigney,et al.  Concurrent mobile (cellular) phone use and driving performance: task demand characteristics and compensatory processes , 2000 .

[32]  D. Boas,et al.  HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain. , 2009, Applied optics.

[33]  Natasha Merat,et al.  Surrogate in-vehicle information systems and driver behaviour: effects of visual and cognitive load in simulated rural driving , 2005 .

[34]  David A. Boas,et al.  Motion artifacts in functional near-infrared spectroscopy: A comparison of motion correction techniques applied to real cognitive data , 2014, NeuroImage.

[35]  Wendy Ju,et al.  Monitoring driver cognitive load using functional near infrared spectroscopy in partially autonomous cars , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[36]  Yoshitaka Marumo,et al.  Measurement of Frontal Cortex Brain Activity Attributable to the Driving Workload and Increased Attention , 2009 .