Towards the automated recognition of assistance need for drivers with impaired visual field

Mobility enabled through driving is a crucial aspect of today’s social lives. It concerns young and elderly people and is critical for those among us suffering from visual field defects. Since driving primarily involves visual input, such people are often considered as unsafe drivers and banned from driving, although several recent studies, including our own, provide evidence that even severe visual field defects can be compensated through effective visual search strategies. In this context, this work pursues the challenging vision of adaptive driving assistance systems that take the visual deficits of the driver into account to enable a safer driving experience. The main challenges towards this vision are: (1) individual analysis and detection of visual field defects, (2) online analysis of visual search behavior, and (3) integrated analysis of visual deficits, search behavior, and traffic objects to identify and draw the driver’s attention towards potential hazards. Each of the above challenges is approached by customized methods. For (1), a mobile method for the assessment of the visual field and an algorithm for the recognition of the type of the visual field defect are proposed. For (2), an online probabilistic method is combined with algebraic analysis of the driver’s gaze. For (3), a detailed analysis of the driving scene is combined with the above methods to reliably detect hazardous traffic objects that might be overlooked by the driver. The methods were evaluated on real-world data from driving experiments with patients suffering from visual field defects. In combination, they improve over state-of-the-art techniques by being flexible, adaptive, and reliable. The feasibility of detecting objects that might be overlooked by the driver, and thus an adaptive assistance need, is demonstrated in different user studies. The methods developed in this work have a broad applicability that reaches beyond the driving context. Their application to a variety of tasks involving visual perception might help better understand its underlying mechanisms. Some of these tasks are already being investigated and will also be presented in this thesis.

[1]  Douglas DeCarlo,et al.  Robust clustering of eye movement recordings for quantification of visual interest , 2004, ETRA.

[2]  Nathan Congdon,et al.  Traffic Gap Judgment in People with Significant Peripheral Field Loss , 2008, Optometry and vision science : official publication of the American Academy of Optometry.

[3]  George W. McConkie,et al.  Evaluating and reporting data quality in eye movement research , 1980 .

[4]  H. Widdel.  Operational Problems in Analysing Eye Movements , 1984 .

[5]  Uwe Siebert,et al.  Long-Term Outcome After Stroke Evaluating Health-Related Quality of Life Using Utility Measurement , 2006 .

[6]  W. Rosenstiel,et al.  Driving with Binocular Visual Field Loss? A Study on a Supervised On-Road Parcours with Simultaneous Eye and Head Tracking , 2014, PloS one.

[7]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[8]  A. C. Kooijman,et al.  Hemianopic visual field defects elicit hemianopic scanning , 2002, Vision Research.

[9]  Robert N Weinreb,et al.  Patterns of glaucomatous visual field loss in sita fields automatically identified using independent component analysis. , 2009, Transactions of the American Ophthalmological Society.

[10]  Heikki Summala,et al.  Maintaining Lane Position with Peripheral Vision during In-Vehicle Tasks , 1996, Hum. Factors.

[11]  Erkan Mutlukan,et al.  Computerised campimetry with static dark-on-bright stimuli , 2005, Documenta Ophthalmologica.

[12]  Pierre Baldi,et al.  Assessing the accuracy of prediction algorithms for classification: an overview , 2000, Bioinform..

[13]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[14]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[15]  J. Henderson,et al.  Object-based attentional selection in scene viewing. , 2010, Journal of vision.

[16]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[17]  Luke Fletcher,et al.  Driver Inattention Detection based on Eye Gaze—Road Event Correlation , 2009, Int. J. Robotics Res..

[18]  D. S. Wooding,et al.  Fixation maps: quantifying eye-movement traces , 2002, ETRA.

[19]  C. Owsley,et al.  Is glaucoma associated with motor vehicle collision involvement and driving avoidance? , 2004, Investigative ophthalmology & visual science.

[20]  Chris A Johnson,et al.  Comparison of the new perimetric GATE strategy with conventional full-threshold and SITA standard strategies. , 2009, Investigative ophthalmology & visual science.

[21]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[22]  Oleg V. Komogortsev,et al.  Qualitative and quantitative scoring and evaluation of the eye movement classification algorithms , 2010, ETRA.

[23]  Brian Scassellati,et al.  The incomplete fixation measure , 2008, ETRA.

[24]  Daniel T. Larose,et al.  Discovering Knowledge in Data: An Introduction to Data Mining , 2005 .

[25]  Reinhard Vonthein,et al.  Comparison of Advanced Visual Field Defects Measured with the Tübingen Mobile Campimeter and the Octopus 101 Perimeter , 2010, European journal of ophthalmology.

[26]  Hanspeter A. Mallot,et al.  Collision avoidance in persons with homonymous visual field defects under virtual reality conditions , 2012, Vision Research.

[27]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[28]  Yean-Hoon Ong,et al.  A ‘web app’ for diagnosing hemianopia , 2012, Journal of Neurology, Neurosurgery & Psychiatry.

[29]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[30]  D. Ballard,et al.  Eye movements in natural behavior , 2005, Trends in Cognitive Sciences.

[31]  Peter A. Flach,et al.  Machine Learning - The Art and Science of Algorithms that Make Sense of Data , 2012 .

[32]  S K Mannan,et al.  Scanning the visual world: a study of patients with homonymous hemianopia , 2000, Journal of neurology, neurosurgery, and psychiatry.

[33]  Karen Bandeen-Roche,et al.  A prospective, population-based study of the role of visual impairment in motor vehicle crashes among older drivers: the SEE study. , 2007, Investigative ophthalmology & visual science.

[34]  Hanspeter A. Mallot,et al.  Gaze patterns predicting successful collision avoidance in patients with homonymous visual field defects , 2011, Vision Research.

[35]  Chabane Djeraba,et al.  Detecting Eye Fixations by Projection Clustering , 2007 .

[36]  Walter M. Jay,et al.  Clinical Neuro-Ophthalmology: A Practical Guide , 2008 .

[37]  C. Vomscheid,et al.  Analysis of eye tracking movements using innovations generated by a Kalman filter , 2006, Medical and Biological Engineering and Computing.

[38]  L. Itti.  Author address: , 1999 .

[39]  Hubert Kimmig,et al.  Visual search disorders beyond pure sensory failure in patients with acute homonymous visual field defects , 2009, Neuropsychologia.

[40]  J. Zihl,et al.  Visual scanning behavior in patients with homonymous hemianopia , 1995, Neuropsychologia.

[41]  Eli Peli,et al.  Driving with hemianopia, II: lane position and steering in a driving simulator. , 2010, Investigative ophthalmology & visual science.

[42]  M. Land.  Vision, eye movements, and natural behavior , 2009, Visual Neuroscience.

[43]  Susan E. George,et al.  Artificial neural network analysis of noisy visual field data in glaucoma , 1997, Artif. Intell. Medicine.

[44]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[45]  John Paulin Hansen,et al.  Evaluation of a low-cost open-source gaze tracker , 2010, ETRA.

[46]  Anders Heijl,et al.  Trained Artificial Neural Network for Glaucoma Diagnosis Using Visual Field Data: A Comparison With Conventional Algorithms , 2007, Journal of glaucoma.

[47]  F. Cornelissen,et al.  Driving and visuospatial performance in people with hemianopia , 2002 .

[48]  David Shinar,et al.  Imperfect In-Vehicle Collision Avoidance Warning Systems Can Aid Drivers , 2004, Hum. Factors.

[49]  David Keating,et al.  Visual field interpretation with a personal computer based neural network , 1994, Eye.

[50]  J. Szlyk,et al.  Driving Performance of Glaucoma Patients Correlates With Peripheral Visual Field Loss , 2005, Journal of glaucoma.

[51]  B. Damato,et al.  Clinical evaluation of a multi-fixation campimeter for the detection of glaucomatous visual field loss. , 1993, The British journal of ophthalmology.

[52]  Christoph Berger,et al.  GazeAlyze: a MATLAB toolbox for the analysis of eye movement data , 2012, Behavior research methods.

[53]  B Damato,et al.  Multifixation campimetry on line: a perimeter for the detection of visual field loss using the internet , 2003, The British journal of ophthalmology.

[54]  A. Sommer,et al.  Relationship between intraocular pressure and primary open angle glaucoma among white and black Americans. The Baltimore Eye Survey. , 1991, Archives of ophthalmology.

[55]  J F Taylor,et al.  Vision and driving. , 1987, Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians.

[56]  V. Ferrera.  Task-dependent modulation of the sensorimotor transformation for smooth pursuit eye movements. , 2000, Journal of neurophysiology.

[57]  Fiona J Rowe,et al.  Comparison of Damato campimetry and Humphrey automated perimetry results in a clinical population , 2010, British Journal of Ophthalmology.

[58]  M. Sloane,et al.  Visual processing impairment and risk of motor vehicle crash among older adults. , 1998, JAMA.

[59]  J. Barbur,et al.  Exploring Eye Movements in Patients with Glaucoma When Viewing a Driving Scene , 2010, PloS one.

[60]  Douglas R. Anderson.  Perimetry with and without automation , 1987 .

[61]  S K Mannan,et al.  Saccadic visual search training: a treatment for patients with homonymous hemianopia , 2004, Journal of Neurology, Neurosurgery & Psychiatry.

[62]  Laura Chamberlain.  Eye Tracking Methodology; Theory and Practice , 2007 .

[63]  Cynthia Owsley,et al.  On-road driving performance by persons with hemianopia and quadrantanopia. , 2009, Investigative ophthalmology & visual science.

[64]  Linda Wetzel,et al.  Development and assessment of a neuropsychological battery to aid in predicting driving performance. , 2002, Journal of rehabilitation research and development.

[65]  Robert N Weinreb,et al.  Assessing visual field clustering schemes using machine learning classifiers in standard perimetry. , 2007, Investigative ophthalmology & visual science.

[66]  C. Johnson,et al.  Incidence of visual field loss in 20,000 eyes and its relationship to driving performance. , 1983, Archives of ophthalmology.

[67]  Alex R. Bowers,et al.  A Pilot Evaluation of On-Road Detection Performance by Drivers with Hemianopia Using Oblique Peripheral Prisms , 2012, Stroke research and treatment.

[68]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[69]  H. Mallot,et al.  Assessment of vision-related quality of life in patients with homonymous visual field defects , 2007, Graefe's Archive for Clinical and Experimental Ophthalmology.

[70]  David G. Stork,et al.  Pattern classification, 2nd Edition , 2000 .

[71]  W. Marsden.  I and J , 2012 .

[72]  D B Henson,et al.  Frequency Distribution of Early Glaucomatous Visual Field Defects , 1986, American journal of optometry and physiological optics.

[73]  E. Kasten,et al.  Does visual restitution training change absolute homonymous visual field defects? A fundus controlled study , 2004, British Journal of Ophthalmology.

[74]  J. Horton,et al.  Let green lead not astray , 2005, British Journal of Ophthalmology.

[75]  Michael Henry Goldbaum,et al.  Unsupervised learning with independent component analysis can identify patterns of glaucomatous visual field defects. , 2006, Transactions of the American Ophthalmological Society.

[76]  E. Casson,et al.  The Impact of Visual Field Loss on Driving Performance: Evidence From On-Road Driving Assessments , 2005, Optometry and vision science : official publication of the American Academy of Optometry.

[77]  Robert A. Marino,et al.  Free viewing of dynamic stimuli by humans and monkeys. , 2009, Journal of vision.

[78]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[79]  Alex R. Bowers,et al.  Driving with Hemianopia , 2011 .

[80]  Robert L. Tomsak.  Perimetry , 1987, Neurology.

[81]  David West,et al.  A comparison of SOM neural network and hierarchical clustering methods , 1996 .

[82]  Michela Terenzi,et al.  ASTEF: A simple tool for examining fixations , 2008, Behavior research methods.

[83]  Mary Hayhoe,et al.  Visually-guided behavior of homonymous hemianopes in a naturalistic task , 2007, Vision Research.

[84]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[85]  Alexander Spratt,et al.  Visual function and fitness to drive. , 2008, British medical bulletin.

[86]  Alexander C. Schütz,et al.  Eye movements and perception: a selective review. , 2011, Journal of vision.

[87]  Wolfgang Fink.  Neural attractor network for application in visual field data classification. , 2004, Physics in medicine and biology.

[88]  Wolfgang Rosenstiel,et al.  Bayesian online clustering of eye movement data , 2012, ETRA.

[89]  H Strasburger,et al.  Automobile driving performance of brain-injured patients with visual field defects. , 1999, American journal of physical medicine & rehabilitation.

[90]  L Brigatti,et al.  Automatic detection of glaucomatous visual field progression with neural networks. , 1997, Archives of ophthalmology.

[91]  Valérie Biousse,et al.  Homonymous Hemianopia in Stroke , 2006, Journal of neuro-ophthalmology : the official journal of the North American Neuro-Ophthalmology Society.

[92]  P. Blignaut.  Fixation identification: The optimum threshold for a dispersion algorithm , 2009, Attention, perception & psychophysics.

[93]  D. Ballard,et al.  Eye guidance in natural vision: reinterpreting salience. , 2011, Journal of vision.

[94]  Eli Peli,et al.  Driving with hemianopia, I: Detection performance in a driving simulator. , 2009, Investigative Ophthalmology and Visual Science.

[95]  Dongheng Li,et al.  Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[96]  R. Leigh,et al.  The neurology of eye movements , 2006 .

[97]  A. Heijl,et al.  A PORTABLE HEMIANOPSIA TESTER , 1976 .

[98]  Oleg V Komogortsev,et al.  Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades , 2013, Behavior research methods.

[99]  Wolfgang Rosenstiel,et al.  Vishnoo — An open-source software for vision research , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[100]  J. Paetzold,et al.  Visual field defects in acute optic neuritis - distribution of different types of defect pattern, assessed with threshold-related supraliminal perimetry, ensuring high spatial resolution , 2008, Graefe's Archive for Clinical and Experimental Ophthalmology.

[101]  H. Wilhelm,et al.  Pupil campimetry in patients with visual field loss , 2005, European journal of neurology.

[102]  J. Caprioli,et al.  Detection of visual field progression in glaucoma with standard achromatic perimetry: A review and practical implications , 2011, Graefe's Archive for Clinical and Experimental Ophthalmology.

[103]  Cynthia Owsley,et al.  Hemianopic and quadrantanopic field loss, eye and head movements, and driving. , 2011, Investigative ophthalmology & visual science.

[104]  Wiebo H Brouwer,et al.  Predicting Practical Fitness to Drive in Drivers With Visual Field Defects Caused by Ocular Pathology , 2004, Hum. Factors.

[105]  Anders Heijl,et al.  Effects of input data on the performance of a neural network in distinguishing normal and glaucomatous visual fields. , 2005, Investigative ophthalmology & visual science.

[106]  B. Sabel,et al.  Recovery of visual field defects: a large clinical observational study using vision restoration therapy. , 2007, Restorative neurology and neuroscience.

[107]  D. Noton,et al.  Eye movements and visual perception. , 1971, Scientific American.

[108]  J. Zihl,et al.  Rehabilitation of Visual Disorders after Brain Injury , 2000 .

[109]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[110]  C. Kennard,et al.  Can visual function be restored in patients with homonymous hemianopia? , 1997, The British journal of ophthalmology.

[111]  A. Ultsch,et al.  Self-Organizing Neural Networks for Visualisation and Classification , 1993 .

[112]  Eberhard Zeeb.  Daimler's new full-scale, high-dynamic driving simulator - A technical overview , 2010 .

[113]  Darren R Gitelman,et al.  ILAB: A program for postexperimental eye movement analysis , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[114]  Victoria S Pelak,et al.  Homonymous hemianopia: A critical analysis of optical devices, compensatory training, and NovaVision , 2007, Current treatment options in neurology.

[115]  Jonathan W. Peirce,et al.  PsychoPy—Psychophysics software in Python , 2007, Journal of Neuroscience Methods.

[116]  Wiebo Brouwer,et al.  Compensatory viewing training improves practical fitness to drive of subjects with impaired vision , 2004 .

[117]  Wolfgang Rosenstiel,et al.  RELIABLE CLASSIFICATION OF VISUAL FIELD DEFECTS IN AUTOMATED PERIMETRY USING CLUSTERING , 2010, BioMed 2010.

[118]  H. Quigley,et al.  The number of people with glaucoma worldwide in 2010 and 2020 , 2006, British Journal of Ophthalmology.

[119]  A Mandahl,et al.  Red square test for visual field screening , 1994, Acta ophthalmologica.

[120]  William Whittaker,et al.  Autonomous driving in urban environments: Boss and the Urban Challenge , 2008 .

[121]  Wolfgang Rosenstiel,et al.  A New Method for Assessing the Exploratory Field of View (EFOV) , 2013, HEALTHINF.

[122]  Refractor.  Vision , 2000, The Lancet.

[123]  D Keating,et al.  A back propagation neural network for the classification of visual field data , 1993 .

[124]  Wolfgang Rosenstiel,et al.  Online Classification of Eye Tracking Data for Automated Analysis of Traffic Hazard Perception , 2013, ICANN.

[125]  Georg Pölzlbauer.  Survey and Comparison of Quality Measures for Self-Organizing Maps , 2004 .

[126]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[127]  M. Brigell,et al.  Effects of Age and Hemianopic Visual Field Loss on Driving , 1993, Optometry and vision science : official publication of the American Academy of Optometry.

[128]  Bernhard A. Sabel,et al.  Computer-based training for the treatment of partial blindness , 1998, Nature Medicine.

[129]  R. Greenberg.  Biometry , 1969, The Yale Journal of Biology and Medicine.

[130]  D B Henson,et al.  Spatial classification of glaucomatous visual field loss. , 1996, The British journal of ophthalmology.

[131]  L. A. Chiasson,et al.  Glaucoma and on-road driving performance. , 2008, Investigative ophthalmology & visual science.

[132]  Uwe Siebert,et al.  Long-Term Outcome After Stroke: Evaluating Health-Related Quality of Life Using Utility Measurements , 2006, Stroke.

[133]  Amos J Storkey,et al.  Machine Learning and Pattern Recognition : Preliminaries Course , 2009 .

[134]  Jr. G. Forney,et al.  The viterbi algorithm , 1973 .

[135]  C. Owsley,et al.  Visual field defects and the risk of motor vehicle collisions among patients with glaucoma. , 2005, Investigative ophthalmology & visual science.

[136]  F. Chédru,et al.  Visual searching in normal and brain-damaged subjects (contribution to the study of unilateral inattention). , 1973, Cortex; a journal devoted to the study of the nervous system and behavior.

[137]  Robert N Weinreb,et al.  Comparing machine learning classifiers for diagnosing glaucoma from standard automated perimetry. , 2002, Investigative ophthalmology & visual science.

[138]  Nancy Millette,et al.  How People Look at Pictures , 1935 .

[139]  B. Tatler,et al.  Looking and Acting: Vision and eye movements in natural behaviour , 2009 .

[140]  R P Mills,et al.  Screening for glaucoma with frequency-doubling technology and Damato campimetry. , 1999, Archives of ophthalmology.

[141]  G. J. van der Wildt,et al.  Visual training of cerebral blindness patients gradually enlarges the visual field , 2009, British Journal of Ophthalmology.

[142]  Sven Ohl,et al.  Driving Simulation in the Clinic: Testing Visual Exploratory Behavior in Daily Life Activities in Patients with Visual Field Defects , 2012, Journal of visualized experiments : JoVE.

[143]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[144]  L. Kline,et al.  Evaluation of on-road driving in people with hemianopia and quadrantanopia. , 2010, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[145]  Jon Currie,et al.  Homonymous Visual Field Defects and Stroke in an Older Population , 2002, Stroke.

[146]  廣瀬雄一,et al.  Neuroscience , 2019, Workplace Attachments.

[147]  M H Goldbaum,et al.  Interpretation of automated perimetry for glaucoma by neural network. , 1994, Investigative ophthalmology & visual science.

[148]  Bernhard A. Sabel,et al.  Visual field recovery after vision restoration therapy (VRT) is independent of eye movements: An eye tracker study , 2006, Behavioural Brain Research.