PupilNet, Measuring Task Evoked Pupillary Response using Commodity RGB Tablet Cameras

Pupillary diameter monitoring has been proven successful at objectively measuring cognitive load that might otherwise be unobservable. This paper compares three different algorithms for measuring cognitive load using commodity cameras. We compare the performance of modified starburst algorithm (from previous work) and propose two new algorithms: 2 Level Snakuscules and a convolutional neural network which we call PupilNet. In a user study with eleven participants, our comparisons show PupilNet outperforms other algorithms in measuring pupil dilation, is robust to various lighting conditions, and robust to different eye colors. We show that the difference between PupilNet and a gold standard head-mounted gaze tracker varies only from -2.6% to 2.8%. Finally, we also show that PupilNet gives similar conclusions about cognitive load during a longer duration typing task.

[1]  Pan Du,et al.  Bioinformatics Original Paper Improved Peak Detection in Mass Spectrum by Incorporating Continuous Wavelet Transform-based Pattern Matching , 2022 .

[2]  Maria K. Eckstein,et al.  Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? , 2016, Developmental Cognitive Neuroscience.

[3]  C. Berridge,et al.  The locus coeruleus–noradrenergic system: modulation of behavioral state and state-dependent cognitive processes , 2003, Brain Research Reviews.

[4]  W W Tryon,et al.  Pupillometry: a survey of sources of variation. , 1975, Psychophysiology.

[5]  Daniel Lafond,et al.  HCI Dilemmas for Context-Aware Support in Intelligence Analysis , 2014 .

[6]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[7]  Michele Miozzo,et al.  Pupillary Stroop effects , 2010, Cognitive Processing.

[8]  Yann LeCun,et al.  Generalization and network design strategies , 1989 .

[9]  Fang Chen,et al.  Pupillary Response Based Cognitive Workload Measurement under Luminance Changes , 2011, INTERACT.

[10]  Wayne J. Ryan,et al.  Adapting Starburst for Elliptical Iris Segmentation , 2008, 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems.

[11]  Suku Nair,et al.  PupilWare: towards pervasive cognitive load measurement using commodity devices , 2015, PETRA.

[12]  N. Cowan,et al.  Chunk limits and length limits in immediate recall: a reconciliation. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[13]  Qiong Huang,et al.  TabletGaze: Unconstrained Appearance-based Gaze Estimation in Mobile Tablets , 2015 .

[14]  G. Aston-Jones,et al.  Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[15]  Brian P. Bailey,et al.  Categories & Subject Descriptors: H.5.2 [Information , 2022 .

[16]  D. J. Marcus,et al.  Regulation of cognitive resources during sustained attention and working memory in 10-year-olds and adults. , 2007, Psychophysiology.

[17]  D Kahneman,et al.  Pupil Diameter and Load on Memory , 1966, Science.

[18]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[19]  R. Savin-Williams,et al.  The Eyes Have It: Sex and Sexual Orientation Differences in Pupil Dilation Patterns , 2012, PloS one.

[20]  A.B. Albu,et al.  A computer vision-based system for real-time detection of sleep onset in fatigued drivers , 2008, 2008 IEEE Intelligent Vehicles Symposium.

[21]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[22]  Kunihiko Fukushima,et al.  Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition , 1982 .

[23]  J. Diamond,et al.  Sclerotomy complications following pars plana vitrectomy , 2000, The British journal of ophthalmology.

[24]  Andreas Bulling,et al.  EyeTab: model-based gaze estimation on unmodified tablet computers , 2014, ETRA.

[25]  Roland Brünken,et al.  Cognitive Load Theory: THEORY , 2010 .

[26]  Daniel Afergan,et al.  Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State , 2016, CHI.

[27]  Ching Y. Suen,et al.  Unideal Iris Segmentation Using Region-Based Active Contour Model , 2010, ICIAR.

[28]  U. Erdem,et al.  Investigation of Autonomic Nervous System Functions by Pupillometry in Children with Attention-Deficit/ Hyperactivity Disorder , 2013 .

[29]  J. Beatty,et al.  The pupillary system. , 2000 .

[30]  J. Beatty.  Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[31]  Peter Robinson,et al.  Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[32]  Gregory D. Abowd,et al.  Infrastructure mediated sensing , 2008 .

[33]  Suku Nair,et al.  Work-in-Progress, PupilWare-M: Cognitive Load Estimation Using Unmodified Smartphone Cameras , 2015, 2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems.

[34]  John Daugman,et al.  How iris recognition works , 2002, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Edward Cutrell,et al.  Accurate eye center localization using Snakuscule , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[36]  Hans-Werner Gellersen,et al.  Partially-indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction , 2016, CHI.

[37]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[38]  Jonathan D. Cohen,et al.  An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. , 2005, Annual review of neuroscience.

[39]  Zahra Hakimi,et al.  SET: a pupil detection method using sinusoidal approximation , 2015, Front. Neuroeng..

[40]  Dongheng Li,et al.  Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[41]  Richard P. Wildes,et al.  A system for automated iris recognition , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[42]  Pat Hanrahan,et al.  Measuring the task-evoked pupillary response with a remote eye tracker , 2008, ETRA.

[43]  Neil A. Dodgson,et al.  Robust real-time pupil tracking in highly off-axis images , 2012, ETRA.

[44]  Wolfgang Rosenstiel,et al.  ExCuSe: Robust Pupil Detection in Real-World Scenarios , 2015, CAIP.

[45]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[47]  Venu Govindaraju,et al.  A Robust Iris Localization Method Using an Active Contour Model and Hough Transform , 2010, 2010 20th International Conference on Pattern Recognition.

[48]  Michael Unser,et al.  The Snakuscule , 2006, 2006 International Conference on Image Processing.

[49]  P. Beauseroy,et al.  Hough Transform and Active Contour for Enhanced Iris Segmentation , 2012 .

[50]  Kenneth O. Stanley,et al.  Revising the evolutionary computation abstraction: minimal criteria novelty search , 2010, GECCO '10.