Analyzing the Time Course of Pupillometric Data

This article provides a tutorial for analyzing pupillometric data. Pupil dilation has become increasingly popular in psychological and psycholinguistic research as a measure to trace language processing. However, there is no general consensus about procedures to analyze the data, with most studies analyzing extracted features from the pupil dilation data instead of analyzing the pupil dilation trajectories directly. Recent studies have started to apply nonlinear regression and other methods to analyze the pupil dilation trajectories directly, utilizing all available information in the continuously measured signal. This article applies a nonlinear regression analysis, generalized additive mixed modeling, and illustrates how to analyze the full-time course of the pupil dilation signal. The regression analysis is particularly suited for analyzing pupil dilation in the fields of psychological and psycholinguistic research because generalized additive mixed models can include complex nonlinear interactions for investigating the effects of properties of stimuli (e.g., formant frequency) or participants (e.g., working memory score) on the pupil dilation signal. To account for the variation due to participants and items, nonlinear random effects can be included. However, one of the challenges for analyzing time series data is dealing with the autocorrelation in the residuals, which is rather extreme for the pupillary signal. On the basis of simulations, we explain potential causes of this extreme autocorrelation, and on the basis of the experimental data, we show how to reduce their adverse effects, allowing a much more coherent interpretation of pupillary data than possible with feature-based techniques.

[1]  Jacolien van Rij,et al.  Individual Differences in Pupil Dilation during Naming Task , 2016, CogSci.

[2]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[3]  R. Tibshirani,et al.  Generalized Additive Models , 1986 .

[4]  James O. Ramsay,et al.  Applied Functional Data Analysis: Methods and Case Studies , 2002 .

[5]  L. Busse,et al.  Improving behavioral performance under full attention by adjusting response criteria to changes in stimulus predictability. , 2012, Journal of vision.

[6]  D. Whitaker,et al.  Factors affecting light-adapted pupil size in normal human subjects. , 1994, Investigative ophthalmology & visual science.

[7]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[8]  Matthew B. Winn,et al.  The Impact of Auditory Spectral Resolution on Listening Effort Revealed by Pupil Dilation , 2015, Ear and hearing.

[9]  Christoph Scheepers,et al.  Constituent order priming from reading to listening: a visual-world study , 2004 .

[10]  Jelle R. Dalenberg,et al.  Pupil Dilation Co-Varies with Memory Strength of Individual Traces in a Delayed Response Paired-Associate Task , 2012, PloS one.

[11]  X. Lin,et al.  Inference in generalized additive mixed modelsby using smoothing splines , 1999 .

[12]  R. Baayen,et al.  Mixed Effects Regression Models in Linguistics , 2016 .

[13]  Qing Yang,et al.  The Pupil Reflects Motor Preparation for Saccades – Even before the Eye Starts to Move , 2011, Front. Hum. Neurosci..

[14]  W. Levelt,et al.  Pupillary dilation as a measure of attention: a quantitative system analysis , 1993 .

[15]  Michael Schluroff Pupil responses to grammatical complexity of sentences , 1982, Brain and Language.

[16]  Peter Hendrix,et al.  Distinct ERP Signatures of Word Frequency, Phrase Frequency, and Prototypicality in Speech Production , 2017, Journal of experimental psychology. Learning, memory, and cognition.

[17]  J Hyönä,et al.  Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks , 1995, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[18]  Birte U. Forstmann,et al.  Trial-by-trial fluctuations in CNV amplitude reflect anticipatory adjustment of response caution , 2014, NeuroImage.

[19]  Petar Milin,et al.  Discrimination in lexical decision , 2017, PloS one.

[20]  R. Harald Baayen,et al.  Practice makes perfect: the consequences of lexical proficiency for articulation , 2018, Linguistics Vanguard.

[21]  Jacolien van Rij,et al.  Pronoun Processing. Computational, behavioral, and psychophysiological studies in children and adults , 2012 .

[22]  S. Sirois,et al.  Pupillometry , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[23]  Sylvain Sirois,et al.  Infant cognition: going full factorial with pupil dilation. , 2009, Developmental science.

[24]  M. Just,et al.  The intensity dimension of thought: pupillometric indices of sentence processing. , 1993, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[25]  Jonathan Grainger,et al.  The pupillary light response reflects eye-movement preparation. , 2014, Journal of experimental psychology. Human perception and performance.

[26]  Taylor R. Hayes,et al.  Mapping and correcting the influence of gaze position on pupil size measurements , 2015, Behavior Research Methods.

[27]  Stefanie E. Kuchinsky,et al.  Pupil size varies with word listening and response selection difficulty in older adults with hearing loss. , 2013, Psychophysiology.

[28]  Cross-category phonological effects on ERP amplitude demonstrate context-specific processing during reading aloud , 2019 .

[29]  A. Newman,et al.  Modeling nonlinear relationships in ERP data using mixed-effects regression with R examples. , 2015, Psychophysiology.

[30]  J. Beatty,et al.  The pupillary system. , 2000 .

[31]  Ingo Plag,et al.  Prominence in Triconstituent Compounds: Pitch Contours and Linguistic Theory , 2013, Language and speech.

[32]  Jacolien van Rij,et al.  Autocorrelated Errors in Experimental Data in the Language Sciences: Some Solutions Offered by Generalized Additive Mixed Models , 2016, 1601.02043.

[33]  Martijn Wieling Analyzing dynamic phonetic data using generalized additive mixed modeling: A tutorial focusing on articulatory differences between L1 and L2 speakers of English , 2018, J. Phonetics.

[34]  Mathôt Sebastiaan,et al.  A simple way to reconstruct pupil size during eye blinks , 2013 .

[35]  Petra Hendriks,et al.  Pupillary responses reflect ambiguity resolution in pronoun processing , 2016 .

[36]  A. Zekveld,et al.  Cognitive Load During Speech Perception in Noise: The Influence of Age, Hearing Loss, and Cognition on the Pupil Response , 2011, Ear and hearing.

[37]  E. Granholm,et al.  Pupillary responses on the visual backward masking task reflect general cognitive ability. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[38]  Jessie S. Nixon,et al.  The temporal dynamics of perceptual uncertainty: eye movement evidence from Cantonese segment and tone perception , 2016 .

[39]  Martina Zellin,et al.  In the eye of the listener: pupil dilation elucidates discourse processing. , 2011, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[40]  Roger M. Cooper,et al.  The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. , 1974 .

[41]  Alan Y. Chiang,et al.  Generalized Additive Models: An Introduction With R , 2007, Technometrics.

[42]  Sylvain Sirois,et al.  Pupil diameter measurement errors as a function of gaze direction in corneal reflection eyetrackers , 2013, Behavior research methods.

[43]  V. Carey,et al.  Mixed-Effects Models in S and S-Plus , 2001 .

[44]  D. Mirman,et al.  Statistical and computational models of the visual world paradigm: Growth curves and individual differences. , 2008, Journal of memory and language.

[45]  S. Wood Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models , 2011 .

[46]  J. Beatty Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[47]  Stefan M. Wierda,et al.  Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution , 2012, Proceedings of the National Academy of Sciences.

[48]  Hans-Georg Müller,et al.  Functional Data Analysis , 2016 .

[49]  E. Hess,et al.  Pupil Size as Related to Interest Value of Visual Stimuli , 1960, Science.

[50]  S. Wood,et al.  Generalized additive models for large data sets , 2015 .

[51]  F. Hutzler,et al.  Systematic influence of gaze position on pupil size measurement: analysis and correction , 2011, Behavior research methods.

[52]  J. Yellott,et al.  A unified formula for light-adapted pupil size. , 2012, Journal of vision.

[53]  B. Goldwater Psychological significance of pupillary movements. , 1972, Psychological bulletin.

[54]  Jacolien van RIJ,et al.  Alternative quantitative methods in psycholinguistics: Implications for theory and design , 2020, Word Knowledge and Word Usage.

[55]  Jacolien van Rij,et al.  itsadug: Interpreting Time Series and Autocorrelated Data Using GAMMs , 2015 .

[56]  E. Hess,et al.  Pupillometry: The Psychology of the Pupillary Response , 1978 .

[57]  Arno Koning,et al.  Multiple object tracking: anticipatory attention doesn't "bounce". , 2012, Journal of vision.

[58]  Elena G. Patsenko,et al.  The Quarterly Journal of Experimental Psychology Pupillometry Reveals Processing Load during Spoken Language Comprehension , 2022 .

[59]  J. Bradshaw Pupil size and drag state in a reaction time task , 1970 .

[60]  S. Wood,et al.  Smoothing Parameter and Model Selection for General Smooth Models , 2015, 1511.03864.

[61]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[62]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[63]  Reinhold Kliegl,et al.  The cave of Shadows. Addressing the human factor with generalized additive mixed models , 2015, 1511.03120.

[64]  A. Jacobs,et al.  Pupillary responses during lexical decisions vary with word frequency but not emotional valence. , 2007, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[65]  Jacolien van Rij,et al.  Children's eye gaze reveals their use of discourse context in object pronoun resolution , 2016 .

[66]  Klaus Hofmeister,et al.  Pupillary responses to syntactic ambiguity of sentences , 1986, Brain and Language.