Combined fMRI- and eye movement-based decoding of bistable plaid motion perception

&NA; The phenomenon of bistable perception, in which perception alternates spontaneously despite constant sensory stimulation, has been particularly useful in probing the neural bases of conscious perception. The study of such bistability requires access to the observer's perceptual dynamics, which is usually achieved via active report. This report, however, constitutes a confounding factor in the study of conscious perception and can also be biased in the context of certain experimental manipulations. One approach to circumvent these problems is to track perceptual alternations using signals from the eyes or the brain instead of observers' reports. Here we aimed to optimize such decoding of perceptual alternations by combining eye and brain signals. Eye‐tracking and functional magnetic resonance imaging (fMRI) was performed in twenty participants while they viewed a bistable visual plaid motion stimulus and reported perceptual alternations. Multivoxel pattern analysis (MVPA) for fMRI was combined with eye‐tracking in a Support vector machine to decode participants' perceptual time courses from fMRI and eye‐movement signals. While both measures individually already yielded high decoding accuracies (on average 86% and 88% correct, respectively) classification based on the two measures together further improved the accuracy (91% correct). These findings show that leveraging on both fMRI and eye movement data may pave the way for optimized no‐report paradigms through improved decodability of bistable motion perception and hence for a better understanding of the neural correlates of consciousness. HighlightsBistable plaid motion perception was decoded from eye and brain signals.Combining data from eye tracking and fMRI achieved very high decoding accuracies.The proposed experimental setup and analysis will help improving no‐report paradigms.

[1]  Alexis Hervais-Adelman,et al.  Unconscious evaluative conditioning: Attitude change induced by invisible words , 2012 .

[2]  P ENOKSSON,et al.  A METHOD FOR INVESTIGATION OF OCULAR DOMINANCE BASED ON OPTOKINETIC NYSTAGMUS , 1961, Acta ophthalmologica.

[3]  P. Wenderoth,et al.  The depth and selectivity of suppression in binocular rivalry , 2001, Perception & psychophysics.

[4]  A. Norcia,et al.  A method for investigating binocular rivalry in real-time with the steady-state VEP , 1997, Vision Research.

[5]  G. Alpers,et al.  Here is looking at you: emotional faces predominate in binocular rivalry. , 2007, Emotion.

[6]  R. Lansing Electroencephalographic Correlates of Binocular Rivalry in Man , 1964, Science.

[7]  V. Lamme,et al.  No-Report Paradigms: Extracting the True Neural Correlates of Consciousness , 2015, Trends in Cognitive Sciences.

[8]  Philipp Sterzer,et al.  A neural basis for inference in perceptual ambiguity , 2007, Proceedings of the National Academy of Sciences.

[9]  R. Blake,et al.  Multistable Perception and the Role of the Frontoparietal Cortex in Perceptual Inference. , 2018, Annual review of psychology.

[10]  Philipp Sterzer,et al.  Decoding pattern motion information in V1 , 2013, Cortex.

[11]  Lisa Feldman Barrett,et al.  The Visual Impact of Gossip , 2011, Science.

[12]  Synsoplevede Figurer. Studier i psykologisk Analyse , 1922 .

[13]  Lauri Parkkonen,et al.  Early visual brain areas reflect the percept of an ambiguous scene , 2008, Proceedings of the National Academy of Sciences.

[14]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[15]  Mark D'Esposito,et al.  Variation of BOLD hemodynamic responses across subjects and brain regions and their effects on statistical analyses , 2004, NeuroImage.

[16]  Michael S. Pratte,et al.  Decoding patterns of human brain activity. , 2012, Annual review of psychology.

[17]  J. Steffener,et al.  Investigating Hemodynamic Response Variability at the Group Level using Basis Functions , 2009, NeuroImage.

[18]  Jean Lorenceau,et al.  Pupil dynamics during bistable motion perception. , 2009, Journal of vision.

[19]  M. Orne On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[20]  N. Logothetis,et al.  Is the frontal lobe involved in conscious perception? , 2014, Front. Psychol..

[21]  C. Clifford Binocular rivalry , 2009, Current Biology.

[22]  W. Einhäuser,et al.  Binocular Rivalry: Frontal Activity Relates to Introspection and Action But Not to Perception , 2014, The Journal of Neuroscience.

[23]  Martin Wiesmann,et al.  Asymmetric modulation of human visual cortex activity during 10° lateral gaze (fMRI study) , 2005, NeuroImage.

[24]  Nava Rubin,et al.  The dynamics of bi-stable alternation in ambiguous motion displays: a fresh look at plaids , 2003, Vision Research.

[25]  Jean Lorenceau,et al.  Pupil dynamics during bistable form/motion binding , 2010 .

[26]  Nava Rubin,et al.  The oblique plaid effect , 2004, Vision Research.

[27]  G. Rees,et al.  Predicting the Stream of Consciousness from Activity in Human Visual Cortex , 2005, Current Biology.

[28]  Bryan R. Conroy,et al.  A Common, High-Dimensional Model of the Representational Space in Human Ventral Temporal Cortex , 2011, Neuron.

[29]  K. Stephan,et al.  Learning What to See in a Changing World , 2016, Front. Hum. Neurosci..

[30]  L. Lack Selective attention and the control of binocular rivalry , 1978 .

[31]  B. Scholl,et al.  Cognition does not affect perception: Evaluating the evidence for “top-down” effects , 2015, Behavioral and Brain Sciences.

[32]  Ralf Engbert,et al.  Fixational eye movements predict the perceived direction of ambiguous apparent motion. , 2008, Journal of vision.

[33]  Daniel Arteaga,et al.  Brain mechanisms for simple perception and bistable perception , 2013, Proceedings of the National Academy of Sciences.

[34]  Justin L. Gardner,et al.  Modulation of Visual Responses by Gaze Direction in Human Visual Cortex , 2013, The Journal of Neuroscience.

[35]  Michel Dojat,et al.  A BOLD signature of eyeblinks in the visual cortex , 2012, NeuroImage.

[36]  Wolfgang Einhäuser,et al.  Reward modulates perception in binocular rivalry. , 2015, Journal of vision.

[37]  Geraint Rees,et al.  Knowing with Which Eye We See: Utrocular Discrimination and Eye-Specific Signals in Human Visual Cortex , 2010, PloS one.

[38]  Karl J. Friston,et al.  Predictive coding explains binocular rivalry: An epistemological review , 2008, Cognition.

[39]  Wolfgang Einhäuser,et al.  Perceptual Rivalry: Reflexes Reveal the Gradual Nature of Visual Awareness , 2011, PloS one.

[40]  Philipp Sterzer,et al.  fMRI-based decoding of reward effects in binocular rivalry , 2017, Neuroscience of consciousness.

[41]  R. Blake,et al.  Negligible fronto-parietal BOLD activity accompanying unreportable switches in bistable perception , 2015, Nature Neuroscience.

[42]  N. Logothetis,et al.  Multistable phenomena: changing views in perception , 1999, Trends in Cognitive Sciences.

[43]  R. van Ee,et al.  Visual Cortex Allows Prediction of Perceptual States during Ambiguous Structure-From-Motion , 2007, The Journal of Neuroscience.

[44]  P. Walker Binocular rivalry : Central or peripheral selective processes? , 1978 .

[45]  Miriam Spering,et al.  Tracking Without Perceiving , 2011, Psychology Science.

[46]  R. Blake,et al.  Do recognizable figures enjoy an advantage in binocular rivalry? , 1992, Journal of experimental psychology. Human perception and performance.

[47]  M. Erdelyi A new look at the new look: perceptual defense and vigilance. , 1974, Psychological review.

[48]  Andrew T. Smith,et al.  Representation of eye position in the human parietal cortex. , 2010, Journal of neurophysiology.

[49]  G. Rees,et al.  The Neural Bases of Multistable Perception , 2022 .

[50]  Philipp Sterzer,et al.  Reinforcement of perceptual inference: reward and punishment alter conscious visual perception during binocular rivalry , 2014, Front. Psychol..

[51]  Paul Pauli,et al.  Binocular rivalry between emotional and neutral stimuli: a validation using fear conditioning and EEG. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[52]  M. Wiesmann,et al.  Asymmetric modulation of human visual cortex activity during 10 degrees lateral gaze (fMRI study). , 2005, NeuroImage.

[53]  Ludovico Minati,et al.  A Neural Network Approach to fMRI Binocular Visual Rivalry Task Analysis , 2014, PloS one.

[54]  Farid Masrour,et al.  Revisiting the empirical case against perceptual modularity , 2015, Front. Psychol..

[55]  Erich W. Graf,et al.  Extrinsic factors in the perception of bistable motion stimuli , 2010, Vision Research.

[56]  R. Yee,et al.  Optokinetic nystagmus and parietal lobe lesions , 1980, Annals of neurology.

[57]  W. Levelt On binocular rivalry , 1965 .

[58]  R. Blake,et al.  Endogenous attention prolongs dominance durations in binocular rivalry. , 2005, Journal of vision.

[59]  Emily Balcetis,et al.  Subjective value determines initial dominance in binocular rivalry , 2012 .

[60]  J. Haynes A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives , 2015, Neuron.

[61]  Peng Zhang,et al.  Binocular Rivalry Requires Visual Attention , 2011, Neuron.

[62]  M. V. von Grünau,et al.  Ambiguous plaids: switching between coherence and transparency. , 1993, Spatial vision.

[63]  E. Adelson,et al.  Phenomenal coherence of moving visual patterns , 1982, Nature.

[64]  J Lorenceau,et al.  Pupil dilation does not predict subsequent stability in perceptual rivalry , 2008, Proceedings of the National Academy of Sciences.

[65]  John-Dylan Haynes,et al.  Delusions and the role of beliefs in perceptual inference , 2012 .

[66]  Randolph Blake,et al.  The Role of Frontal and Parietal Brain Areas in Bistable Perception , 2011, The Journal of Neuroscience.

[67]  Manabu Tanifuji,et al.  Which image is in awareness during binocular rivalry? Reading perceptual status from eye movements. , 2012, Journal of vision.