Beyond Trial-Based Paradigms: Continuous Behavior, Ongoing Neural Activity, and Natural Stimuli

The vast majority of experiments examining perception and behavior are conducted using experimental paradigms that adhere to a rigid trial structure: each trial consists of a brief and discrete series of events and is regarded as independent from all other trials. The assumptions underlying this structure ignore the reality that natural behavior is rarely discrete, brain activity follows multiple time courses that do not necessarily conform to the trial structure, and the natural environment has statistical structure and dynamics that exhibit long-range temporal correlation. Modern advances in statistical modeling and analysis offer tools that make it feasible for experiments to move beyond rigid independent and identically distributed trial structures. Here we review literature that serves as evidence for the feasibility and advantages of moving beyond trial-based paradigms to understand the neural basis of perception and cognition. Furthermore, we propose a synthesis of these efforts, integrating the characterization of natural stimulus properties with measurements of continuous neural activity and behavioral outputs within the framework of sensory-cognitive-motor loops. Such a framework provides a basis for the study of natural statistics, naturalistic tasks, and/or slow fluctuations in brain activity, which should provide starting points for important generalizations of analytical tools in neuroscience and subsequent progress in understanding the neural basis of perception and cognition.

[1]  Joseph G. Makin,et al.  Learning to Estimate Dynamical State with Probabilistic Population Codes , 2015, PLoS Comput. Biol..

[2]  B A J Reddi,et al.  Accuracy, information, and response time in a saccadic decision task. , 2003, Journal of neurophysiology.

[3]  Bing Liu,et al.  Spatiotemporal Filter for Visual Motion Integration from Pursuit Eye Movements in Humans and Monkeys , 2017, The Journal of Neuroscience.

[4]  Michael J. Black,et al.  Modeling and decoding motor cortical activity using a switching Kalman filter , 2004, IEEE Transactions on Biomedical Engineering.

[5]  Claude Bédard,et al.  Comparative power spectral analysis of simultaneous elecroencephalographic and magnetoencephalographic recordings in humans suggests non-resistive extracellular media , 2010, Journal of Computational Neuroscience.

[6]  Ehtibar N Dzhafarov,et al.  Unfalsifiability and mutual translatability of major modeling schemes for choice reaction time. , 2014, Psychological review.

[7]  I. Nelken,et al.  Sensitivity to Complex Statistical Regularities in Rat Auditory Cortex , 2012, Neuron.

[8]  Andrew W. Kraft,et al.  Spontaneous Infra-slow Brain Activity Has Unique Spatiotemporal Dynamics and Laminar Structure , 2018, Neuron.

[9]  Biyu J. He,et al.  Scale-Free Neural and Physiological Dynamics in Naturalistic Stimuli Processing , 2016, eNeuro.

[10]  Alexandre Pouget,et al.  Optimal Sensorimotor Integration in Recurrent Cortical Networks: A Neural Implementation of Kalman Filters , 2007, The Journal of Neuroscience.

[11]  The challenge of measuring long-term positive aftereffects , 2013, Current Biology.

[12]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[13]  M. Raichle The brain's default mode network. , 2015, Annual review of neuroscience.

[14]  P. Mamassian,et al.  Predictive Properties of Visual Adaptation , 2012, Current Biology.

[15]  Wilson S. Geisler,et al.  Optimal eye movement strategies in visual search , 2005, Nature.

[16]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[17]  G. Barnes,et al.  Factors affecting the predictability of pseudo‐random motion stimuli in the pursuit reflex of man. , 1989, The Journal of physiology.

[18]  David J. Freedman,et al.  A hierarchy of intrinsic timescales across primate cortex , 2014, Nature Neuroscience.

[19]  Brian Maniscalco,et al.  Initial-state-dependent, robust, transient neural dynamics encode conscious visual perception , 2017, bioRxiv.

[20]  Zhaoyu Wei,et al.  The plunging cavities formed by the impinged jet after the entry of a sphere into water , 2014, J. Vis..

[21]  Andreas Kleinschmidt,et al.  Spontaneous local variations in ongoing neural activity bias perceptual decisions , 2008, Proceedings of the National Academy of Sciences.

[22]  Jacob L Yates,et al.  The Role of the Lateral Intraparietal Area in (the Study of) Decision Making. , 2017, Annual review of neuroscience.

[23]  D L Gilden,et al.  1/f noise in human cognition. , 1995, Science.

[24]  E. J. Morris,et al.  Visual motion processing and sensory-motor integration for smooth pursuit eye movements. , 1987, Annual review of neuroscience.

[25]  G. Rhodes,et al.  How is facial expression coded? , 2015, Journal of vision.

[26]  C. Honey,et al.  Hierarchical process memory: memory as an integral component of information processing , 2015, Trends in Cognitive Sciences.

[27]  D. Gilden Cognitive emissions of 1/f noise. , 2001, Psychological review.

[28]  Il Memming Park,et al.  Functional dissection of signal and noise in MT and LIP during decision-making , 2017, Nature Neuroscience.

[29]  David J. Freedman,et al.  Neuronal Mechanisms of Visual Categorization: An Abstract View on Decision Making. , 2016, Annual review of neuroscience.

[30]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[31]  Claude Bédard,et al.  Comparative power spectral analysis of simultaneous elecroencephalographic and magnetoencephalographic recordings in humans suggests non-resistive extracellular media , 2010, Journal of Computational Neuroscience.

[32]  Kathryn Bonnen,et al.  Dynamic mechanisms of visually guided 3D motion tracking. , 2017, Journal of neurophysiology.

[33]  John A. Nevin,et al.  SIGNAL DETECTION THEORY AND OPERANT BEHAVIOR: A Review of David M. Green and John A. Swets' Signal Detection Theory and Psychophysics.1 , 1969 .

[34]  H. Seo,et al.  A reservoir of time constants for memory traces in cortical neurons , 2011, Nature Neuroscience.

[35]  W. Geisler,et al.  Optimal Eye Movement Strategies in Visual Search ( Supplement ) , 2005 .

[36]  Jeffrey N. Rouder,et al.  Modeling Response Times for Two-Choice Decisions , 1998 .

[37]  Philip L. Smith,et al.  The accumulator model of two-choice discrimination , 1988 .

[38]  D. Heeger,et al.  Slow Cortical Dynamics and the Accumulation of Information over Long Timescales , 2012, Neuron.

[39]  S. Petersen,et al.  Brain Networks and Cognitive Architectures , 2015, Neuron.

[40]  K. Clayton,et al.  Studies of Mental “Noise” , 1997 .

[41]  Lawrence K. Cormack,et al.  Reflexive and voluntary control of smooth eye movements , 2013, Electronic Imaging.

[42]  Floris P de Lange,et al.  Serial Dependence in Perceptual Decisions Is Reflected in Activity Patterns in Primary Visual Cortex , 2016, The Journal of Neuroscience.

[43]  R C Miall,et al.  System Identification Applied to a Visuomotor Task: Near-Optimal Human Performance in a Noisy Changing Task , 2003, The Journal of Neuroscience.

[44]  Justin M. Ales,et al.  The steady-state visual evoked potential in vision research: A review. , 2015, Journal of vision.

[45]  Wei Ji Ma,et al.  Bayesian microsaccade detection , 2017, Journal of vision.

[46]  M. Ernst,et al.  The statistical determinants of adaptation rate in human reaching. , 2008, Journal of vision.

[47]  H. W. Sorenson,et al.  Kalman Filtering Techniques , 1966 .

[48]  Mark F. Bear,et al.  Learned spatiotemporal sequence recognition and prediction in primary visual cortex , 2014, Nature Neuroscience.

[49]  T A Carpenter,et al.  Colored noise and computational inference in neurophysiological (fMRI) time series analysis: Resampling methods in time and wavelet domains , 2001, Human brain mapping.

[50]  Bing Liu,et al.  Efficient sensory cortical coding optimizes pursuit eye movements , 2016, Nature Communications.

[51]  Andrew M. Clark,et al.  Stimulus onset quenches neural variability: a widespread cortical phenomenon , 2010, Nature Neuroscience.

[52]  Biyu J. He,et al.  Electrophysiological correlates of the brain's intrinsic large-scale functional architecture , 2008, Proceedings of the National Academy of Sciences.

[53]  D. Ballard,et al.  Eye movements in natural behavior , 2005, Trends in Cognitive Sciences.

[54]  D. Heeger,et al.  Reliability of cortical activity during natural stimulation , 2010, Trends in Cognitive Sciences.

[55]  Michael J. Berry,et al.  Predictive information in a sensory population , 2013, Proceedings of the National Academy of Sciences.

[56]  S. Dehaene,et al.  Neural signature of the conscious processing of auditory regularities , 2009, Proceedings of the National Academy of Sciences.

[57]  Biyu J. He,et al.  The Temporal Structures and Functional Significance of Scale-free Brain Activity , 2010, Neuron.

[58]  Marie-Laure Bocca,et al.  The saccadic component of ocular pursuit is influenced by the predictability of the target motion in humans , 2005, Experimental Brain Research.

[59]  Sarah M Cormiea,et al.  Winter is coming: How humans forage in a temporally structured environment. , 2014, Journal of vision.

[60]  Dieter Jaeger,et al.  Infraslow LFP correlates to resting-state fMRI BOLD signals , 2013, NeuroImage.

[61]  H. Seo,et al.  Lateral Intraparietal Cortex and Reinforcement Learning during a Mixed-Strategy Game , 2009, Journal of Neuroscience.

[62]  M. Shadlen,et al.  The effect of stimulus strength on the speed and accuracy of a perceptual decision. , 2005, Journal of vision.

[63]  R. Ratcliff A diffusion model account of response time and accuracy in a brightness discrimination task: Fitting real data and failing to fit fake but plausible data , 2002, Psychonomic bulletin & review.

[64]  J. Atick,et al.  STATISTICS OF NATURAL TIME-VARYING IMAGES , 1995 .

[65]  R. Voss,et al.  ‘1/fnoise’ in music and speech , 1975, Nature.

[66]  Lawrence K. Cormack,et al.  The cost of time in multi-object tracking tasks. , 2017 .

[67]  Ralf D. Wimmer,et al.  Thalamic amplification of cortical connectivity sustains attentional control , 2017, Nature.

[68]  J. Henderson Human gaze control during real-world scene perception , 2003, Trends in Cognitive Sciences.

[69]  William P. Banks,et al.  Signal detection theory and human memory. , 1970 .

[70]  F. D. Lange,et al.  Opposite Effects of Recent History on Perception and Decision , 2017, Current Biology.

[71]  Biyu J He,et al.  Spatiotemporal Dissociation of Brain Activity Underlying Subjective Awareness, Objective Performance and Confidence , 2014, The Journal of Neuroscience.

[72]  Jochen Ditterich,et al.  Stochastic models of decisions about motion direction: Behavior and physiology , 2006, Neural Networks.

[73]  H Stanislaw,et al.  Calculation of signal detection theory measures , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[74]  David M. Santucci,et al.  Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates , 2003, PLoS biology.

[75]  J. Palva,et al.  Very Slow EEG Fluctuations Predict the Dynamics of Stimulus Detection and Oscillation Amplitudes in Humans , 2008, The Journal of Neuroscience.

[76]  Peter Janssen,et al.  Visual guidance in control of grasping. , 2015, Annual review of neuroscience.

[77]  Jeffrey K. Uhlmann,et al.  New extension of the Kalman filter to nonlinear systems , 1997, Defense, Security, and Sensing.

[78]  Biyu J. He Scale-free brain activity: past, present, and future , 2014, Trends in Cognitive Sciences.

[79]  Samuel Gershman,et al.  A Unifying Probabilistic View of Associative Learning , 2015, PLoS Comput. Biol..

[80]  Biyu J. He Scale-Free Properties of the Functional Magnetic Resonance Imaging Signal during Rest and Task , 2011, The Journal of Neuroscience.

[81]  John A. Swets,et al.  Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers , 1996 .

[82]  Justin L. Vincent,et al.  Intrinsic Fluctuations within Cortical Systems Account for Intertrial Variability in Human Behavior , 2007, Neuron.

[83]  D. Botteldooren,et al.  1/f Noise in Rural and Urban Soundscapes , 2003 .

[84]  C. Gaál [About hierarchy]. , 2003, Orvosi Hetilap.

[85]  D. Whitney,et al.  Serial dependence in visual perception , 2011 .

[86]  Roger Ratcliff,et al.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks , 2008, Neural Computation.

[87]  Fenna M. Krienen,et al.  Opportunities and limitations of intrinsic functional connectivity MRI , 2013, Nature Neuroscience.

[88]  Don H. Johnson,et al.  Statistical Signal Processing , 2009, Encyclopedia of Biometrics.

[89]  D. Whitney,et al.  Serial Dependence in the Perception of Faces , 2014, Current Biology.

[90]  C. Cierpka,et al.  Particle imaging techniques for volumetric three-component (3D3C) velocity measurements in microfluidics , 2011, Journal of Visualization.

[91]  M. Boly,et al.  Baseline brain activity fluctuations predict somatosensory perception in humans , 2007, Proceedings of the National Academy of Sciences.

[92]  M. Shadlen,et al.  A role for neural integrators in perceptual decision making. , 2003, Cerebral cortex.

[93]  Jeremy R. Manning,et al.  Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans , 2009, The Journal of Neuroscience.

[94]  Jeffrey N. Rouder,et al.  A diffusion model account of masking in two-choice letter identification. , 2000, Journal of experimental psychology. Human perception and performance.

[95]  Eero P. Simoncelli,et al.  Partitioning neuronal variability , 2014, Nature Neuroscience.

[96]  Johannes Burge,et al.  Continuous psychophysics: Target-tracking to measure visual sensitivity. , 2015, Journal of vision.

[97]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[98]  Patrice Abry,et al.  Neural Integration of Stimulus History Underlies Prediction for Naturalistically Evolving Sequences , 2018, The Journal of Neuroscience.

[99]  J. Gordon,et al.  Thalamic projections sustain prefrontal activity during working memory maintenance , 2017, Nature Neuroscience.

[100]  Philip L. Smith,et al.  Psychology and neurobiology of simple decisions , 2004, Trends in Neurosciences.

[101]  C. Koch,et al.  Neuronal Shot Noise and Brownian 1/f2 Behavior in the Local Field Potential , 2008, PloS one.

[102]  E. Miller,et al.  Gamma and Beta Bursts Underlie Working Memory , 2016, Neuron.

[103]  Zoubin Ghahramani,et al.  Computational principles of movement neuroscience , 2000, Nature Neuroscience.

[104]  Phillipp Meister,et al.  Statistical Signal Processing Detection Estimation And Time Series Analysis , 2016 .

[105]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[106]  Stephen G Lisberger,et al.  Visual Guidance of Smooth Pursuit Eye Movements. , 2015, Annual review of vision science.

[107]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[108]  Jeffrey G. Ojemann,et al.  Power-Law Scaling in the Brain Surface Electric Potential , 2009, PLoS Comput. Biol..