Track It to Crack It: Dissecting Processing Stages with Finger Tracking

A central goal in cognitive science is to parse the series of processing stages underlying a cognitive task. A powerful yet simple behavioral method that can resolve this problem is finger trajectory tracking: by continuously tracking the finger position and speed as a participant chooses a response, and by analyzing which stimulus features affect the trajectory at each time point during the trial, we can estimate the absolute timing and order of each processing stage, and detect transient effects, changes of mind, serial versus parallel processing, and real-time fluctuations in subjective confidence. We suggest that trajectory tracking, which provides considerably more information than mere response times, may provide a comprehensive understanding of the fast temporal dynamics of cognitive operations.

[1]  Jonas M. B. Haslbeck,et al.  Mouse-tracking: A practical guide to implementation and analysis , 2018 .

[2]  M. Shadlen,et al.  Choice Certainty Is Informed by Both Evidence and Decision Time , 2014, Neuron.

[3]  S. Dehaene,et al.  Acquisition and processing of an artificial mini-language combining semantic and syntactic elements , 2019, Cognition.

[4]  Jason Friedman,et al.  Linking cognitive and reaching trajectories via intermittent movement control , 2013 .

[5]  J. Freeman,et al.  Advanced mouse-tracking analytic techniques for enhancing psychological science , 2015 .

[6]  Alfonso Caramazza,et al.  Engaging the motor system with masked orthographic primes: A kinematic analysis , 2008 .

[7]  H. Bekkering,et al.  Gaze anchoring to a pointing target is present during the entire pointing movement and is driven by a non-visual signal. , 2001, Journal of neurophysiology.

[8]  D. Wolpert,et al.  Changing your mind: a computational mechanism of vacillation , 2009, Nature.

[9]  Jonathan B Freeman,et al.  A Perceptual Pathway to Bias , 2016, Psychological science.

[10]  R. Eisner A Distributed Lag Investment Function , 1960 .

[11]  Tyler Marghetis,et al.  The Quarterly Journal of Experimental Psychology Doing Arithmetic by Hand: Hand Movements during Exact Arithmetic Reveal Systematic, Dynamic Spatial Processing , 2022 .

[12]  J. Houk,et al.  Deciding when and how to correct a movement: discrete submovements as a decision making process , 2007, Experimental Brain Research.

[13]  Stephen H Scott,et al.  Influence of the behavioral goal and environmental obstacles on rapid feedback responses. , 2012, Journal of neurophysiology.

[14]  M. Shadlen,et al.  Representation of Confidence Associated with a Decision by Neurons in the Parietal Cortex , 2009, Science.

[15]  Craig S. Chapman,et al.  One to Four, and Nothing More , 2011, Psychological science.

[16]  Ken Nakayama,et al.  Numeric comparison in a visually-guided manual reaching task , 2008, Cognition.

[17]  Michael J. Spivey,et al.  Tracking the Continuity of Language Comprehension: Computer Mouse Trajectories Suggest Parallel Syntactic Processing , 2007, Cogn. Sci..

[18]  Nathan F. Lepora,et al.  Embodied Choice: How Action Influences Perceptual Decision Making , 2015, PLoS Comput. Biol..

[19]  Craig S. Chapman,et al.  Decision-making in sensorimotor control , 2018, Nature Reviews Neuroscience.

[20]  N. Ambady,et al.  Motions of the Hand Expose the Partial and Parallel Activation of Stereotypes , 2009, Psychological science.

[21]  F. Donders On the speed of mental processes. , 1969, Acta psychologica.

[22]  Pascal J. Kieslich,et al.  Mouse-tracking: Detecting Types in Movement Trajectories , 2018 .

[23]  C. Prablanc,et al.  Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement , 1986, Nature.

[24]  Children's Online Processing of Complex Sentences: New Evidence from a New Technique , 2007 .

[25]  T. Goschke,et al.  How decisions emerge: action dynamics in intertemporal decision making. , 2013, Journal of experimental psychology. General.

[26]  Tom Verguts,et al.  Distance in Motion: Response Trajectories Reveal the Dynamics of Number Comparison , 2011, PloS one.

[27]  Michael J. Spivey,et al.  Continuous attraction toward phonological competitors. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[28]  R A Abrams,et al.  Optimality in human motor performance: ideal control of rapid aimed movements. , 1988, Psychological review.

[29]  M. Goodale How (and why) the visual control of action differs from visual perception , 2014, Proceedings of the Royal Society B: Biological Sciences.

[30]  F. Hermens When do arrows start to compete? A developmental mouse-tracking study. , 2018, Acta psychologica.

[31]  G. Miller,et al.  Plans and the structure of behavior , 1960 .

[32]  H Pashler,et al.  Processing stages in overlapping tasks: evidence for a central bottleneck. , 1984, Journal of experimental psychology. Human perception and performance.

[33]  Stanislas Dehaene,et al.  Moving along the Number Line: Operational Momentum in Nonsymbolic Arithmetic , 2006 .

[34]  Ariel Zylberberg,et al.  Pupil Dilation: A Fingerprint of Temporal Selection During the “Attentional Blink” , 2012, Front. Psychology.

[35]  Stanislas Dehaene,et al.  How do we convert a number into a finger trajectory? , 2013, Cognition.

[36]  M. Land,et al.  The Roles of Vision and Eye Movements in the Control of Activities of Daily Living , 1998, Perception.

[37]  Rick Dale,et al.  The Cognitive Dynamics of Negated Sentence Verification , 2011, Cogn. Sci..

[38]  Rick Dale,et al.  Assessing bimodality to detect the presence of a dual cognitive process , 2013, Behavior research methods.

[39]  Stanislas Dehaene,et al.  Dynamic representations underlying symbolic and nonsymbolic calculation: Evidence from the operational momentum effect , 2009, Attention, perception & psychophysics.

[40]  Christopher D. Erb,et al.  Cognitive control in action: Tracking the dynamics of rule switching in 5- to 8-year-olds and adults , 2017, Cognition.

[41]  T. Goschke,et al.  How decisions evolve: The temporal dynamics of action selection , 2010, Cognition.

[42]  P. Fitts The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[43]  Joo-Hyun Song,et al.  Dynamic modulation of illusory and physical target size on separate and coordinated eye and hand movements , 2017, Journal of vision.

[44]  T. Flash,et al.  The coordination of arm movements: an experimentally confirmed mathematical model , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[45]  Thomas J. Faulkenberry,et al.  Response trajectories capture the continuous dynamics of the size congruity effect. , 2016, Acta psychologica.

[46]  Jeff Moher,et al.  Reach tracking reveals dissociable processes underlying cognitive control , 2016, Cognition.

[47]  C. Ghez,et al.  Discrete and continuous planning of hand movements and isometric force trajectories , 1997, Experimental Brain Research.

[48]  James L. McClelland On the time relations of mental processes: An examination of systems of processes in cascade. , 1979 .

[49]  Michael S Landy,et al.  Motor control is decision-making , 2012, Current Opinion in Neurobiology.

[50]  Michael J. Spivey,et al.  Action Dynamics Reveal Parallel Competition in Decision Making , 2008, Psychological science.

[51]  A Zgonnikov,et al.  Decision landscapes: visualizing mouse-tracking data , 2017, Royal Society Open Science.

[52]  D. Wolpert,et al.  A common mechanism underlies changes of mind about decisions and confidence , 2015, eLife.

[53]  Cendri A. C. Hutcherson,et al.  Dietary Self-Control Is Related to the Speed With Which Attributes of Healthfulness and Tastiness Are Processed , 2015, Psychological science.

[54]  Michael J. Spivey,et al.  Continuous Dynamics in Real-Time Cognition , 2006 .

[55]  Christopher D. Erb,et al.  The developing mind in action: measuring manual dynamics in childhood , 2018 .

[56]  M. Jeannerod,et al.  Optimal response of eye and hand motor systems in pointing at a visual target , 1979, Biological Cybernetics.

[57]  Jonathan B Freeman,et al.  MouseTracker: Software for studying real-time mental processing using a computer mouse-tracking method , 2010, Behavior research methods.

[58]  Stanislas Dehaene,et al.  Finger Tracking Reveals the Covert Stages of Mental Arithmetic , 2017, Open Mind.

[59]  Florent Meyniel,et al.  On-line confidence monitoring during decision making , 2018, Cognition.

[60]  Jeff Moher,et al.  A comparison of simple movement behaviors across three different devices , 2019, Attention, Perception, & Psychophysics.

[61]  Daniel C. Richardson,et al.  The movement of eye and hand as a window into language and cognition , 2009 .

[62]  Michael J. Spivey,et al.  Graded motor responses in the time course of categorizing atypical exemplars , 2007, Memory & cognition.

[63]  Jonathan B. Freeman,et al.  Doing Psychological Science by Hand , 2018, Current directions in psychological science.

[64]  Martin H. Fischer,et al.  Pushing forward in embodied cognition: may we mouse the mathematical mind? , 2014, Front. Psychol..

[65]  John M. Tomlinson,et al.  Possibly All of that and Then Some: Scalar Implicatures Are Understood in Two Steps. , 2013 .

[66]  Melvyn A. Goodale,et al.  Grasping without vision: Time normalizing grip aperture profiles yields spurious grip scaling to target size , 2013, Neuropsychologia.

[67]  M. Landy,et al.  Statistical decision theory and trade-offs in the control of motor response. , 2003, Spatial vision.

[68]  Shirley Almon The Distributed Lag Between Capital Appropriations and Expenditures , 1965 .

[69]  M. Sigman,et al.  Parsing a Cognitive Task: A Characterization of the Mind's Bottleneck , 2005, PLoS biology.

[70]  Michael N. Shadlen,et al.  Probabilistic reasoning by neurons , 2007, Nature.

[71]  Michael I. Jordan,et al.  Optimal feedback control as a theory of motor coordination , 2002, Nature Neuroscience.

[72]  Jason P. Gallivan,et al.  Three-dimensional reach trajectories as a probe of real-time decision-making between multiple competing targets , 2014, Front. Neurosci..

[73]  T. Verguts,et al.  Losing the boundary: Cognition biases action well after action selection. , 2015, Journal of experimental psychology. General.

[74]  Florent Meyniel,et al.  The Sense of Confidence during Probabilistic Learning: A Normative Account , 2015, PLoS Comput. Biol..

[75]  Catherine M. Arrington,et al.  Tracking the Multitasking Mind , 2013 .

[76]  Jeff Moher,et al.  Context-dependent sequential effects of target selection for action. , 2013, Journal of vision.

[77]  K. Nakayama,et al.  Hidden cognitive states revealed in choice reaching tasks , 2009, Trends in Cognitive Sciences.

[78]  S. Sirois,et al.  Pupillometry , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[79]  Konrad Paul Kording,et al.  Bayesian integration in sensorimotor learning , 2004, Nature.

[80]  T. Verguts,et al.  Continuous track paths reveal additive evidence integration in multistep decision making , 2017, Proceedings of the National Academy of Sciences.

[81]  Rico Fischer,et al.  Action dynamics in multitasking: the impact of additional task factors on the execution of the prioritized motor movement , 2015, Front. Psychol..

[82]  N. Berthier Learning to reach: A mathematical model. , 1996 .

[83]  Naama Friedmann,et al.  Breaking down number syntax: Spared comprehension of multi-digit numbers in a patient with impaired digit-to-word conversion , 2014, Cortex.

[84]  Michael J. Spivey,et al.  On the temporal dynamics of language-mediated vision and vision-mediated language. , 2011, Acta psychologica.

[85]  Takashi Yamauchi,et al.  The role of attention in subliminal semantic processing: A mouse tracking study , 2017, PloS one.

[86]  Stanislas Dehaene,et al.  On the origins of logarithmic number-to-position mapping. , 2016, Psychological review.

[87]  Stefan Scherbaum,et al.  Stuck at the starting line: How the starting procedure influences mouse-tracking data , 2017, Behavior Research Methods.

[88]  R. Johansson,et al.  Eye–Hand Coordination in Object Manipulation , 2001, The Journal of Neuroscience.

[89]  Craig S. Chapman,et al.  Reaching for the unknown: Multiple target encoding and real-time decision-making in a rapid reach task , 2010, Cognition.

[90]  Simon J. Thorpe,et al.  Ultra-rapid object detection with saccadic eye movements: Visual processing speed revisited , 2006, Vision Research.

[91]  Teenie Matlock,et al.  Grammatical aspect and temporal distance in motion descriptions , 2013, Front. Psychol..

[92]  Jeff Moher,et al.  Numerical Cognition in Action: Reaching Behavior Reveals Numerical Distance Effects in 5- to 6-Year-Olds , 2018, J. Numer. Cogn..

[93]  Stanislas Dehaene,et al.  The Organization of Brain Activations in Number Comparison: Event-Related Potentials and the Additive-Factors Method , 1996, Journal of Cognitive Neuroscience.

[94]  J. Enns,et al.  On-line control of pointing is modified by unseen visual shapes , 2007, Consciousness and Cognition.

[95]  J. Townsend Serial vs. Parallel Processing: Sometimes They Look like Tweedledum and Tweedledee but they can (and Should) be Distinguished , 1990 .

[96]  Jörn Diedrichsen,et al.  Perceptual decisions are biased by the cost to act , 2017, eLife.

[97]  Santiago Alonso-Diaz,et al.  A threshold-free model of numerosity comparisons , 2018, PloS one.

[98]  Saul Sternberg,et al.  The discovery of processing stages: Extensions of Donders' method , 1969 .

[99]  Paul Cisek,et al.  Cortical mechanisms of action selection: the affordance competition hypothesis , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[100]  Paul E. Stillman,et al.  How Mouse-tracking Can Advance Social Cognitive Theory , 2018, Trends in Cognitive Sciences.

[101]  S. Sirois,et al.  Pupillometry , 2012, Eye Movement Research.

[102]  Rick Dale,et al.  The Self-Organization of Explicit Attitudes , 2009, Psychological science.