Effects of reward expectancy on sequential eye movements in monkeys

Desirability of an action, often referred to as utility or value, is determined by various factors, such as the probability and timing of expected reward. We investigated how performance of monkeys in an oculomotor serial reaction time task is influenced by multiple motivational factors. The animals produced a series of visually-guided eye movements, while the sequence of target locations and the location of the rewarded target were systematically manipulated. The results show that error rates as well as saccade latencies were consistently influenced by the number of remaining movements necessary to obtain a reward. In addition, when the animal produced multiple saccades before fixating a given target, the first saccade tended to be directed towards the rewarded location, suggesting that saccades to rewarded location and visual target might be programmed concurrently. These results show that monkeys can utilize information about the required sequence of movements to update their subjective values.

[1]  B. Richmond,et al.  Anterior Cingulate: Single Neuronal Signals Related to Degree of Reward Expectancy , 2002, Science.

[2]  J. Joseph,et al.  Prefrontal cortex and spatial sequencing in macaque monkey , 2004, Experimental Brain Research.

[3]  M. Nissen,et al.  Attentional requirements of learning: Evidence from performance measures , 1987, Cognitive Psychology.

[4]  Jun Tanji,et al.  Participation of the primate presupplementary motor area in sequencing multiple saccades. , 2004, Journal of neurophysiology.

[5]  A. Georgopoulos,et al.  Neural activity in prefrontal cortex during copying geometrical shapes , 2003, Experimental brain research.

[6]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[7]  J. Tanji,et al.  Neuronal activity in the supplementary and presupplementary motor areas for temporal organization of multiple movements. , 2000, Journal of neurophysiology.

[8]  J. Davenport,et al.  The interaction of magnitude and delay of reinforcement in spatial discrimination. , 1962, Journal of comparative and physiological psychology.

[9]  Thomas P. Ryan,et al.  Modern Regression Methods , 1996 .

[10]  J. B. Wolfe.  The effect of delayed reward upon learning in the white rat. , 1934 .

[11]  E. J. Anastasio,et al.  Effects of constant delay of reinforcement on acquisition asymptote and resistance to extinction. , 1967, Journal of experimental psychology.

[12]  E. Procyk,et al.  Characterization of serial order encoding in the monkey anterior cingulate sulcus , 2001, The European journal of neuroscience.

[13]  J. Ashe,et al.  Cerebellum Activation Associated with Performance Change but Not Motor Learning , 2002, Science.

[14]  J. Kalaska,et al.  Neural Correlates of Reaching Decisions in Dorsal Premotor Cortex: Specification of Multiple Direction Choices and Final Selection of Action , 2005, Neuron.

[15]  F A LOGAN,et al.  The role of delay of reinforcement in determining reaction potential. , 1952, Journal of experimental psychology.

[16]  R. Herrnstein,et al.  Choice and delay of reinforcement. , 1967, Journal of the experimental analysis of behavior.

[17]  L. Crespi.  Quantitative variation of incentive and performance in the white rat. , 1942 .

[18]  K. Edward Renner,et al.  Influence of deprivation and availability of goal box cues on the temporal gradient of reinforcement. , 1963 .

[19]  F A LOGAN,et al.  DECISION MAKING BY RATS: DELAY VERSUS AMOUNT OF REWARD. , 1965, Journal of comparative and physiological psychology.

[20]  Donald W. Marquaridt.  Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation , 1970 .

[21]  Daeyeol Lee,et al.  Activity in the supplementary motor area related to learning and performance during a sequential visuomotor task. , 2003, Journal of neurophysiology.

[22]  Scott T. Grafton,et al.  Functional Mapping of Sequence Learning in Normal Humans , 1995, Journal of Cognitive Neuroscience.

[23]  W. Newsome,et al.  Matching Behavior and the Representation of Value in the Parietal Cortex , 2004, Science.

[24]  O. Hikosaka,et al.  Modulation of saccadic eye movements by predicted reward outcome , 2001, Experimental Brain Research.

[25]  G. Pellizzer,et al.  Motor planning: effect of directional uncertainty with discrete spatial cues , 2003, Experimental Brain Research.

[26]  Daeyeol Lee,et al.  Activity in prefrontal cortex during dynamic selection of action sequences , 2006, Nature Neuroscience.

[27]  J. Tanji,et al.  Oculomotor sequence learning: a positron emission tomography study , 1998, Experimental Brain Research.

[28]  E. Keller,et al.  Short-term priming, concurrent processing, and saccade curvature during a target selection task in the monkey , 2001, Vision Research.

[29]  Michael L. Platt,et al.  Neural correlates of decision variables in parietal cortex , 1999, Nature.

[30]  D. Zeaman,et al.  Response latency as a function of the amount of reinforcement. , 1949, Journal of experimental psychology.

[31]  Frank A. Logan,et al.  Incentive: How the Conditions of Reinforcement Affect the Performance of Rats , 1960 .

[32]  S. Rauch,et al.  Striatal recruitment during an implicit sequence learning task as measured by functional magnetic resonance imaging , 1997, Human brain mapping.

[33]  Daeyeol Lee,et al.  Behavioral Context and Coherent Oscillations in the Supplementary Motor Area , 2022 .

[34]  O. Hikosaka,et al.  Expectation of reward modulates cognitive signals in the basal ganglia , 1998, Nature Neuroscience.

[35]  Scott T. Grafton,et al.  Attention and stimulus characteristics determine the locus of motor-sequence encoding. A PET study. , 1997, Brain : a journal of neurology.

[36]  S. Kosslyn,et al.  A PET investigation of implicit and explicit sequence learning , 1995 .

[37]  J. Tanji,et al.  Contrasting neuronal activity in the supplementary and frontal eye fields during temporal organization of multiple saccades. , 2003, Journal of neurophysiology.

[38]  E. Procyk,et al.  The effects of sequence structure and reward schedule on serial reaction time learning in the monkey. , 2000, Brain research. Cognitive brain research.

[39]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[40]  E. Thorndike.  Animal Intelligence; Experimental Studies , 2009 .

[41]  B. Skinner,et al.  Principles of Behavior , 1944 .

[42]  M. Hallett,et al.  Modulation of cortical motor output maps during development of implicit and explicit knowledge. , 1994, Science.

[43]  H. Simon,et al.  Rational choice and the structure of the environment. , 1956, Psychological review.

[44]  J. Joseph,et al.  Activity in the caudate nucleus of monkey during spatial sequencing. , 1995, Journal of neurophysiology.

[45]  C. T. Perin.  A quantitative investigation of the delay-of-reinforcement gradient. , 1943 .

[46]  M. Shadlen,et al.  Effect of Expected Reward Magnitude on the Response of Neurons in the Dorsolateral Prefrontal Cortex of the Macaque , 1999, Neuron.

[47]  B. Richmond,et al.  Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. , 1996, Journal of neurophysiology.

[48]  D Lee,et al.  Effects of exogenous and endogenous attention on visually guided hand movements. , 1999, Brain research. Cognitive brain research.

[49]  G S HARKER,et al.  Delay of reward and performance of an instrumental response. , 1956, Journal of experimental psychology.

[50]  D. Barraclough,et al.  Prefrontal cortex and decision making in a mixed-strategy game , 2004, Nature Neuroscience.

[51]  C. B. Ferster,et al.  Schedules of reinforcement , 1957 .

[52]  M. Bonem,et al.  Elucidating the effects of reinforcement magnitude. , 1988, Psychological bulletin.

[53]  C. Stern,et al.  An fMRI Study of the Role of the Medial Temporal Lobe in Implicit and Explicit Sequence Learning , 2003, Neuron.

[54]  G. E. Alexander,et al.  Movement sequence-related activity reflecting numerical order of components in supplementary and presupplementary motor areas. , 1998, Journal of neurophysiology.

[55]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[56]  Wolfram Schultz,et al.  Behavioral reactions reflecting differential reward expectations in monkeys , 2001, Experimental Brain Research.

[57]  Katsuyuki Sakai,et al.  Learning of sequences of finger movements and timing: frontal lobe and action-oriented representation. , 2002, Journal of neurophysiology.

[58]  J. Gabrieli,et al.  Direct comparison of neural systems mediating conscious and unconscious skill learning. , 2002, Journal of neurophysiology.

[59]  W. Becker,et al.  An analysis of the saccadic system by means of double step stimuli , 1979, Vision Research.

[60]  Leslie G. Ungerleider,et al.  Experience-dependent changes in cerebellar contributions to motor sequence learning , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[61]  M. Roesch,et al.  Impact of expected reward on neuronal activity in prefrontal cortex, frontal and supplementary eye fields and premotor cortex. , 2003, Journal of neurophysiology.

[62]  D. Barraclough,et al.  Reinforcement learning and decision making in monkeys during a competitive game. , 2004, Brain research. Cognitive brain research.

[63]  G. Berns,et al.  Brain regions responsive to novelty in the absence of awareness. , 1997, Science.