What should I do next? Using shared representations to solve interaction problems

Studies on how “the social mind” works reveal that cognitive agents engaged in joint actions actively estimate and influence another’s cognitive variables and form shared representations with them. (How) do shared representations enhance coordination? In this paper, we provide a probabilistic model of joint action that emphasizes how shared representations help solving interaction problems. We focus on two aspects of the model. First, we discuss how shared representations permit to coordinate at the level of cognitive variables (beliefs, intentions, and actions) and determine a coherent unfolding of action execution and predictive processes in the brains of two agents. Second, we discuss the importance of signaling actions as part of a strategy for sharing representations and the active guidance of another’s actions toward the achievement of a joint goal. Furthermore, we present data from a human-computer experiment (the Tower Game) in which two agents (human and computer) have to build together a tower made of colored blocks, but only the human knows the constellation of the tower to be built (e.g., red-blue-red-blue-$$\ldots$$). We report evidence that humans use signaling strategies that take another’s uncertainty into consideration, and that in turn our model is able to use humans’ actions as cues to “align” its representations and to select complementary actions.

[1]  Cynthia Breazeal,et al.  Cost-Based Anticipatory Action Selection for Human–Robot Fluency , 2007, IEEE Transactions on Robotics.

[2]  S. Levinson,et al.  Brain Mechanisms Underlying Human Communication , 2009, Front. Hum. Neurosci..

[3]  H. Bekkering,et al.  The mirror neuron system is more active during complementary compared with imitative action , 2007, Nature Neuroscience.

[4]  Bruno Galantucci,et al.  An Experimental Study of the Emergence of Human Communication Systems , 2005, Cogn. Sci..

[5]  Natalie Sebanz,et al.  Prediction in Joint Action: What, When, and Where , 2009, Top. Cogn. Sci..

[6]  Afdc Hamilton,et al.  The motor hierarchy: from kinematics to goals and intentions , 2007 .

[7]  C. Frith,et al.  Follow you, Follow me: Continuous Mutual Prediction and Adaptation in Joint Tapping , 2010, Quarterly journal of experimental psychology.

[8]  P. Haggard,et al.  Sensorimotor foundations of higher cognition , 1993 .

[9]  Louise K. Comfort,et al.  Goals and Means , 1981 .

[10]  Stuart J. Russell,et al.  Dynamic bayesian networks: representation, inference and learning , 2002 .

[11]  R. Conte,et al.  Cognitive and social action , 1995 .

[12]  Michael E. Bratman,et al.  Shared Cooperative Activity , 1991 .

[13]  W. Prinz Perception and Action Planning , 1997 .

[14]  W. Prinz,et al.  Relationships Between Perception and Action: Current Approaches , 1990 .

[15]  Karl J. Friston Hierarchical Models in the Brain , 2008, PLoS Comput. Biol..

[16]  M. Botvinick,et al.  Conflict monitoring and cognitive control. , 2001, Psychological review.

[17]  Raymond J. Dolan,et al.  Game Theory of Mind , 2008, PLoS Comput. Biol..

[18]  Chris L. Baker,et al.  Action understanding as inverse planning , 2009, Cognition.

[19]  R. Johansson,et al.  Prediction Precedes Control in Motor Learning , 2003, Current Biology.

[20]  Ronald A. Howard,et al.  Information Value Theory , 1966, IEEE Trans. Syst. Sci. Cybern..

[21]  B. Grosz Collaborative Systems , 1996 .

[22]  Estela Bicho,et al.  Neuro-cognitive mechanisms of decision making in joint action: a human-robot interaction study. , 2011, Human movement science.

[23]  Scott T. Grafton,et al.  Forward modeling allows feedback control for fast reaching movements , 2000, Trends in Cognitive Sciences.

[24]  G. Pezzulo,et al.  Thinking as the control of imagination: a conceptual framework for goal-directed systems , 2009, Psychological research.

[25]  W. Prinz A common-coding approach to perception and action , 1990 .

[26]  G. Knoblich,et al.  The case for motor involvement in perceiving conspecifics. , 2005, Psychological bulletin.

[27]  Philip R. Cohen,et al.  Intentions in Communication , 1992, Language.

[28]  E. Pacherie The phenomenology of action: A conceptual framework , 2008, Cognition.

[29]  H. Bekkering,et al.  Joint action: bodies and minds moving together , 2006, Trends in Cognitive Sciences.

[30]  U. Castiello,et al.  Different action patterns for cooperative and competitive behaviour , 2007, Cognition.

[31]  Rick Grush,et al.  The emulation theory of representation: Motor control, imagery, and perception , 2004, Behavioral and Brain Sciences.

[32]  Roel M. Willems,et al.  Complementary Systems for Understanding Action Intentions , 2008, Current Biology.

[33]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[34]  K. Doya,et al.  A unifying computational framework for motor control and social interaction. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[35]  S. Levinson On the Human "Interaction Engine" , 2020, Roots of Human Sociality.

[36]  G. Csibra,et al.  'Obsessed with goals': functions and mechanisms of teleological interpretation of actions in humans. , 2007, Acta psychologica.

[37]  G. Csibra,et al.  Teleological reasoning in infancy: the naı̈ve theory of rational action , 2003, Trends in Cognitive Sciences.

[38]  H. H. Clark,et al.  Speaking while monitoring addressees for understanding , 2004 .

[39]  M. Jeannerod Neural Simulation of Action: A Unifying Mechanism for Motor Cognition , 2001, NeuroImage.

[40]  A. Borst Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[41]  Csr Young,et al.  How to Do Things With Words , 2009 .

[42]  Uta Frith,et al.  Implicit and Explicit Processes in Social Cognition , 2008, Neuron.

[43]  D. Kirsh Distributed Cognition , Coordination and Environment Design , 1999 .

[44]  G. Pezzulo Grounding Procedural and Declarative Knowledge in Sensorimotor Anticipation , 2011 .

[45]  R. Ellis,et al.  Action priming by briefly presented objects. , 2004, Acta psychologica.

[46]  Guy Theraulaz,et al.  Self-Organization in Biological Systems , 2001, Princeton studies in complexity.

[47]  Robrecht P R D van der Wel,et al.  Let the force be with us: dyads exploit haptic coupling for coordination. , 2011, Journal of experimental psychology. Human perception and performance.

[48]  Yiannis Demiris,et al.  Hierarchical attentive multiple models for execution and recognition of actions , 2006, Robotics Auton. Syst..

[49]  Giovanni Pezzulo,et al.  Coordinating with the Future: The Anticipatory Nature of Representation , 2008, Minds and Machines.

[50]  D. Wolpert,et al.  Motor prediction , 2001, Current Biology.

[51]  M. Pickering,et al.  Why is conversation so easy? , 2004, Trends in Cognitive Sciences.

[52]  Philip R. Cohen,et al.  Collective Intentions and Actions , 2003 .

[53]  J. Searle Consciousness and Language: Collective Intentions and Actions , 2002 .

[54]  R. Johnstone,et al.  Animal signals , 2013, Current Biology.

[55]  Simon Garrod,et al.  Joint Action, Interactive Alignment, and Dialog , 2009, Top. Cogn. Sci..

[56]  Siobhan Chapman Logic and Conversation , 2005 .

[57]  Ran R. Hassin,et al.  Goal contagion: perceiving is for pursuing. , 2004, Journal of personality and social psychology.

[58]  Karl J. Friston,et al.  Predictive coding: an account of the mirror neuron system , 2007, Cognitive Processing.

[59]  C. Frith,et al.  How we predict what other people are going to do , 2006, Brain Research.

[60]  W. James,et al.  The Principles of Psychology. , 1983 .

[61]  Cordula Vesper,et al.  A minimal architecture for joint action , 2010, Neural Networks.

[62]  N. Sebanz,et al.  Evolving intentions for social interaction: from entrainment to joint action , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[63]  Barbara J. Grosz,et al.  Collaborative Systems (AAAI-94 Presidential Address) , 1996 .

[64]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[65]  Michael Bacharach,et al.  Beyond Individual Choice: Teams and Frames in Game Theory , 2006 .

[66]  M. Botvinick Hierarchical models of behavior and prefrontal function , 2008, Trends in Cognitive Sciences.

[67]  M. Tomasello,et al.  Understanding and sharing intentions: The origins of cultural cognition , 2005, Behavioral and Brain Sciences.

[68]  Ruud G. J. Meulenbroek,et al.  Anatomical substrates of cooperative joint-action in a continuous motor task: Virtual lifting and balancing , 2008, NeuroImage.

[69]  Giovanni Pezzulo,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Motor Simulation via Coupled Internal Models Using Sequential Monte Carlo , 2022 .

[70]  Raymond H. Cuijpers,et al.  Goals and means in action observation: A computational approach , 2006, Neural Networks.