Embodied Language Learning and Cognitive Bootstrapping: Methods and Design Principles

Co-development of action, conceptualization and social interaction mutually scaffold and support each other within a virtuous feedback cycle in the development of human language in children. Within this framework, the purpose of this article is to bring together diverse but complementary accounts of research methods that jointly contribute to our understanding of cognitive development and in particular, language acquisition in robots. Thus, we include research pertaining to developmental robotics, cognitive science, psychology, linguistics and neuroscience, as well as practical computer science and engineering. The different studies are not at this stage all connected into a cohesive whole; rather, they are presented to illuminate the need for multiple different approaches that complement each other in the pursuit of understanding cognitive development in robots. Extensive experiments involving the humanoid robot iCub are reported, while human learning relevant to developmental robotics has also contributed useful results. Disparate approaches are brought together via common underlying design principles. Without claiming to model human language acquisition directly, we are nonetheless inspired by analogous development in humans and consequently, our investigations include the parallel co-development of action, conceptualization and social interaction. Though these different approaches need to ultimately be integrated into a coherent, unified body of knowledge, progress is currently also being made by pursuing individual methods.

[1]  P. Kuhl Early language acquisition: cracking the speech code , 2004, Nature Reviews Neuroscience.

[2]  Emanuel Todorov,et al.  Optimal Control Theory , 2006 .

[3]  J. Werker,et al.  Learning words’ sounds before learning how words sound: 9-Month-olds use distinct objects as cues to categorize speech information , 2009, Cognition.

[4]  N. Hogan,et al.  Does the nervous system use equilibrium-point control to guide single and multiple joint movements? , 1992, The Behavioral and brain sciences.

[5]  Pietro G. Morasso,et al.  Towards Reasoning and Coordinating Action in the Mental Space , 2007, Int. J. Neural Syst..

[6]  L. Schillingmann,et al.  A Computational Model of Acoustic Packaging , 2009, IEEE Transactions on Autonomous Mental Development.

[7]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[8]  N. A. Bernshteĭn The co-ordination and regulation of movements , 1967 .

[9]  Angelo Cangelosi,et al.  Multiple Time Scales Recurrent Neural Network for Complex Action Acquisition , 2011 .

[10]  A. Glenberg Language and action: creating sensible combinations of ideas , 2007 .

[11]  Giulio Sandini,et al.  A biomimetic, force-field based computational model for motion planning and bimanual coordination in humanoid robots , 2009, Auton. Robots.

[12]  Katharina J. Rohlfing,et al.  Towards Learning by Interacting , 2009, Creating Brain-Like Intelligence.

[13]  Leila Takayama,et al.  Judging a bot by its cover: An experiment on expectation setting for personal robots , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[14]  Stevan Harnad The Symbol Grounding Problem , 1999, ArXiv.

[15]  Angelo Cangelosi,et al.  Epigenetic Robotics Architecture (ERA) , 2010, IEEE Transactions on Autonomous Mental Development.

[16]  Jun Tani,et al.  How Hierarchical Control Self-organizes in Artificial Adaptive Systems , 2005, Adapt. Behav..

[17]  H. H. Clark SPACE, TIME, SEMANTICS, AND THE CHILD , 1973 .

[18]  Amir Karniel,et al.  Open questions in computational motor control. , 2011, Journal of integrative neuroscience.

[19]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[20]  Kerstin Fischer,et al.  Getting Acquainted with a Developing Robot , 2012, HBU.

[21]  Steven Greenberg,et al.  Speaking in shorthand - A syllable-centric perspective for understanding pronunciation variation , 1999, Speech Commun..

[22]  W. Smith The Integrative Action of the Nervous System , 1907, Nature.

[23]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[24]  Karl J. Friston What Is Optimal about Motor Control? , 2011, Neuron.

[25]  G. Sandini,et al.  Motor Contagion during Human-Human and Human-Robot Interaction , 2014, PloS one.

[26]  Stefano Nolfi,et al.  Bottom-up learning of feedback in a categorization task , 2012, 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL).

[27]  M. Jeannerod Neural Simulation of Action: A Unifying Mechanism for Motor Cognition , 2001, NeuroImage.

[28]  Pietro G. Morasso,et al.  A Forward / Inverse Motor Controller for Cognitive Robotics , 2006, ICANN.

[29]  Pietro Morasso,et al.  From Cortical Maps to the Control of Muscles , 1997 .

[30]  R. Pea The development of negation in early child language , 1978 .

[31]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[32]  Linda B. Smith,et al.  Thinking With Your Body: Modelling Spatial Biases in Categorization Using a Real Humanoid Robot , 2010 .

[33]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[34]  S. Nolfi,et al.  Developing the Ability to Manipulate Objects : A Comparative Study with Human and Artificial Agents , 2010 .

[35]  Chrystopher L. Nehaniv,et al.  Robot acquisition of lexical meaning - moving towards the two-word stage , 2012, 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL).

[36]  J. Brobeck The Integrative Action of the Nervous System , 1948, The Yale Journal of Biology and Medicine.

[37]  Frank Broz,et al.  Interaction and experience in enactive intelligence and humanoid robotics , 2013, 2013 IEEE Symposium on Artificial Life (ALife).

[38]  C. Hofsten,et al.  Infants predict other people's action goals , 2006, Nature Neuroscience.

[39]  Emanuel Todorov,et al.  From task parameters to motor synergies: A hierarchical framework for approximately optimal control of redundant manipulators , 2005, J. Field Robotics.

[40]  L. Steels Self-organising vocabularies , 1996 .

[41]  Maura Casadio,et al.  A neural mechanism of synergy formation for whole body reaching , 2009, Biological Cybernetics.

[42]  David J. Chalmers,et al.  Syntactic Transformations on Distributed Representations , 1990 .

[43]  Chrystopher L. Nehaniv,et al.  Robots That Say 'No' , 2009, ECAL.

[44]  Morten H. Christiansen,et al.  Domain generality versus modality specificity: the paradox of statistical learning , 2015, Trends in Cognitive Sciences.

[45]  G. Sandini,et al.  Robots can be perceived as goal-oriented agents , 2013 .

[46]  Luc Steels,et al.  Language Grounding in Robots , 2012, Springer US.

[47]  F A Mussa-Ivaldi,et al.  Computations underlying the execution of movement: a biological perspective. , 1991, Science.

[48]  J. Decety Do imagined and executed actions share the same neural substrate? , 1996, Brain research. Cognitive brain research.

[49]  Pietro G. Morasso,et al.  Self-Organization, Computational Maps, and Motor Control , 1997 .

[50]  Kerstin Fischer,et al.  Mindful tutors: Linguistic choice and action demonstration in speech to infants and a simulated robot , 2011 .

[51]  Kerstin Fischer,et al.  Tutor Spotter: Proposing a Feature Set and Evaluating It in a Robotic System , 2011, International Journal of Social Robotics.

[52]  Marina Basu The Embodied Mind: Cognitive Science and Human Experience , 2004 .

[53]  R N Aslin,et al.  Statistical Learning by 8-Month-Old Infants , 1996, Science.

[54]  Pietro G. Morasso,et al.  Passive Motion Paradigm: An Alternative to Optimal Control , 2011, Front. Neurorobot..

[55]  Mark Rapley,et al.  Ethnomethodology/Conversation Analysis , 2011 .

[56]  K. Rohlfing,et al.  People modify their tutoring behavior in robot-directed interaction for action learning , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[57]  Adele E. Goldberg,et al.  Constructions at Work , 2005 .

[58]  Rajesh P. N. Rao,et al.  Bayesian brain : probabilistic approaches to neural coding , 2006 .

[59]  Stefano Nolfi,et al.  Co-development of Linguistic and Behavioural Skills: Compositional Semantics and Behaviour Generalisation , 2010, SAB.

[60]  Alexandre Bernardino,et al.  Multimodal saliency-based bottom-up attention a framework for the humanoid robot iCub , 2008, 2008 IEEE International Conference on Robotics and Automation.

[61]  Giorgio Metta,et al.  Actions and Imagined Actions in Cognitive Robots , 2011 .

[62]  Rna Henson,et al.  Short-term memory for serial order. , 2001 .

[63]  L. Fadiga,et al.  Automatic versus Voluntary Motor Imitation: Effect of Visual Context and Stimulus Velocity , 2010, PloS one.

[64]  A. van Knippenberg,et al.  Mimicry and Prosocial Behavior , 2004, Psychological science.

[65]  Matthew M Botvinick,et al.  Short-term memory for serial order: a recurrent neural network model. , 2006, Psychological review.

[66]  Angelo Cangelosi,et al.  Modeling U Shaped Performance Curves in Ongoing Development , 2011, CogSci.

[67]  R. Mishra,et al.  Self-Organization , 2021, Encyclopedic Dictionary of Archaeology.

[68]  Julian M. Pine,et al.  Constructing a Language: A Usage-Based Theory of Language Acquisition. , 2004 .

[69]  Chrystopher L. Nehaniv,et al.  Interactive Language Learning by Robots: The Transition from Babbling to Word Forms , 2012, PloS one.

[70]  G. Cheng,et al.  Social cognitive neuroscience and humanoid robotics , 2009, Journal of Physiology-Paris.

[71]  Peter Ford Dominey Emergence of grammatical constructions: evidence from simulation and grounded agent experiments , 2005, Connect. Sci..

[72]  T. Tweed Space , 2011, STEM Education in the Primary School.

[73]  Kerstin Fischer,et al.  Levels of embodiment: Linguistic analyses of factors influencing HRI , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[74]  Chrystopher L. Nehaniv,et al.  Preparing to Talk: Interaction between a Linguistically Enabled Agent and a Human Teacher , 2010, AAAI Fall Symposium: Dialog with Robots.

[75]  A. Noë,et al.  A sensorimotor account of vision and visual consciousness. , 2001, The Behavioral and brain sciences.

[76]  J. Elman,et al.  Language input and semantic categories: a relation between cognition and early word learning , 2006, Journal of Child Language.

[77]  R. Johansson,et al.  Action plans used in action observation , 2003, Nature.

[78]  Giulio Sandini,et al.  Human Machine Interaction and Communication in Cooperative Actions , 2014 .

[79]  P. Bloom How Children Learn the Meaning of Words and How LSA Does It ( Too ) , 2005 .

[80]  Walter Daelemans,et al.  Memory-Based Language Processing , 2009, Studies in natural language processing.

[81]  Angelo Cangelosi,et al.  An Embodied Model for Sensorimotor Grounding and Grounding Transfer: Experiments With Epigenetic Robots , 2006, Cogn. Sci..

[82]  Stefano Nolfi,et al.  The ITALK Project: A Developmental Robotics Approach to the Study of Individual, Social, and Linguistic Learning , 2014, Top. Cogn. Sci..

[83]  Caroline Lyon,et al.  Beyond Vision: Extending the Scope of a Sensorimotor Account of Perception , 2014 .

[84]  P. Morasso,et al.  Kinematic networks , 1988, Biological Cybernetics.

[85]  Giulio Sandini,et al.  Measuring Human-Robot Interaction Through Motor Resonance , 2012, Int. J. Soc. Robotics.

[86]  Jun Tani,et al.  Learning Semantic Combinatoriality from the Interaction between Linguistic and Behavioral Processes , 2005, Adapt. Behav..

[87]  Anthony F. Morse Snapshots of Sensorimotor Perception: Putting the Body Back into Embodiment , 2011, PT-AI.

[88]  Kerstin Fischer,et al.  The impact of the contingency of robot feedback on HRI , 2013, 2013 International Conference on Collaboration Technologies and Systems (CTS).

[89]  Jun Tani,et al.  Emergence of Functional Hierarchy in a Multiple Timescale Neural Network Model: A Humanoid Robot Experiment , 2008, PLoS Comput. Biol..

[90]  Jannik Fritsch,et al.  Tutoring in adult-child interaction: On the loop of the tutor’s action modification and the recipient’s gaze , 2014 .

[91]  G. Rizzolatti,et al.  Resonance behaviors and mirror neurons. , 1999, Archives italiennes de biologie.

[92]  Giulio Sandini,et al.  In Press, Ieee Transactions on Autonomous Mental Development , 2010 .

[93]  Nick Chater,et al.  Toward a connectionist model of recursion in human linguistic performance , 1999 .

[94]  P. Kuhl Is speech learning 'gated' by the social brain? , 2007, Developmental science.

[95]  Giulio Sandini,et al.  Teaching a humanoid robot to draw ‘Shapes’ , 2011, Auton. Robots.

[96]  Giulio Sandini,et al.  Inference Through Embodied Simulation in Cognitive Robots , 2013, Cognitive Computation.

[97]  Angelo Cangelosi,et al.  Scaling-up action learning neuro-controllers with GPUs , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[98]  D. Poeppel,et al.  Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language , 2004, Cognition.

[99]  Randall D. Beer,et al.  On the Dynamics of Small Continuous-Time Recurrent Neural Networks , 1995, Adapt. Behav..

[100]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[101]  Kerstin Fischer,et al.  Alignment or collaboration? How implicit views of communication influence robot design , 2014, 2014 International Conference on Collaboration Technologies and Systems (CTS).

[102]  K. Nelson Some Evidence for the Cognitive Primacy of Categorization and Its Functional Basis. , 1973 .

[103]  Linda B. Smith,et al.  Naming in young children: a dumb attentional mechanism? , 1996, Cognition.

[104]  N. Hogan,et al.  Modularity and Causality in Physical System Modelling , 1987 .

[105]  B. MacWhinney The CHILDES project: tools for analyzing talk , 1992 .

[106]  Marilyn M. Vihman,et al.  A dynamic systems approach to babbling and words , 2008 .

[107]  Chrystopher L. Nehaniv,et al.  Towards using prosody to scaffold lexical meaning in robots , 2011, 2011 IEEE International Conference on Development and Learning (ICDL).

[108]  Kerstin Fischer,et al.  Interpersonal variation in understanding robots as social actors , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[109]  Volkmar Glauche,et al.  Ventral and dorsal pathways for language , 2008, Proceedings of the National Academy of Sciences.

[110]  Chrystopher L. Nehaniv,et al.  From babbling towards first words: The emergence of speech in a robot in real-time interaction , 2011, 2011 IEEE Symposium on Artificial Life (ALIFE).

[111]  A. Bigelow,et al.  Sensitivity to Social Contingency From Mothers and Strangers in 2‐, 4‐, and 6‐Month‐Old Infants , 2003 .

[112]  Chrystopher L. Nehaniv,et al.  Robot learning of lexical semantics from sensorimotor interaction and the unrestricted speech of human tutors , 2010 .

[113]  F. Pulvermüller,et al.  Walking or Talking?: Behavioral and Neurophysiological Correlates of Action Verb Processing , 2001, Brain and Language.

[114]  Tony Belpaeme,et al.  Why Robots Should Be Social: Enhancing Machine Learning through Social Human-Robot Interaction , 2015, PloS one.

[115]  L. Steels,et al.  coordinating perceptually grounded categories through language: a case study for colour , 2005, Behavioral and Brain Sciences.

[116]  Frank Foerster Robots that Say ’No’: Acquisition of Linguistic Behaviour in Interaction Games with Humans , 2013 .

[117]  Steven Greenberg,et al.  An elitist approach to automatic articulatory-acoustic feature classification for phonetic characterization of spoken language , 2005, Speech Commun..

[118]  Stefanie Tellex,et al.  The Human Speechome Project , 2006, EELC.

[119]  Tuomas Teinonen,et al.  Language learning in infancy , 2009 .

[121]  Stefan Kopp,et al.  Appropriate feedback in asymmetric interactions , 2010 .

[122]  Rachid Alami,et al.  Physiological and subjective evaluation of a human-robot object hand-over task. , 2011, Applied ergonomics.

[123]  Bénédicte de Boysson-Bardies How language comes to children , 2009 .

[124]  J. Mandler How to build a baby: II. Conceptual primitives. , 1992, Psychological review.

[125]  Katharina J. Rohlfing,et al.  How can multimodal cues from child-directed interaction reduce learning complexity in robots? , 2006, Adv. Robotics.

[126]  P J Webros BACKPROPAGATION THROUGH TIME: WHAT IT DOES AND HOW TO DO IT , 1990 .

[127]  Mary E. Hughes,et al.  Using corpora to examine discourse effects in syntax , 2008 .

[128]  John Mark Bishop,et al.  Philosophy and Theory of Artificial Intelligence , 2013 .

[129]  P. Jusczyk,et al.  Infants′ Detection of the Sound Patterns of Words in Fluent Speech , 1995, Cognitive Psychology.