Telerobotic Pointing Gestures Shape Human Spatial Cognition

This paper aimed to explore whether human beings can understand gestures produced by telepresence robots. If it were the case, they can derive meaning conveyed in telerobotic gestures when processing spatial information. We conducted two experiments over Skype in the present study. Participants were presented with a robotic interface that had arms, which were teleoperated by an experimenter. The robot can point to virtual locations that represented certain entities. In Experiment 1, the experimenter described spatial locations of fictitious objects sequentially in two conditions: speech only condition (SO, verbal descriptions clearly indicated the spatial layout) and speech and robotic gesture condition (SR, verbal descriptions were ambiguous but accompanied by robotic pointing gestures). Participants were then asked to recall the objects’ spatial locations. We found that the number of spatial locations recalled in the SR condition was on par with that in the SO condition, suggesting that telerobotic pointing gestures compensated ambiguous speech during the process of spatial information. In Experiment 2, the experimenter described spatial locations non-sequentially in the SR and SO conditions. Surprisingly, the number of spatial locations recalled in the SR condition was even higher than that in the SO condition, suggesting that telerobotic pointing gestures were more powerful than speech in conveying spatial information when information was presented in an unpredictable order. The findings provide evidence that human beings are able to comprehend telerobotic gestures, and importantly, integrate these gestures with co-occurring speech. This work promotes engaging remote collaboration among humans through a robot intermediary.

[1]  Vilayanur S. Ramachandran,et al.  EEG evidence for mirror neuron activity during the observation of human and robot actions: Toward an analysis of the human qualities of interactive robots , 2007, Neurocomputing.

[2]  De Ruiter,et al.  Can gesticulation help aphasic people speak, or rather, communicate? , 2006 .

[3]  Raymond W. Kulhavy,et al.  Conjoint retention of maps and related discourse , 1985 .

[4]  H. Ishiguro,et al.  CAN A TELEOPERATED ANDROID REPRESENT PERSONAL PRESENCE? - A CASE STUDY WITH CHILDREN , 2007 .

[5]  Enhancing Museum Visitor Access Through Robotic Avatars Connected to the Web , 2001 .

[6]  R. W. Kulhavy,et al.  Chapter 10 Reference Maps as a Framework for Remembering Text , 1994 .

[7]  Ming-Yuan Lee,et al.  Clinical application of NMP22 and urinary cytology in patients with hematuria or a history of urothelial carcinoma , 2000, World Journal of Urology.

[8]  Kevin K Chung,et al.  Robotic telepresence: past, present, and future. , 2007, Journal of cardiothoracic and vascular anesthesia.

[9]  D. Kimura,et al.  Hand movement asymmetry during speech: No effect of speaking topic , 1987, Neuropsychologia.

[10]  Honghai Liu,et al.  Intelligent Robotics and Applications , 2014, Lecture Notes in Computer Science.

[11]  H. Deng,et al.  Building artificial humans to understand humans , 2007, Journal of Artificial Organs.

[12]  Cynthia Breazeal,et al.  MeBot: A robotic platform for socially embodied telepresence , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[13]  Robert E. Kraut,et al.  Visual Information as a Conversational Resource in Collaborative Physical Tasks , 2003, Hum. Comput. Interact..

[14]  M. Anvari,et al.  The role of telementoring and telerobotic assistance in the provision of laparoscopic colorectal surgery in rural areas , 2006, Surgical Endoscopy And Other Interventional Techniques.

[15]  Susan R. Fussell,et al.  Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks , 2004, Hum. Comput. Interact..

[16]  Erico Guizzo.  When My Avatar Went to Work , 2010, IEEE Spectrum.

[17]  Naoyuki Kubota,et al.  Essential Technology for Team Teaching Using Robot Partners , 2010, ICIRA.

[18]  Li-Ming Su,et al.  The RoboConsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. , 2007, Urology.

[19]  D. L. Hintzman,et al.  Orientation in cognitive maps , 1981, Cognitive Psychology.

[20]  Tetsuo Ono,et al.  Can an android persuade you? , 2018, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[21]  Medi Nazar,et al.  Pointing Gestures for a Robot Mediated Communication Interface , 2009, ICIRA.

[22]  Judith S. Olson,et al.  Distance Matters , 2000, Hum. Comput. Interact..

[23]  C Daniel Smith,et al.  Remote Presence Proctoring by Using a Wireless Remote-Control Videoconferencing System , 2005, Surgical innovation.

[24]  Steven S Rothenberg,et al.  Initial experience with surgical telementoring in pediatric laparoscopic surgery using remote presence technology. , 2008, Journal of laparoendoscopic & advanced surgical techniques. Part A.

[25]  Kazuya Takemata,et al.  Design of Engineering Education System Using Long-Distance-Controlled Robots , 2009, 2009 Ninth IEEE International Conference on Advanced Learning Technologies.

[26]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[27]  Tetsuo Ono,et al.  A humanoid robot that pretends to listen to route guidance from a human , 2007, Auton. Robots.

[28]  R. Moore,et al.  International telementoring: a feasible method of instruction , 2000, World Journal of Urology.

[29]  D. Sampsel,et al.  Robots as Faculty: Student and Faculty Perceptions , 2011 .

[30]  J. Petelin,et al.  Deployment and early experience with remote-presence patient care in a community hospital , 2005, Surgical Endoscopy.

[31]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[32]  M. Alibali,et al.  Gesture and the process of speech production: We think, therefore we gesture , 2000 .

[33]  J. Huttenlocher,et al.  Making Space: The Development of Spatial Representation and Reasoning , 2000 .

[34]  Dan Stoianovici,et al.  Telerounding and patient satisfaction after surgery. , 2004, Journal of the American College of Surgeons.

[35]  A. Minelli.  BIO , 2009, Evolution & Development.

[36]  Wolfram Burgard,et al.  Experiences with an Interactive Museum Tour-Guide Robot , 1999, Artif. Intell..

[37]  Jacques Marescaux,et al.  Telementoring for Minimally Invasive Surgical Training by Wireless Robot , 2007, Surgical innovation.

[38]  Myeong-Ho Sohn,et al.  Viewpoint alignment and response conflict during spatial judgment , 2003, Psychonomic bulletin & review.

[39]  Mitesh Parekh,et al.  Robotic telepresence: profit analysis in reducing length of stay after laparoscopic gastric bypass. , 2007, Journal of the American College of Surgeons.

[40]  D. McNeill,et al.  Abstract deixis , 1993 .

[41]  W. Gill Woodall,et al.  Nonverbal cue context and episodic memory: On the availability and endurance of nonverbal behaviors as retrieval cues , 1985 .

[42]  Martha W. Alibali,et al.  Gesture in Spatial Cognition: Expressing, Communicating, and Thinking About Spatial Information , 2005, Spatial Cogn. Comput..

[43]  Rifat Latifi,et al.  Telepresence and telementoring in surgery. , 2004, Studies in health technology and informatics.

[44]  Marios N Avraamides,et al.  Locating targets from imagined perspectives: Comparing labelling with pointing responses , 2007, Quarterly journal of experimental psychology.

[45]  A. Bangerter,et al.  Using Pointing and Describing to Achieve Joint Focus of Attention in Dialogue , 2004, Psychological science.

[46]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[47]  Cynthia Breazeal,et al.  MeBot: a robotic platform for socially embodied presence , 2010, HRI.

[48]  S. Kita.  Pointing: Where language, culture, and cognition meet , 2003 .

[49]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[50]  M. Talamini,et al.  Long distance telementoring. A novel tool for laparoscopy aboard the USS Abraham Lincoln. , 1999, Surgical endoscopy.

[51]  David McNeill,et al.  Language and Gesture: Frontmatter , 2000 .

[52]  Susan Goldin-Meadow,et al.  Using the Hands to Identify Who Does What to Whom: Gesture and Speech Go Hand-in-Hand , 2009, Cogn. Sci..

[53]  John F. Canny,et al.  Social Tele-Embodiment: Understanding Presence , 2001, Auton. Robots.

[54]  Raymond W. Kulhavy,et al.  How geographic maps increase recall of instructional text , 1993 .

[55]  Chad M. Miller,et al.  Intensive care unit robotic telepresence facilitates rapid physician response to unstable patients and decreased cost in neurointensive care. , 2007, Surgical neurology.

[56]  David B. Kaber,et al.  Telepresence , 1998, Hum. Factors.

[57]  H. H. Clark.  Pointing and placing. , 2003 .