Introducing the NEMO-Lowlands iconic gesture dataset, collected through a gameful human–robot interaction

This paper describes a novel dataset of iconic gestures, together with a publicly available robot-based elicitation method to record these gestures, which consists of playing a game of charades with a humanoid robot. The game was deployed at a science museum (NEMO) and a large popular music festival (Lowlands) in the Netherlands. This resulted in recordings of 428 participants, both adults and children, performing 3715 silent iconic gestures for 35 different objects in a naturalistic setting. Our dataset adds to existing collections of iconic gesture recordings in two important ways. First, participants were free to choose how they represented the broad concepts using gestures, and they were asked to perform a second attempt if the robot did not recognize their gesture the first time. This provides insight into potential repair strategies that might be used. Second, by making the interactive game available we enable other researchers to collect additional recordings, for different concepts, and in diverse cultures or contexts. This can be done in a consistent manner because a robot is used as a confederate in the elicitation procedure, which ensures that every data collection session plays out in the same way. The current dataset can be used for research into human gesturing behavior, and as input for the gesture recognition and production capabilities of robots and virtual agents.

[1]  Xiangshi Ren,et al.  Jump and shoot!: prioritizing primary and alternative body gestures for intense gameplay , 2014, CHI.

[2]  P. Prieto,et al.  Effects of Encouraging the Use of Gestures on Speech. , 2019, Journal of speech, language, and hearing research : JSLHR.

[3]  Emiel Krahmer,et al.  Production and Comprehension of Pantomimes Used to Depict Objects , 2017, Front. Psychol..

[4]  T. Trabasso,et al.  Offering a Hand to Pragmatic Understanding: The Role of Speech and Gesture in Comprehension and Memory , 1999 .

[5]  Cornelia Müller,et al.  128. Gestural modes of representation as techniques of depiction , 2014 .

[6]  Sonia Chernova,et al.  Humanoid robot control using depth camera , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[7]  Alexander H. Waibel,et al.  Multimodal error correction for speech user interfaces , 2001, TCHI.

[8]  Sotaro Kita,et al.  Gestural depiction of motion events in narrative increases symbolic distance with age , 2018 .

[9]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[10]  Juan Pablo Wachs,et al.  What Makes a Gesture a Gesture? Neural Signatures Involved in Gesture Recognition , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[11]  Stefan Kopp,et al.  The Effect of a Robot’s Gestures and Adaptive Tutoring on Children’s Acquisition of Second Language Vocabularies , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[12]  R. Krauss,et al.  Changes in reference phrases as a function of frequency of usage in social interaction: a preliminary study , 1964 .

[13]  Joseph J. LaViola,et al.  GestureBar: improving the approachability of gesture-based interfaces , 2009, CHI.

[14]  Linda Drijvers,et al.  Visual Context Enhanced: The Joint Contribution of Iconic Gestures and Visible Speech to Degraded Speech Comprehension. , 2017, Journal of speech, language, and hearing research : JSLHR.

[15]  Irina Simanova,et al.  Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research , 2018, Behavior Research Methods.

[16]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[17]  C. Bartneck,et al.  A design-centred framework for social human-robot interaction , 2004, RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759).

[18]  Tomoaki Nakamura,et al.  Generation of Gestures During Presentation for Humanoid Robots , 2018, 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[19]  Elena Mugellini,et al.  A Survey of Datasets for Human Gesture Recognition , 2014, HCI.

[20]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[21]  James A. Dixon,et al.  Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles , 2020 .

[22]  Miguel Angel Salichs,et al.  A social robot as an aloud reader: Putting together recognition and synthesis of voice and gestures for HRI experimentation , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[23]  Emiel Krahmer,et al.  Pantomime Strategies: On Regularities in How People Translate Mental Representations into the Gesture Modality , 2014, CogSci.

[24]  Naomi Sweller,et al.  Not All Gestures are Created Equal: The Effects of Typical and Atypical Iconic Gestures on Narrative Comprehension , 2018 .

[25]  Youngwoo Yoon,et al.  Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[26]  Emiel Krahmer,et al.  Playing Charades with a Robot: Collecting a Large Dataset of Human Gestures Through HRI , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[27]  E. Krahmer,et al.  The comprehensibility of pantomimes produced by people with aphasia. , 2018, International journal of language & communication disorders.

[28]  Jörg Müller,et al.  StrikeAPose: revealing mid-air gestures on public displays , 2013, CHI.

[29]  Lauren J. Stites,et al.  Who Did What to Whom? Children Track Story Referents First in Gesture , 2017, Journal of psycholinguistic research.

[30]  D. McNeill.  Hand and Mind: What Gestures Reveal about Thought , 1992 .

[31]  Anne Marie Piper,et al.  A Wizard-of-Oz elicitation study examining child-defined gestures with a whole-body interface , 2013, IDC.

[32]  Stefan Kopp,et al.  Gesture-based Object Recognition using Histograms of Guiding Strokes , 2012, BMVC.

[33]  Asli Özyürek,et al.  Generalisable patterns of gesture distinguish semantic categories in communication without language , 2016, CogSci.

[34]  Eakta Jain,et al.  Kinder-Gator: The UF Kinect Database of Child and Adult Motion , 2018, Eurographics.

[35]  J. Holler,et al.  Co-Speech Gesture Mimicry in the Process of Collaborative Referring During Face-to-Face Dialogue , 2011 .

[36]  Wenbing Zhao,et al.  A Survey of Applications and Human Motion Recognition with Microsoft Kinect , 2015, Int. J. Pattern Recognit. Artif. Intell..

[37]  Aslı Özyürek,et al.  Systematic mappings between semantic categories and types of iconic representations in the manual modality: A normed database of silent gesture , 2019, Behavior Research Methods.

[38]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[39]  Katherine Guérard,et al.  Bank of Standardized Stimuli (BOSS) Phase II: 930 New Normative Photos , 2014, PloS one.

[40]  Daniel J. Wigdor,et al.  Gesture play: motivating online gesture learning with fun, positive reinforcement and physical metaphors , 2010, ITS '10.

[41]  Jan de Wit,et al.  Varied Human-Like Gestures for Social Robots: Investigating the Effects on Children’s Engagement and Language Learning , 2020, 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[42]  Isaac Wang,et al.  EGGNOG: A Continuous, Multi-modal Data Set of Naturally Occurring Gestures with Ground Truth Labels , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[43]  S. B. Needleman,et al.  A general method applicable to the search for similarities in the amino acid sequence of two proteins. , 1970, Journal of molecular biology.

[44]  Juan Pablo Wachs,et al.  A Human-Centered Approach to One-Shot Gesture Learning , 2017, Front. Robot. AI.

[45]  S. Goldin-Meadow,et al.  Hearing Gesture: How Our Hands Help Us Think , 2003 .

[46]  mc schraefel,et al.  A Taxonomy of Gestures in Human Computer Interactions , 2005 .

[47]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[48]  Emiel Krahmer,et al.  How What We See and What We Know Influence Iconic Gesture Production , 2017, Journal of Nonverbal Behavior.

[49]  N. Altman.  An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression , 1992 .

[50]  Hao Chen,et al.  Towards More Realistic Human-Robot Conversation: A Seq2Seq-based Body Gesture Interaction System , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[51]  Eakta Jain,et al.  Is the Motion of a Child Perceivably Different from the Motion of an Adult? , 2016, ACM Trans. Appl. Percept..

[52]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Radu-Daniel Vatavu,et al.  The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies , 2019, CHI.

[54]  Abhinav Dhall,et al.  Automatic Speech-Gesture Mapping and Engagement Evaluation in Human Robot Interaction , 2018, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[55]  Hannes Rieser,et al.  On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA) , 2009, Gesture Workshop.

[56]  Emiel Krahmer,et al.  Coming of age in gesture : A comparative study of gesturing and pantomiming in older children and adults , 2015 .

[57]  Sotaro Kita,et al.  Cross-cultural variation of speech-accompanying gesture: A review , 2009, Speech Accompanying-Gesture.

[58]  M. Brysbaert,et al.  Age-of-acquisition ratings for 30,000 English words , 2012, Behavior research methods.

[59]  Tarik Arici,et al.  Robust gesture recognition using feature pre-processing and weighted dynamic time warping , 2014, Multimedia Tools and Applications.