A Corpus of Gesture-Annotated Dialogues for Monologue-to-Dialogue Generation from Personal Narratives

Story-telling is a fundamental and prevalent aspect of human social behavior. In the wild, stories are told conversationally in social settings, often as a dialogue and with accompanying gestures and other nonverbal behavior. This paper presents a new corpus, the Story Dialogue with Gestures (SDG) corpus, consisting of 50 personal narratives regenerated as dialogues, complete with annotations of gesture placement and accompanying gesture forms. The corpus includes dialogues generated by human annotators, gesture annotations on the human generated dialogues, videos of story dialogues generated from this representation, video clips of each gesture used in the gesture annotations, and annotations of the original personal narratives with a deep representation of story called a Story Intention Graph. Our long term goal is the automatic generation of story co-tellings as animated dialogues from the Story Intention Graph. We expect this corpus to be a useful resource for researchers interested in natural language generation, intelligent virtual agents, generation of nonverbal behavior, and story and narrative representations.

[1]  Hung-Hsuan Huang,et al.  From observation to simulation: generating culture-specific behavior for interactive systems , 2009, AI & SOCIETY.

[2]  Michael Neff,et al.  Storytelling Agents with Personality and Adaptivity , 2015, IVA.

[3]  Michael Neff,et al.  The Influence of Prosody on the Requirements for Gesture-Text Alignment , 2013, IVA.

[4]  Justine Cassell,et al.  Virtual peers as partners in storytelling and literacy learning , 2003, J. Comput. Assist. Learn..

[5]  Sotaro Kita,et al.  Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders , 1997, Gesture Workshop.

[6]  Michael Neff,et al.  Evaluating the Effect of Gesture and Language on Personality Perception in Conversational Agents , 2010, IVA.

[7]  Thomas Rist,et al.  The automated design of believable dialogues for animated presentation teams , 2001 .

[8]  Michael Neff,et al.  Gestural Adaptation in Extravert-Introvert Pairs and Implications for IVAs , 2013 .

[9]  Marilyn A. Walker,et al.  Narrative Variations in a Virtual Storyteller , 2015, IVA.

[10]  Marilyn A. Walker,et al.  Generating Different Story Tellings from Semantic Representations of Narrative , 2013, ICIDS.

[11]  Michael Neff,et al.  A Multimodal Motion-Captured Corpus of Matched and Mismatched Extravert-Introvert Conversational Pairs , 2016, LREC.

[12]  Michael Neff,et al.  A Verbal and Gestural Corpus of Story Retellings to an Expressive Embodied Virtual Character , 2016, LREC.

[13]  Hans-Peter Seidel,et al.  Annotated New Text Engine Animation Animation Lexicon Animation Gesture Profiles MR : . . . JL : . . . Gesture Generation Video Annotated Gesture Script , 2007 .

[14]  Michael Neff,et al.  Don't Scratch! Self-adaptors Reflect Emotional Stability , 2011, IVA.

[15]  Michael Neff,et al.  Two Techniques for Assessing Virtual Agent Personality , 2016, IEEE Transactions on Affective Computing.

[16]  Katherine Nelson,et al.  Narrative and self, myth and memory: Emergence of the cultural self. , 2003 .

[17]  J. Pennebaker,et al.  Forming a story: the health benefits of narrative. , 1999, Journal of clinical psychology.

[18]  Seiji Yamada,et al.  Persuasion through overheard communication by life-like agents , 2004 .

[19]  R. Swanson,et al.  Identifying Personal Stories in Millions of Weblog Entries , 2009, ICWSM 2009.

[20]  Kathleen McKeown,et al.  A Tool for Deep Semantic Encoding of Narrative Texts , 2009, ACL/IJCNLP.

[21]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[22]  David K. Elson,et al.  DramaBank: Annotating Agency in Narrative Discourse , 2012, LREC.

[23]  J. Bruner.  The Narrative Construction of Reality , 1991, Critical Inquiry.

[24]  S. Gosling,et al.  A very brief measure of the Big-Five personality domains , 2003 .

[25]  David K. Elson.  Detecting Story Analogies from Annotations of Time, Action and Agency , 2012 .

[26]  Kathleen McKeown,et al.  Automatic Attribution of Quoted Speech in Literary Narrative , 2010, AAAI.

[27]  Akshay Java,et al.  The ICWSM 2009 Spinn3r Dataset , 2009 .

[28]  Albert A. Rizzo,et al.  User-State Sensing for Virtual Health Agents and TeleHealth Applications , 2013, MMVR.

[29]  Robyn Fivush,et al.  Family narrative interaction and children's sense of self. , 2006, Family process.

[30]  James C. Lester,et al.  Life-Like Pedagogical Agents in Constructivist Multimedia Environments: Cognitive Consequences of their Interaction , 2000 .

[31]  W. Labov,et al.  Narrative analysis: Oral versions of personal experience. , 1997 .

[32]  Marilyn A. Walker,et al.  Generating Sentence Planning Variations for Story Telling , 2015, SIGDIAL Conference.

[33]  Marilyn A. Walker,et al.  PersonaBank: A Corpus of Personal Narratives and Their Story Intention Graphs , 2016, LREC.

[34]  John Lee,et al.  Supporting Student Discussions: it isn't Just Talk , 1998, Education and Information Technologies.