Multimodality and Dialogue Act Classification in the RoboHelper Project

We describe the annotation of a multimodal corpus that includes pointing gestures and haptic actions (force exchanges). Haptic actions are rarely analyzed as fullfledged components of dialogue, but our data shows haptic actions are used to advance the state of the interaction. We report our experiments on recognizing Dialogue Acts in both offline and online modes. Our results show that multimodal features and the dialogue game aid in DA classification.

[1]  Timothy Baldwin,et al.  Classifying Dialogue Acts in Multi-party Live Chats , 2012, PACLIC.

[2]  Alon Lavie,et al.  A discourse coding scheme for conversational Spanish , 1998, ICSLP.

[3]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[4]  Timothy Baldwin,et al.  Classifying Dialogue Acts in One-on-One Live Chats , 2010, EMNLP.

[5]  Helen F. Hastie,et al.  Automatically predicting dialogue structure using prosodic features , 2002, Speech Commun..

[6]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[7]  Andreas Stolcke,et al.  Dialogue act modeling for automatic tagging and recognition of conversational speech , 2000, CL.

[8]  Kristy Elizabeth Boyer,et al.  Combining Verbal and Nonverbal Features to Overcome the “Information Gap” in Task-Oriented Dialogue , 2012, SIGDIAL Conference.

[9]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[10]  Barbara Di Eugenio,et al.  Improving Pronominal and Deictic Co-Reference Resolution with Multi-Modal Features , 2011, SIGDIAL Conference.

[11]  Kristy Elizabeth Boyer,et al.  An Affect-Enriched Dialogue Act Classification Model for Task-Oriented Dialogue , 2011, ACL.

[12]  Lauri Carlson.  Dialogue Games: An Approach to Discourse Analysis , 1982 .

[13]  Jean Carletta,et al.  Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus , 2007, Lang. Resour. Evaluation.

[14]  Elizabeth Shriberg,et al.  The ICSI Meeting Recorder Dialog Act (MRDA) Corpus , 2004, SIGDIAL Workshop.

[15]  Edward Ivanovic,et al.  Automatic instant messaging dialogue using statistical models and dialogue acts , 2008 .

[16]  Christopher D. Manning,et al.  The Stanford Typed Dependencies Representation , 2008, CF+CDPE@COLING.

[17]  K. Krapp.  The Gale encyclopedia of nursing & allied health , 2002 .

[18]  Barbara Di Eugenio,et al.  Co-reference via Pointing and Haptics in Multi-Modal Dialogues , 2012, HLT-NAACL.

[19]  Andreas Stolcke,et al.  Joint Segmentation and Classification of Dialog Acts in Multiparty Meetings , 2005, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[20]  Anne H. Anderson,et al.  The Hcrc Map Task Corpus , 1991 .

[21]  Jezekiel Ben-Arie,et al.  Speech recognition by indexing and sequencing , 2010, 2010 International Conference of Soft Computing and Pattern Recognition.

[22]  Barbara Di Eugenio,et al.  Dialogue Act Classification, Higher Order Dialogue Structure, and Instance-Based Learning , 2010 .

[23]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[24]  Kai Ma,et al.  Multi-view multi-class object detection via exemplar compounding , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[25]  Alois Knoll,et al.  The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[26]  Shrikanth S. Narayanan,et al.  Combining lexical, syntactic and prosodic cues for improved online dialog act tagging , 2009, Comput. Speech Lang..

[27]  Kay Mills.  Towards Effective Communication , 2004 .

[28]  Barbara Di Eugenio,et al.  Towards Effective Communication with Robotic Assistants for the Elderly: Integrating Speech, Vision and Haptics , 2010, AAAI Fall Symposium: Dialog with Robots.