A Deep Learning approach to modeling competitiveness in spoken conversations

The motivation behind the research on overlapping speech has always been dominated by the need to model human-machine interaction for dialog systems and conversation analysis. To have more complex insights of the interlocutors' intentions behind the interaction, we need to understand the type of overlaps. Overlapping speech signals the interlocutor's intention to grab the floor. This act could be a competitive or non-competitive act, which either signals a problem or indicates assistance in communication. In this paper, we present a Deep Learning approach to modeling competitiveness in overlapping speech using acoustic and lexical features and their combination. We compare a fully-connected feed-forward neural network to the Support Vector Machine (SVM) models on real call center human-human conversations. We have observed that feature combination with DNN (significantly) outperforms SVM models, both the individual feature baselines and the feature combination model by 4% and 2% respectively.

[1]  Björn W. Schuller,et al.  Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge , 2011, Speech Commun..

[2]  Guy J. Brown,et al.  Resources for turn competition in overlapping talk , 2013, Speech Commun..

[3]  Morena Danieli,et al.  Annotating and categorizing competition in overlap speech , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  Khiet P. Truong,et al.  Classification of cooperative and competitive overlaps in speech using cues from the context, overlapper, and overlappee , 2013, INTERSPEECH.

[5]  Mattias Heldner,et al.  Pauses, gaps and overlaps in conversations , 2010, J. Phonetics.

[6]  Peter Glöckner,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2013 .

[7]  J. Goldberg.  Interrupting the discourse on interruptions , 1990 .

[8]  Firoj Alam,et al.  Emotion Unfolding and Affective Scenes: A Case Study in Spoken Conversations , 2015, ERM4CT@ICMI.

[9]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[10]  Firoj Alam,et al.  Unsupervised recognition and clustering of speech overlaps in spoken conversations , 2014, SLAM@INTERSPEECH.

[11]  Björn W. Schuller,et al.  Recent developments in openSMILE, the munich open-source multimedia feature extractor , 2013, ACM Multimedia.

[12]  C. West,et al.  AGAINST OUR WILL: MALE INTERRUPTIONS OF FEMALES IN CROSS‐SEX CONVERSATION * , 1979 .

[13]  William J. Dally,et al.  The GPU Computing Era , 2010, IEEE Micro.

[14]  Shrikanth S. Narayanan,et al.  An analysis of multimodal cues of interruption in dyadic spoken interactions , 2008, INTERSPEECH.

[15]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[16]  Ming Zhou,et al.  A Recursive Recurrent Neural Network for Statistical Machine Translation , 2014, ACL.

[17]  Firoj Alam,et al.  Comparative study of speaker personality traits recognition in conversational and broadcast news speech , 2013, INTERSPEECH.

[18]  Catharine Oertel,et al.  Context Cues For Classification Of Competitive And Collaborative Overlaps , 2012 .

[19]  E. Schegloff.  Overlapping talk and the organization of turn-taking for conversation , 2000, Language in Society.

[20]  S. A. Chowdhury,et al.  Predicting User Satisfaction from Turn-Taking in Spoken Conversations , 2016, INTERSPEECH.

[21]  E. Schegloff,et al.  A simplest systematics for the organization of turn-taking for conversation , 2015 .

[22]  S. Duncan,et al.  Some Signals and Rules for Taking Speaking Turns in Conversations , 1972 .

[23]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[24]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[25]  Peter French,et al.  Turn-competitive incomings , 1983 .

[26]  Morena Danieli,et al.  The role of speakers and context in classifying competition in overlapping speech , 2015, INTERSPEECH.

[27]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..