Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation

Sequence-to-sequence models for open-domain dialogue generation tend to favor generic, uninformative responses. Past work has focused on word frequency-based approaches to improving specificity, such as penalizing responses with only common words. In this work, we examine whether specificity is solely a frequency-related notion and find that more linguistically-driven specificity measures are better suited to improving response informativeness. However, we find that forcing a sequence-to-sequence model to be more specific can expose a host of other problems in the responses, including flawed discourse and implausible semantics. We rerank our model’s outputs using externally-trained classifiers targeting each of these identified factors. Experiments show that our final model using linguistically motivated specificity and plausibility reranking improves the informativeness, reasonableness, and grammatically of responses.

[1]  Junyi Jessy Li,et al.  The Instantiation Discourse Relation: A Corpus Analysis of Its Properties and Improved Detection , 2016, NAACL.

[2]  Dan Klein,et al.  An Empirical Investigation of Statistical Significance in NLP , 2012, EMNLP.

[3]  Giuseppe Riccardi,et al.  Semantic Information Processing Of Spoken Language , 2000 .

[4]  Joelle Pineau,et al.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models , 2015, AAAI.

[5]  Luca Lugini,et al.  Predicting Specificity in Classroom Discussion , 2017, BEA@EMNLP.

[6]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[7]  Osmar R. Zaïane,et al.  Automatic Dialogue Generation with Expressed Emotions , 2018, NAACL.

[8]  Xueqi Cheng,et al.  Learning to Control the Specificity in Neural Response Generation , 2018, ACL.

[9]  Di He,et al.  FRAGE: Frequency-Agnostic Word Representation , 2018, NeurIPS.

[10]  Joelle Pineau,et al.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation , 2016, EMNLP.

[11]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[12]  Jianfeng Gao,et al.  Deep Reinforcement Learning for Dialogue Generation , 2016, EMNLP.

[13]  Karla A. Lassonde,et al.  Contextual Specificity in the Activation of Predictive Inferences , 2009 .

[14]  Yejin Choi,et al.  Learning to Write with Cooperative Discriminators , 2018, ACL.

[15]  Alan Ritter,et al.  Adversarial Learning for Neural Dialogue Generation , 2017, EMNLP.

[16]  Jianfeng Gao,et al.  A Diversity-Promoting Objective Function for Neural Conversation Models , 2015, NAACL.

[17]  Jianfeng Gao,et al.  A Persona-Based Neural Conversation Model , 2016, ACL.

[18]  Katrin Erk,et al.  Modeling Semantic Plausibility by Injecting World Knowledge , 2018, NAACL.

[19]  Jason Weston,et al.  Personalizing Dialogue Agents: I have a dog, do you have pets too? , 2018, ACL.

[20]  Jun Xu,et al.  Reinforcing Coherence for Sequence to Sequence Model in Dialogue Generation , 2018, IJCAI.

[21]  Ani Nenkova,et al.  Automatic identification of general and specific sentences by leveraging discourse annotations , 2011, IJCNLP.

[22]  Verena Rieser,et al.  Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity , 2018, EMNLP.

[23]  Pierre Lison,et al.  Not All Dialogues are Created Equal: Instance Weighting for Neural Conversational Models , 2017, SIGDIAL Conference.

[24]  Zhoujun Li,et al.  Neural Response Generation with Dynamic Vocabularies , 2017, AAAI.

[25]  Raffaella Bernardi,et al.  There Is No Logical Negation Here, But There Are Alternatives: Modeling Conversational Negation with Distributional Semantics , 2016, Computational Linguistics.

[26]  Robert L. Mercer,et al.  An Estimate of an Upper Bound for the Entropy of English , 1992, CL.

[27]  Kam-Fai Wong,et al.  An Attentional Neural Conversation Model with Improved Specificity , 2016, ArXiv.

[28]  Alan Ritter,et al.  Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints , 2018, EMNLP.

[29]  Jörg Tiedemann,et al.  News from OPUS — A collection of multilingual parallel corpora with tools and interfaces , 2009 .

[30]  Daniel Jurafsky,et al.  A Simple, Fast Diverse Decoding Algorithm for Neural Generation , 2016, ArXiv.

[31]  Junyi Jessy Li,et al.  Improving the Annotation of Sentence Specificity , 2016, LREC.

[32]  Tomas Mikolov,et al.  RNNLM - Recurrent Neural Network Language Modeling Toolkit , 2011 .

[33]  Samuel R. Bowman,et al.  Neural Network Acceptability Judgments , 2018, Transactions of the Association for Computational Linguistics.

[34]  Wei-Ying Ma,et al.  Topic Aware Neural Response Generation , 2016, AAAI.

[35]  Bo Chen,et al.  Mechanism-Aware Neural Machine for Dialogue Response Generation , 2017, AAAI.

[36]  Junyi Jessy Li,et al.  Domain Agnostic Real-Valued Specificity Prediction , 2018, AAAI.

[37]  Dongyan Zhao,et al.  RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems , 2017, AAAI.

[38]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[39]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[40]  Holger Schwenk,et al.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data , 2017, EMNLP.

[41]  Shuming Shi,et al.  Towards Less Generic Responses in Neural Conversation Models: A Statistical Re-weighting Method , 2018, EMNLP.

[42]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[43]  Denny Britz,et al.  Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models , 2017, EMNLP.

[44]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[45]  Jianfeng Gao,et al.  A Neural Network Approach to Context-Sensitive Generation of Conversational Responses , 2015, NAACL.

[46]  P. Dixon.  The processing of organizational and component step information in written directions , 1987 .

[47]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[48]  Joshua Goodman,et al.  A bit of progress in language modeling , 2001, Comput. Speech Lang..

[49]  Xiaoyu Shen,et al.  Improving Variational Encoder-Decoders in Dialogue Generation , 2018, AAAI.

[50]  Junyi Jessy Li,et al.  Fast and Accurate Prediction of Sentence Specificity , 2015, AAAI.

[51]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.