Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

A long-term goal of machine learning is to build intelligent conversational agents. One recent popular approach is to train end-to-end models on a large amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals & Le, 2015; Shang et al., 2015). However, this approach leaves many questions unanswered as an understanding of the precise successes and shortcomings of each model is hard to assess. A contrasting recent proposal are the bAbI tasks (Weston et al., 2015b) which are synthetic data that measure the ability of learning machines at various reasoning tasks over toy language. Unfortunately, those tests are very small and hence may encourage methods that do not scale. In this work, we propose a suite of new tasks of a much larger scale that attempt to bridge the gap between the two regimes. Choosing the domain of movies, we provide tasks that test the ability of models to answer factual questions (utilizing OMDB), provide personalization (utilizing MovieLens), carry short conversations about the two, and finally to perform on natural dialogs from Reddit. We provide a dataset covering 75k movie entities and with 3.5M training examples. We present results of various models on these tasks, and evaluate their performance.

[1]  Lewis M. Norton,et al.  Beyond Class A: A Proposal for Automatic Evaluation of Discourse , 1990, HLT.

[2]  Marilyn A. Walker,et al.  PARADISE: A Framework for Evaluating Spoken Dialogue Agents , 1997, ACL.

[3]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[4]  Peter Stone,et al.  Cobot in LambdaMOO: A Social Statistics Agent , 2000, AAAI/IAAI.

[5]  Tim Paek.  Empirical Methods for Evaluating Dialog Systems , 2001, SIGDIAL Workshop.

[6]  Johanna D. Moore,et al.  Fish or Fowl:A Wizard of Oz Evaluation of Dialogue Strategies in the Restaurant Domain , 2002, LREC.

[7]  Marilyn A. Walker,et al.  A trainable generator for recommendations in multimodal dialog , 2003, INTERSPEECH.

[8]  David Griol,et al.  A statistical approach to spoken dialog systems design and evaluation , 2008, Speech Commun..

[9]  Alex Graves,et al.  Supervised Sequence Labelling with Recurrent Neural Networks , 2012, Studies in Computational Intelligence.

[10]  Yehuda Koren,et al.  Factorization meets the neighborhood: a multifaceted collaborative filtering model , 2008, KDD.

[11]  Yanjun Qi,et al.  Supervised semantic indexing , 2009, ECIR.

[12]  Sina Jafarpour,et al.  Filter, Rank, and Transfer the Knowledge: Learning to Chat , 2010 .

[13]  Roberto Turrin,et al.  Performance of recommender algorithms on top-n recommendation tasks , 2010, RecSys '10.

[14]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Alan Ritter,et al.  Data-Driven Response Generation in Social Media , 2011, EMNLP.

[16]  Yehuda Koren,et al.  Advances in Collaborative Filtering , 2011, Recommender Systems Handbook.

[17]  Marie-Francine Moens,et al.  A survey on question answering technology from an information retrieval perspective , 2011, Inf. Sci..

[18]  Geoffrey Zweig,et al.  Context dependent recurrent neural network language model , 2012, 2012 IEEE Spoken Language Technology Workshop (SLT).

[19]  Claire Gardent,et al.  An End-to-End Evaluation of Two Situated Dialog Systems , 2012, SIGDIAL Conference.

[20]  Antoine Raux,et al.  The Dialog State Tracking Challenge , 2013, SIGDIAL Conference.

[21]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[22]  Andrew Chou,et al.  Semantic Parsing on Freebase from Question-Answer Pairs , 2013, EMNLP.

[23]  Eunsol Choi,et al.  Scaling Semantic Parsers with On-the-Fly Ontology Matching , 2013, EMNLP.

[24]  Jason Weston,et al.  Learning to rank recommendations with the k-order statistic loss , 2013, RecSys.

[25]  Matthew Henderson,et al.  The Second Dialog State Tracking Challenge , 2014, SIGDIAL Conference.

[26]  Jason Weston,et al.  Question Answering with Subgraph Embeddings , 2014, EMNLP.

[27]  Jonathan Berant,et al.  Semantic Parsing via Paraphrasing , 2014, ACL.

[28]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[29]  Peter Z. Yeh,et al.  An end-to-end dialog system for TV program discovery , 2014, 2014 IEEE Spoken Language Technology Workshop (SLT).

[30]  Oren Etzioni,et al.  Open question answering over curated and extracted knowledge bases , 2014, KDD.

[31]  Jason Weston,et al.  End-To-End Memory Networks , 2015, NIPS.

[32]  Matthew Henderson,et al.  Machine Learning for Dialog State Tracking: A Review , 2015 .

[33]  Jason Weston,et al.  A Neural Attention Model for Abstractive Sentence Summarization , 2015, EMNLP.

[34]  Jianfeng Gao,et al.  A Neural Network Approach to Context-Sensitive Generation of Conversational Responses , 2015, NAACL.

[35]  Jason Weston,et al.  Large-scale Simple Question Answering with Memory Networks , 2015, ArXiv.

[36]  Jason Weston,et al.  Memory Networks , 2014, ICLR.

[37]  Quoc V. Le,et al.  A Neural Conversational Model , 2015, ArXiv.

[38]  Joelle Pineau,et al.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems , 2015, SIGDIAL Conference.

[39]  Hang Li,et al.  Neural Responding Machine for Short-Text Conversation , 2015, ACL.

[40]  Regina Barzilay,et al.  Language Understanding for Text-based Games using Deep Reinforcement Learning , 2015, EMNLP.

[41]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[42]  Tomas Mikolov,et al.  Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets , 2015, NIPS.

[43]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[44]  Jason Weston,et al.  The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations , 2015, ICLR.

[45]  Xinyun Chen.  Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks , 2016 .

[46]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[47]  Omer Levy,et al.  Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS , 2018 .

[48]  Sashank J. Reddi,et al.  On the Convergence of Adam and Beyond , 2018, ICLR.