A Fast and Robust BERT-based Dialogue State Tracker for Schema Guided Dialogue Dataset

Dialog State Tracking (DST) is one of the most crucial modules for goal-oriented dialogue systems. In this paper, we introduce FastSGT (Fast Schema Guided Tracker), a fast and robust BERT-based model for state tracking in goal-oriented dialogue systems. The proposed model is designed for the Schema-Guided Dialogue (SGD) dataset which contains natural language descriptions for all the entities including user intents, services, and slots. The model incorporates two carry-over procedures for handling the extraction of the values not explicitly mentioned in the current user utterance. It also uses multi-head attention projections in some of the decoders to have a better modelling of the encoder outputs. In the conducted experiments we compared FastSGT to the baseline model for the SGD dataset. Our model keeps the efficiency in terms of computational and memory consumption while improving the accuracy significantly. Additionally, we present ablation studies measuring the impact of different parts of the model on its performance. We also show the effectiveness of data augmentation for improving the accuracy without increasing the amount of computational resources.

[1]  Zheng Zhang,et al.  Recent advances and challenges in task-oriented dialog systems , 2020, Science China Technological Sciences.

[2]  Richard Socher,et al.  Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems , 2019, ACL.

[3]  Stefan Ultes,et al.  MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling , 2018, EMNLP.

[4]  Raghav Gupta,et al.  Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset , 2020, AAAI.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[7]  Gökhan Tür,et al.  Building a Conversational Agent Overnight with Dialogue Self-Play , 2018, ArXiv.

[8]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[9]  Hao Wu,et al.  Mixed Precision Training , 2017, ICLR.

[10]  Gyuwan Kim,et al.  Efficient Dialogue State Tracking by Selectively Overwriting Memory , 2020, ACL.

[11]  Quan Liu,et al.  Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking , 2020, ArXiv.

[12]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[13]  Haoqi Xiong,et al.  The SPPD System for Schema Guided Dialogue State Tracking Challenge , 2020, ArXiv.

[14]  Mikhail Burtsev,et al.  Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker , 2020, ArXiv.

[15]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[16]  Bernardo Magnini,et al.  Domain-Aware Dialogue State Tracker for Multi-Domain Dialogue Systems , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[17]  Kevin Gimpel,et al.  Gaussian Error Linear Units (GELUs) , 2016 .

[18]  Xuan Li,et al.  An End-to-End Dialogue State Tracking System with Machine Reading Comprehension and Wide & Deep Classification , 2019, ArXiv.