The Impact of Data Challenges on Intent Detection and Slot Filling for the Home Assistant Scenario

Natural Language Understanding (NLU) is currently a very high-interest domain to both academia and the commercial environment, due in the largest part to the recent increased popularity of conversational systems. In this paper we focus on the home assistant application context and identify a set of language and data-related challenges that can occur in such a scenario, such as: distribution shift, missing information and class imbalance. We systematically generate datasets in the Romanian language that model these data complexities and further investigate how well two of the most prominent tools – Wit.ai and Rasa NLU – solve the tasks of intent detection and slot filling, given the considered data complexities. We perform a thorough analysis of the errors produced by both tools, and provide the most probable justification for their occurrence. We found that both tools focus extensively on the verb for identifying intents, and that antonyms, class-imbalance and certain small variations in formulation greatly impact intent and slot identification. This opens new research directions to directly address these shortcomings.

[1]  Rodica Potolea,et al.  Imbalanced Classification Problems: Systematic Study, Issues and Best Practices , 2011, ICEIS.

[2]  Bhuvana Ramabhadran,et al.  Deep belief nets for natural language call-routing , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Mitchell P. Marcus,et al.  Text Chunking using Transformation-Based Learning , 1995, VLC@ACL.

[4]  Andreas Niekler,et al.  Document Retrieval for Large Scale Content Analysis using Contextualized Dictionaries , 2014, ArXiv.

[5]  Philip S. Yu,et al.  Joint Slot Filling and Intent Detection via Capsule Neural Networks , 2018, ACL.

[6]  Mayank Agarwal,et al.  Machine Translation: A Literature Review , 2018, ArXiv.

[7]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[8]  Bing Liu,et al.  Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling , 2016, INTERSPEECH.

[9]  Philipp Koehn,et al.  Abstract Meaning Representation for Sembanking , 2013, LAW@ACL.

[10]  Ari Rappoport,et al.  UCCA: A Semantics-based Grammatical Annotation Scheme , 2013, IWCS.

[11]  Philip S. Yu,et al.  Zero-shot User Intent Detection via Capsule Neural Networks , 2018, EMNLP.

[12]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[13]  Bing Liu,et al.  Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding , 2015 .

[14]  Lihong Li,et al.  Neural Approaches to Conversational AI , 2019, Found. Trends Inf. Retr..

[15]  Joakim Nivre,et al.  Universal Dependency Annotation for Multilingual Parsing , 2013, ACL.

[16]  Gary M. Weiss.  Mining with rarity: a unifying framework , 2004, SKDD.

[17]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[18]  Ruhi Sarikaya,et al.  Convolutional neural network based triangular CRF for joint intent detection and slot filling , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.