Bidirectional LSTM Joint Model for Intent Classification and Named Entity Recognition in Natural Language Understanding

The aim of this paper is to present a Simple LSTM - Bidirectional LSTM in a joint model framework, for Intent Classification and Named Entity Recognition (NER) tasks. Both the models are approached as a classification task. This paper discuss the comparison of single models and joint models in the respective tasks, a data augmentation algorithm and how the joint model framework helped in learning a poor performing NER model in by adding learned weights from well performing Intent Classification model in their respective tasks. The experiment in the paper shows that there is approximately 44% improvement in performance of NER model when in joint model compared to when tested as independent model.

[1]  Gary Geunbae Lee,et al.  Triangular-Chain Conditional Random Fields , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[2]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Geoffrey Zweig,et al.  Joint semantic utterance classification and slot filling with recursive neural networks , 2014, 2014 IEEE Spoken Language Technology Workshop (SLT).

[4]  Bing Liu,et al.  Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks , 2016, SIGDIAL Conference.

[5]  Frédéric Béchet,et al.  Joint Syntactic and Semantic Analysis with a Multitask Deep Learning Framework for Spoken Language Understanding , 2016, INTERSPEECH.

[6]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[7]  Geoffrey Zweig,et al.  Recurrent conditional random field for language understanding , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Dan Klein,et al.  Named Entity Recognition with Character-Level Models , 2003, CoNLL.

[9]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[10]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Ting Liu,et al.  Document Modeling with Gated Recurrent Neural Network for Sentiment Classification , 2015, EMNLP.

[12]  Bowen Zhou,et al.  Jointly Trained Sequential Labeling and Classification by Sparse Attention Neural Networks , 2017, INTERSPEECH.

[13]  Sungjin Lee,et al.  ONENET: Joint domain, intent, slot prediction for spoken language understanding , 2017, 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).

[14]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[16]  Wojciech Czarnecki,et al.  On Loss Functions for Deep Neural Networks in Classification , 2017, ArXiv.

[17]  Ruhi Sarikaya,et al.  Convolutional neural network based triangular CRF for joint intent detection and slot filling , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.