Réseaux neuronaux profonds pour l'étiquetage de séquences (Deep Neural Networks for Sequence Labeling)

Deep Neural Networks for Sequence Labeling Since a couple of years, neural networks prove to be very effective on all NLP tasks. Recently, a variant of neural network particularly suited for sequence labeling has been proposed, which uses label embeddings. In this paper we propose a deep version of the above-mentioned variant of neural network, where several hidden layers allow to take into account separately the different types of information given as input to the model. We evaluate our variant on the same tasks as the basic version. Results show that our variant is not only more effective than the other neural models, but also that it outperforms all the other models evaluated on the same tasks, reaching state-of-the-art performances. MOTS-CLÉS : Réseaux neuronaux, apprentissage artificiel, compréhension de la parole, étiquetage de séquences.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[3]  Isabelle Tellier,et al.  Improving Recurrent Neural Networks For Sequence Labelling , 2016, ArXiv.

[4]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[5]  Alexander I. Rudnicky,et al.  Expanding the Scope of the ATIS Task: The ATIS-3 Corpus , 1994, HLT.

[6]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Jonathan G. Fiscus,et al.  A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER) , 1997, 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings.

[8]  Wei Xu,et al.  Bidirectional LSTM-CRF Models for Sequence Tagging , 2015, ArXiv.

[9]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[10]  Michael I. Jordan Serial Order: A Parallel Distributed Processing Approach , 1997 .

[11]  Hermann Ney,et al.  Comparing Stochastic Approaches to Spoken Language Understanding in Multiple Languages , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[12]  Frédéric Béchet,et al.  Results of the French Evalda-Media evaluation campaign for literal understanding , 2006, LREC.

[13]  Guillaume Gravier,et al.  A Step Beyond Local Observations with a Dialog Aware Bidirectional GRU Network for Spoken Language Understanding , 2016, INTERSPEECH.

[14]  Isabelle Tellier,et al.  Étude de réseaux de neurones récurrents pour l'étiquetage de séquences , 2016 .

[15]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[16]  Yoshua Bengio,et al.  Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding , 2013, INTERSPEECH.

[17]  Isabelle Tellier,et al.  New Recurrent Neural Network Variants for Sequence Labeling , 2016, CICLing.

[18]  Isabelle Tellier,et al.  Label-Dependencies Aware Recurrent Neural Networks , 2017, CICLing.

[19]  Eduard H. Hovy,et al.  End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF , 2016, ACL.

[20]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[21]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[23]  Guillaume Gravier,et al.  Is it time to Switch to word embedding and recurrent neural networks for spoken language understanding? , 2015, INTERSPEECH.

[24]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[25]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..