Two End-to-End Quantum-Inspired Deep Neural Networks for Text Classification

In linguistics, the uncertainty of context due to polysemy is widespread, which attracts much attention. Quantum-inspired complex word embedding based on Hilbert space plays an important role in natural language processing (NLP), which fully leverages the similarity between quantum states and word tokens. A word containing multiple meanings could correspond to a single quantum particle which may exist in several possible states, and a sentence could be analogous to the quantum system where particles interfere with each other. Motivated by quantum-inspired complex word embedding, interpretable complex-valued word embedding (ICWE) is proposed to design two end-to-end quantum-inspired deep neural networks (ICWE-QNN and CICWE-QNN representing convolutional complex-valued neural network based on ICWE) for binary text classification. They have the proven feasibility and effectiveness in the application of NLP and can solve the problem of text information loss in CE-Mix [1] model caused by neglecting the important linguistic features of text, since linguistic feature extraction is presented in our model with deep learning algorithms, in which gated recurrent unit (GRU) extracts the sequence information of sentences, attention mechanism makes the model focus on important words in sentences and convolutional layer captures the local features of projected matrix. The model ICWE-QNN can avoid random combination of word tokens and CICWE-QNN fully considers textual features of the projected matrix. Experiments conducted on five benchmarking classification datasets demonstrate our proposed models have higher accuracy than the compared traditional models including CaptionRep BOW, DictRep BOW and Paragram-Phrase, and they also have great performance on F1-score. Eespecially, CICWE-QNN model has higher accuracy than the quantum-inspired model CE-Mix as well for four datasets including SST, SUBJ, CR and MPQA. It is a meaningful and effictive exploration to design quantum-inspired deep neural networks to promote the performance of text classification.

[1]  Dawei Song,et al.  CFN: A Complex-Valued Fuzzy Network for Sarcasm Detection in Conversations , 2021, IEEE Transactions on Fuzzy Systems.

[2]  Bing Liu,et al.  Character-level text classification via convolutional neural network and gated recurrent unit , 2020, Int. J. Mach. Learn. Cybern..

[3]  S. Rahamat Basha,et al.  A Novel Summarization-based Approach for Feature Reduction Enhancing Text Classification Accuracy , 2019 .

[4]  Georgios Siolas,et al.  A transformer-based approach to irony and sarcasm detection , 2019, Neural Computing and Applications.

[5]  Philip S. Yu,et al.  Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification , 2019, IEEE Transactions on Knowledge and Data Engineering.

[6]  Massimo Melucci,et al.  CNM: An Interpretable Complex-valued Network for Matching , 2019, NAACL.

[7]  Gang Liu,et al.  Bidirectional LSTM with attention mechanism and convolutional layer for text classification , 2019, Neurocomputing.

[8]  Fangfang Li,et al.  Sentiment-Aware Word Embedding for Emotion Classification , 2019, Applied Sciences.

[9]  Andrea Esuli,et al.  Learning to Weight for Text Classification , 2019, IEEE Transactions on Knowledge and Data Engineering.

[10]  Jimmy J. Lin,et al.  End-to-End Open-Domain Question Answering with BERTserini , 2019, NAACL.

[11]  Dawei Song,et al.  Quantum-Inspired Complex Word Embedding , 2018, Rep4NLP@ACL.

[12]  Xiaofeng Zhu,et al.  Local and Global Structure Preservation for Robust Unsupervised Spectral Feature Selection , 2018, IEEE Transactions on Knowledge and Data Engineering.

[13]  Dawei Song,et al.  End-to-End Quantum-like Language Models with Application to Question Answering , 2018, AAAI.

[14]  Ming Zhou,et al.  Gated Self-Matching Networks for Reading Comprehension and Question Answering , 2017, ACL.

[15]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[16]  Xindong Wu,et al.  Learning on Big Graph: Label Inference and Regularization with Anchor Hierarchy , 2017, IEEE Transactions on Knowledge and Data Engineering.

[17]  Spyros Kotoulas,et al.  Medical Text Classification using Convolutional Neural Networks , 2017, Studies in health technology and informatics.

[18]  Wenpeng Yin,et al.  Comparative Study of CNN and RNN for Natural Language Processing , 2017, ArXiv.

[19]  Rajib Rana,et al.  Gated Recurrent Unit (GRU) for Emotion Classification from Noisy Speech , 2016, ArXiv.

[20]  Yongdong Zhang,et al.  AC-BLSTM: Asymmetric Convolutional Bidirectional LSTM Networks for Text Classification , 2016, ArXiv.

[21]  Su-Juan Qin,et al.  Asymptotic quantum algorithm for the Toeplitz systems , 2016, Physical Review A.

[22]  Bob Coecke,et al.  Quantum Algorithms for Compositional Natural Language Processing , 2016, SLPCS@QPL.

[23]  Bowen Zhou,et al.  Improved Representation Learning for Question Answer Matching , 2016, ACL.

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Yoshua Bengio,et al.  Learning to Understand Phrases by Embedding the Dictionary , 2015, TACL.

[26]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[27]  Jun Zhao,et al.  Recurrent Convolutional Neural Networks for Text Classification , 2015, AAAI.

[28]  Ronald de Wolf,et al.  Quantum Computing: Lecture Notes , 2015, ArXiv.

[29]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[30]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[31]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[32]  Shichao Zhang,et al.  "Missing is useful": missing values in cost-sensitive decision trees , 2005, IEEE Transactions on Knowledge and Data Engineering.

[33]  B. Pang,et al.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales , 2005, ACL 2005.

[34]  Bing Liu,et al.  Mining and summarizing customer reviews , 2004, KDD.

[35]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[36]  Buyue Qian,et al.  Stacked Residual Recurrent Neural Networks With Cross-Layer Attention for Text Classification , 2020, IEEE Access.

[37]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[38]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[39]  A. Gleason.  Measures on the Closed Subspaces of a Hilbert Space , 1957 .