Generative Text Summary Based on Enhanced Semantic Attention and Gain-Benefit Gate

Generative text summary is an important branch of natural language processing. Aiming at the problems of insufficient use of semantic information, insufficient summary precision and the problem of semantics-loss in the current generated text summary method, an enhanced semantic model is proposed based on dual-encoder, which can provide richer semantic information for sequence-to-sequence architecture through dual-encoder. The enhanced attention architecture with dual-channel semantics is optimized, and the empirical distribution and Gain-Benefit gate are built for decoding. In addition, the position embedding and word embedding are merged into the word embedding technology, and the TF-IDF(term frequency-inverse document frequency), part of speech, key score are added to word’s feature. Meanwhile, the optimal dimension of word embedding is optimized. This paper aims to optimize the traditional sequence mapping and word feature representation, enhance the model’s semantic understanding, and improve the quality of the summary. The LCSTS and SOGOU datasets are used to validate proposed method. The experimental results show that the proposed method can improve the performance of the ROUGE evaluation system by 10–13 percentage points compared with other listed algorithms. We can observe that the semantic understanding of the text summaries is more accurate and the generation effect is better, which has a better application prospect.

[1]  Guocai Yang,et al.  A Reading Comprehension Style Question Answering Model Based On Attention Mechanism , 2018, 2018 IEEE 29th International Conference on Application-specific Systems, Architectures and Processors (ASAP).

[2]  Bowen Zhou,et al.  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond , 2016, CoNLL.

[3]  Haijun Zhang,et al.  Understanding Subtitles by Character-Level Sequence-to-Sequence Learning , 2017, IEEE Transactions on Industrial Informatics.

[4]  GambhirMahak,et al.  Recent automatic text summarization techniques , 2017 .

[5]  Sanjeev Arora,et al.  A Simple but Tough-to-Beat Baseline for Sentence Embeddings , 2017, ICLR.

[6]  Jason Weston,et al.  A Neural Attention Model for Abstractive Sentence Summarization , 2015, EMNLP.

[7]  Yann Dauphin,et al.  Convolutional Sequence to Sequence Learning , 2017, ICML.

[8]  Qingcai Chen,et al.  LCSTS: A Large Scale Chinese Short Text Summarization Dataset , 2015, EMNLP.

[9]  Guanqin Chen,et al.  Chinese Short Text Summary Generation Model Integrating Multi-Level Semantic Information , 2018 .

[10]  Omer Levy,et al.  Linguistic Regularities in Sparse and Explicit Word Representations , 2014, CoNLL.

[11]  Jiajun Zhang,et al.  Synchronous Bidirectional Neural Machine Translation , 2019, TACL.

[12]  Yidong Chen,et al.  Deep Semantic Role Labeling with Self-Attention , 2017, AAAI.

[13]  Zi Yin,et al.  On the Dimensionality of Word Embedding , 2018, NeurIPS.

[14]  Christos Mousas,et al.  Generative Adversarial Network with Policy Gradient for Text Summarization , 2019, 2019 IEEE 13th International Conference on Semantic Computing (ICSC).

[15]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[16]  Konstantin Lopyrev,et al.  Generating News Headlines with Recurrent Neural Networks , 2015, ArXiv.

[17]  Lemao Liu,et al.  A Neural Approach to Source Dependence Based Context Model for Statistical Machine Translation , 2018, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[18]  Eduard H. Hovy,et al.  Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[19]  Yang Liu,et al.  Modeling Coverage for Neural Machine Translation , 2016, ACL.

[20]  Ji Wang,et al.  Pretraining-Based Natural Language Generation for Text Summarization , 2019, CoNLL.

[21]  Pan Wang,et al.  MS-Pointer Network: Abstractive Text Summary Based on Multi-Head Self-Attention , 2019, IEEE Access.

[22]  Timothy Baldwin,et al.  An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation , 2016, Rep4NLP@ACL.

[23]  Quoc V. Le,et al.  QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension , 2018, ICLR.

[24]  Holger Schwenk,et al.  Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond , 2018, Transactions of the Association for Computational Linguistics.

[25]  Athena Vakali,et al.  MultiSpot: Spotting Sentiments with Semantic Aware Multilevel Cascaded Analysis , 2015, DaWaK.

[26]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.