Gated Graph Neural Attention Networks for abstractive summarization

Abstract Sequence to sequence (Seq2Seq) model for abstractive summarization have aroused widely attention due to their powerful ability to represent sequence. However, the sequence structured data is a simple format, which cannot describe the complexity of graphs and may lead to ambiguous, and hurt the performance of summarization. In this paper, we propose a Gated Graph Neural Attention Networks (GGNANs) for abstractive summarization. The proposed GGNANs unified graph neural network and the celebrated Seq2seq for better encoding the full graph-structured information. We propose a graph transform method based on PMI, self-connection, forward-connection and backward-connection to better combine graph-structured information and the sequence-structured information. Extensive experimental results on the LCSTS and Gigaword show that our proposed model outperforms most of strong baseline models.

[1]  Lukasz Kaiser,et al.  Sentence Compression by Deletion with LSTMs , 2015, EMNLP.

[2]  S. K. Gupta,et al.  Abstractive summarization: An overview of the state of the art , 2019, Expert Syst. Appl..

[3]  Jindrich Libovický,et al.  Attention Strategies for Multi-Source Sequence-to-Sequence Learning , 2017, ACL.

[4]  Gholamreza Haffari,et al.  Graph-to-Sequence Learning using Gated Graph Neural Networks , 2018, ACL.

[5]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[6]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[7]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[8]  Piji Li,et al.  Deep Recurrent Generative Decoder for Abstractive Text Summarization , 2017, EMNLP.

[9]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[10]  Yansong Feng,et al.  Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks , 2018, ArXiv.

[11]  Li Wang,et al.  A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization , 2018, IJCAI.

[12]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[13]  Hang Li,et al.  “ Tony ” DNN Embedding for “ Tony ” Selective Read for “ Tony ” ( a ) Attention-based Encoder-Decoder ( RNNSearch ) ( c ) State Update s 4 SourceVocabulary Softmax Prob , 2016 .

[14]  Mirella Lapata,et al.  Neural Summarization by Extracting Sentences and Words , 2016, ACL.

[15]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[16]  Zhen-Hua Ling,et al.  Distraction-based neural networks for modeling documents , 2016, IJCAI 2016.

[17]  Kathleen McKeown,et al.  Cut and Paste Based Text Summarization , 2000, ANLP.

[18]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[19]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[20]  Alexander M. Rush,et al.  Coarse-to-Fine Attention Models for Document Summarization , 2017, NFiS@EMNLP.

[21]  Jason Weston,et al.  A Neural Attention Model for Abstractive Sentence Summarization , 2015, EMNLP.

[22]  Xu Sun,et al.  Autoencoder as Assistant Supervisor: Improving Text Representation for Chinese Social Media Text Summarization , 2018, ACL.

[23]  Alexander M. Rush,et al.  Abstractive Sentence Summarization with Attentive Recurrent Neural Networks , 2016, NAACL.

[24]  Bowen Zhou,et al.  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond , 2016, CoNLL.

[25]  Daniel Marcu,et al.  Statistics-Based Summarization - Step One: Sentence Compression , 2000, AAAI/IAAI.

[26]  Xu Sun,et al.  Global Encoding for Abstractive Summarization , 2018, ACL.

[27]  Xu Sun,et al.  Improving Semantic Relevance for Sequence-to-Sequence Learning of Chinese Social Media Text Summarization , 2017, ACL.

[28]  Ani Nenkova,et al.  A Survey of Text Summarization Techniques , 2012, Mining Text Data.

[29]  Elizabeth D. Liddy,et al.  Advances in Automatic Text Summarization , 2001, Information Retrieval.

[30]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[31]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[32]  Alexander M. Rush,et al.  Sequence-to-Sequence Learning as Beam-Search Optimization , 2016, EMNLP.

[33]  Ming Zhou,et al.  Selective Encoding for Abstractive Sentence Summarization , 2017, ACL.

[34]  Yuan Luo,et al.  Graph Convolutional Networks for Text Classification , 2018, AAAI.

[35]  Qingcai Chen,et al.  LCSTS: A Large Scale Chinese Short Text Summarization Dataset , 2015, EMNLP.

[36]  Junping Du,et al.  Abstractive social media text summarization using selective reinforced Seq2Seq attention model , 2020, Neurocomputing.

[37]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[38]  Yue Zhang,et al.  A Graph-to-Sequence Model for AMR-to-Text Generation , 2018, ACL.

[39]  Khalil Sima'an,et al.  Graph Convolutional Encoders for Syntax-aware Neural Machine Translation , 2017, EMNLP.