Leveraging Graph to Improve Abstractive Multi-Document Summarization

Graphs that capture relations between textual units have great benefits for detecting salient information from multiple documents and generating overall coherent summaries. In this paper, we develop a neural abstractive multi-document summarization (MDS) model which can leverage well-known graph representations of documents such as similarity graph and discourse graph, to more effectively process multiple input documents and produce abstractive summaries. Our model utilizes graphs to encode documents in order to capture cross-document relations, which is crucial to summarizing long documents. Our model can also take advantage of graphs to guide the summary generation process, which is beneficial for generating coherent and concise summaries. Furthermore, pre-trained language models can be easily combined with our model, which further improve the summarization performance significantly. Empirical results on the WikiSum and MultiNews dataset show that the proposed architecture brings substantial improvements over several strong baselines.

[1]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[2]  Zhi-Hong Deng,et al.  An Unsupervised Multi-Document Summarization Framework Based on Neural Document Model , 2016, COLING.

[3]  Hao Tian,et al.  ERNIE 2.0: A Continual Pre-training Framework for Language Understanding , 2019, AAAI.

[4]  Dragomir R. Radev,et al.  LexRank: Graph-based Lexical Centrality as Salience in Text Summarization , 2004, J. Artif. Intell. Res..

[5]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[6]  Claire Cardie,et al.  Towards Dynamic Computation Graphs via Sparse Latent Structure , 2018, EMNLP.

[7]  Fei Liu,et al.  Abstract Meaning Representation for Multi-Document Summarization , 2018, COLING.

[8]  Kathleen R. McKeown,et al.  Information fusion for multidocument summarization: paraphrasing and generation , 2003 .

[9]  Piji Li,et al.  Abstractive Multi-Document Summarization via Phrase Selection and Merging , 2015, ACL.

[10]  Zhe Hu,et al.  An Entity-Driven Framework for Abstractive Summarization , 2019, EMNLP.

[11]  Shashi Narayan,et al.  Leveraging Pre-trained Checkpoints for Sequence Generation Tasks , 2019, Transactions of the Association for Computational Linguistics.

[12]  Michael Elhadad,et al.  Query Focused Abstractive Summarization: Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints into seq2seq Models , 2018, ArXiv.

[13]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[14]  Alexander M. Rush,et al.  Bottom-Up Abstractive Summarization , 2018, EMNLP.

[15]  Rui Zhang,et al.  Graph-based Neural Multi-Document Summarization , 2017, CoNLL.

[16]  Hai Zhuge,et al.  Abstractive Multi-Document Summarization Based on Semantic Link Network , 2021, IEEE Transactions on Knowledge and Data Engineering.

[17]  Xinyan Xiao,et al.  Improving Neural Abstractive Document Summarization with Structural Regularization , 2018, EMNLP.

[18]  Yejin Choi,et al.  Deep Communicating Agents for Abstractive Summarization , 2018, NAACL.

[19]  Mirella Lapata,et al.  Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization , 2018, EMNLP.

[20]  Fei Liu,et al.  Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization , 2018, EMNLP.

[21]  Dragomir R. Radev,et al.  Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model , 2019, ACL.

[22]  Mirella Lapata,et al.  Text Summarization with Pretrained Encoders , 2019, EMNLP.

[23]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[24]  John Brown.  Modelling Events through Memory-based , Open-IE Patterns for Abstractive Summarization , 2014 .

[25]  Prasenjit Mitra,et al.  Multi-Document Abstractive Summarization Using ILP Based Multi-Sentence Compression , 2015, IJCAI.

[26]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[27]  Wei Li.  Abstractive Multi-document Summarization with Semantic Information Extraction , 2015, EMNLP.

[28]  Marc Brockschmidt,et al.  Structured Neural Summarization , 2018, ICLR.

[29]  Weijia Jia,et al.  Interactive Variance Attention based Online Spoiler Detection for Time-Sync Comments , 2019, CIKM.

[30]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[31]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[32]  Guy Lapalme,et al.  Framework for Abstractive Summarization using Text-to-Text Generation , 2011, Monolingual@ACL.

[33]  Xiaojun Wan,et al.  Towards a Neural Network Approach to Abstractive Multi-Document Summarization , 2018, ArXiv.

[34]  Michael Strube,et al.  Sentence Fusion via Dependency Graph Compression , 2008, EMNLP.

[35]  Dan Klein,et al.  Jointly Learning to Extract and Compress , 2011, ACL.

[36]  Chin-Yew Lin,et al.  Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics , 2004, ACL.

[37]  Xinyan Xiao,et al.  Improving Neural Abstractive Document Summarization with Explicit Information Selection Modeling , 2018, EMNLP.

[38]  Rada Mihalcea,et al.  TextRank: Bringing Order into Text , 2004, EMNLP.

[39]  Lukasz Kaiser,et al.  Generating Wikipedia by Summarizing Long Sequences , 2018, ICLR.

[40]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[41]  Mirella Lapata,et al.  Generating Summaries with Topic Templates and Structured Convolutional Decoders , 2019, ACL.

[42]  Eric Chu,et al.  MeanSum: A Neural Model for Unsupervised Multi-Document Abstractive Summarization , 2018, ICML.

[43]  Xiaojun Wan,et al.  An Exploration of Document Impact on Graph-Based Multi-Document Summarization , 2008, EMNLP.

[44]  Richard Socher,et al.  A Deep Reinforced Model for Abstractive Summarization , 2017, ICLR.

[45]  Wenjie Li,et al.  Mutually Reinforced Manifold-Ranking Based Relevance Propagation Model for Query-Focused Multi-Document Summarization , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[46]  Sergey Edunov,et al.  Pre-trained language model representations for language generation , 2019, NAACL.

[47]  Oren Etzioni,et al.  Towards Coherent Multi-Document Summarization , 2013, NAACL.

[48]  Mirella Lapata,et al.  Sentence Compression as Tree Transduction , 2009, J. Artif. Intell. Res..

[49]  Mirella Lapata,et al.  Hierarchical Transformers for Multi-Document Summarization , 2019, ACL.

[50]  Claire Cardie,et al.  Domain-Independent Abstract Generation for Focused Meeting Summarization , 2013, ACL.

[51]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[53]  Dragomir R. Radev.  A Common Theory of Information Fusion from Multiple Text Sources Step One: Cross-Document Structure , 2000, SIGDIAL Workshop.

[54]  Xiaodong Liu,et al.  Unified Language Model Pre-training for Natural Language Understanding and Generation , 2019, NeurIPS.

[55]  Claire Gardent,et al.  Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs , 2019, EMNLP.

[56]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[57]  Regina Barzilay,et al.  Sentence Fusion for Multidocument News Summarization , 2005, CL.

[58]  Jiebo Luo,et al.  Graph-based Neural Sentence Ordering , 2019, IJCAI.

[59]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.