Enriching and Controlling Global Semantics for Text Summarization

Recently, Transformer-based models have been proven effective in the abstractive summarization task by creating fluent and informative summaries. Nevertheless, these models still suffer from the short-range dependency problem, causing them to produce summaries that miss the key points of document. In this paper, we attempt to address this issue by introducing a neural topic model empowered with normalizing flow to capture the global semantics of the document, which are then integrated into the summarization model. In addition, to avoid the overwhelming effect of global semantics on contextualized representation, we introduce a mechanism to control the amount of global semantics supplied to the text generation module. Our method outperforms state-of-the-art summarization models on five common text summarization datasets, namely CNN/DailyMail, XSum, Reddit TIFU, arXiv, and PubMed.

[1]  Ling Fan,et al.  Topic-Aware Abstractive Text Summarization , 2020, ArXiv.

[2]  Nancy F. Chen,et al.  Topic-Aware Pointer-Generator Networks for Summarizing Spoken Conversations , 2019, 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).

[3]  Richard Socher,et al.  A Deep Reinforced Model for Abstractive Summarization , 2017, ICLR.

[4]  Phil Blunsom,et al.  Discovering Discrete Latent Topics with Neural Variational Inference , 2017, ICML.

[5]  Yang Liu,et al.  Fine-tune BERT for Extractive Summarization , 2019, ArXiv.

[6]  Wen Xiao,et al.  Systematically Exploring Redundancy Reduction in Summarizing Long Documents , 2020, AACL.

[7]  Xiaodong Liu,et al.  Unified Language Model Pre-training for Natural Language Understanding and Generation , 2019, NeurIPS.

[8]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[9]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[10]  Yao Zhao,et al.  PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization , 2020, ICML.

[11]  Alexander M. Rush,et al.  Bottom-Up Abstractive Summarization , 2018, EMNLP.

[12]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[13]  Qing Li,et al.  Incorporating Effective Global Information via Adaptive Gate Attention for Text Classification , 2020, ArXiv.

[14]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[15]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[16]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[17]  Melissa Ailem,et al.  Topic Augmented Generator for Abstractive Summarization , 2019, ArXiv.

[18]  Ramesh Nallapati,et al.  Coherence-Aware Neural Topic Modeling , 2018, EMNLP.

[19]  Ming Zhou,et al.  HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization , 2019, ACL.

[20]  Mirella Lapata,et al.  Text Summarization with Pretrained Encoders , 2019, EMNLP.

[21]  Claire Cardie,et al.  Intrinsic Evaluation of Summarization Datasets , 2020, EMNLP.

[22]  Ming Zhou,et al.  Pre-training for Abstractive Document Summarization by Reinstating Source Text , 2020, EMNLP.

[23]  Zhenglu Yang,et al.  Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism , 2020, AAAI.

[24]  Franck Dernoncourt,et al.  A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents , 2018, NAACL.

[25]  Alex Wang,et al.  What do you learn from context? Probing for sentence structure in contextualized word representations , 2019, ICLR.

[26]  Li Wang,et al.  A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization , 2018, IJCAI.

[27]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[28]  Daniele Pighin,et al.  Stepwise Extractive Summarization and Planning with Structured Transformers , 2020, EMNLP.

[29]  Regina Barzilay,et al.  Capturing Greater Context for Question Generation , 2019, AAAI.

[30]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[31]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[32]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[33]  Pengfei Liu,et al.  Extractive Summarization as Text Matching , 2020, ACL.

[34]  Bo Chen,et al.  Friendly Topic Assistant for Transformer Based Abstractive Summarization , 2020, EMNLP.

[35]  Jing Li,et al.  Topic Memory Networks for Short Text Classification , 2018, EMNLP.

[36]  Mirella Lapata,et al.  Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization , 2018, EMNLP.

[37]  Ruipeng Jia,et al.  Neural Extractive Summarization with Hierarchical Attentive Heterogeneous Graph Network , 2020, EMNLP.

[38]  Sandeep Subramanian,et al.  On Extractive and Abstractive Neural Document Summarization with Transformer Language Models , 2020, EMNLP.

[39]  Xu Tan,et al.  MASS: Masked Sequence to Sequence Pre-training for Language Generation , 2019, ICML.

[40]  Diyi Yang,et al.  Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization , 2020, EMNLP.

[41]  Gunhee Kim,et al.  Abstractive Summarization of Reddit Posts with Multi-level Memory Networks , 2018, NAACL.

[42]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.