Simple Unsupervised Summarization by Contextual Matching

We propose an unsupervised method for sentence summarization using only language modeling. The approach employs two language models, one that is generic (i.e. pretrained), and the other that is specific to the target domain. We show that by using a product-of-experts criteria these are enough for maintaining continuous contextual matching while maintaining output fluency. Experiments on both abstractive and extractive sentence summarization data sets show promising results of our method without being exposed to any paired data.

[1]  Richard Socher,et al.  A Deep Reinforced Model for Abstractive Summarization , 2017, ICLR.

[2]  Jason Weston,et al.  A Neural Attention Model for Abstractive Sentence Summarization , 2015, EMNLP.

[3]  Lei Yu,et al.  The Neural Noisy Channel , 2016, ICLR.

[4]  Navdeep Jaitly,et al.  Towards Better Decoding and Language Model Integration in Sequence to Sequence Models , 2016, INTERSPEECH.

[5]  Phil Blunsom,et al.  Language as a Latent Variable: Discrete Generative Models for Sentence Compression , 2016, EMNLP.

[6]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[7]  Colin Raffel,et al.  Online and Linear-Time Attention by Enforcing Monotonic Alignments , 2017, ICML.

[8]  Jason Phang,et al.  Unsupervised Sentence Compression using Denoising Auto-Encoders , 2018, CoNLL.

[9]  Bowen Zhou,et al.  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond , 2016, CoNLL.

[10]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[11]  Emily Pitler,et al.  Methods for Sentence Compression , 2010 .

[12]  Hung-yi Lee,et al.  Learning to Encode Text as Human-Readable Summaries using Generative Adversarial Networks , 2018, EMNLP.

[13]  Yang Zhao,et al.  A Language Model based Evaluator for Sentence Compression , 2018, ACL.

[14]  Alexander M. Rush,et al.  Abstractive Sentence Summarization with Attentive Recurrent Neural Networks , 2016, NAACL.

[15]  Raphael Schumann,et al.  Unsupervised Abstractive Sentence Summarization using Length Controlled Variational Autoencoder , 2018, ArXiv.

[16]  Lukasz Kaiser,et al.  Sentence Compression by Deletion with LSTMs , 2015, EMNLP.

[17]  AurenhammerFranz.  Voronoi diagramsa survey of a fundamental geometric data structure , 1991 .

[18]  Lei Yu,et al.  Online Segment to Segment Neural Transduction , 2016, EMNLP.

[19]  Franz Aurenhammer,et al.  Voronoi diagrams—a survey of a fundamental geometric data structure , 1991, CSUR.

[20]  Yasemin Altun,et al.  Overcoming the Lack of Parallel Data in Sentence Compression , 2013, EMNLP.

[21]  Furu Wei,et al.  Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization , 2018, ACL.