Induction of Word and Phrase Alignments for Automatic Document Summarization

Current research in automatic single-document summarization is dominated by two effective, yet nave approaches: summarization by sentence extraction and headline generation via bagof-words models. While successful in some tasks, neither of these models is able to adequately capture the large set of linguistic devices utilized by humans when they produce summaries. One possible explanation for the widespread use of these models is that good techniques have been developed to extract appropriate training data for them from existing document/abstract and document/ headline corpora. We believe that future progress in automatic summarization will be driven both by the development of more sophisticated, linguistically informed models, as well as a more effective leveraging of document/abstract corpora. In order to open the doors to simultaneously achieving both of these goals, we have developed techniques for automatically producing word-to-word and phrase-to-phrase alignments between documents and their human-written abstracts. These alignments make explicit the correspondences that exist in such document/abstract pairs and create a potentially rich data source from which complex summarization algorithms may learn. This paper describes experiments we have carried out to analyze the ability of humans to perform such alignments, and based on these analyses, we describe experiments for creating them automatically. Our model for the alignment task is based on an extension of the standard hidden Markov model and learns to create alignments in a completely unsupervised fashion. We describe our model in detail and present experimental results that show that our model is able to learn to reliably identify word- and phrase-level alignments in a corpus of (document, abstract) pairs.

[1]  J. Jensen.  Sur les fonctions convexes et les inégalités entre les valeurs moyennes , 1906 .

[2]  Hans Peter Luhn,et al.  The Automatic Creation of Literature Abstracts , 1958, IBM J. Res. Dev..

[3]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[4]  L. Baum,et al.  An inequality with applications to statistical estimation for probabilistic functions of Markov processes and to a model for ecology , 1967 .

[5]  H. P. Edmundson,et al.  New Methods in Automatic Extracting , 1969, JACM.

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  R. A. Boyles.  On the Convergence of the EM Algorithm , 1983 .

[8]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[9]  Mark J. F. Gales,et al.  The theory of segmental hidden Markov models , 1993 .

[10]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[11]  Chin-Hui Lee,et al.  Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains , 1994, IEEE Trans. Speech Audio Process..

[12]  Mary P. Harper,et al.  On the complexity of explicit duration HMM's , 1995, IEEE Trans. Speech Audio Process..

[13]  Francine Chen,et al.  A trainable document summarizer , 1995, SIGIR '95.

[14]  Hermann Ney,et al.  HMM-Based Word Alignment in Statistical Translation , 1996, COLING.

[15]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[16]  Mari Ostendorf,et al.  From HMM's to segment models: a unified view of stochastic modeling for speech recognition , 1996, IEEE Trans. Speech Audio Process..

[17]  Raman Chandrasekar,et al.  Motivations and Methods for Text Simplification , 1996, COLING.

[18]  Eugene Charniak,et al.  Statistical Parsing with a Context-Free Grammar and Word Statistics , 1997, AAAI/IAAI.

[19]  Simone Teufel,et al.  Sentence extraction as a classification task , 1997 .

[20]  Kavi Mahesh.  Hypertext Summary Extraction for Fast Document Browsing , 1997 .

[21]  Michael I. Jordan,et al.  Probabilistic Independence Networks for Hidden Markov Probability Models , 1997, Neural Computation.

[22]  Gregory Grefenstette.  Producing Intelligent Telegraphic Text Reduction to provide an Audio Scanning Service for the Blind , 1998 .

[23]  Daniel Marcu,et al.  The automatic construction of large-scale corpora for summarization research , 1999, SIGIR '99.

[24]  M. Sanderson.  Book Reviews: Advances in Automatic Text Summarization , 2000, Computational Linguistics.

[25]  Vibhu O. Mittal,et al.  Query-Relevant Summarization using FAQs , 2000, ACL.

[26]  The Theory and Practice of Discourse Parsing and Summarization , 2000 .

[27]  Hongyan Jing,et al.  Sentence Reduction for Automatic Text Summarization , 2000, ANLP.

[28]  Padhraic Smyth,et al.  Segmental Semi-Markov Models for Change-Point Detection with Applications to Semiconductor Manufactu , 2000 .

[29]  Michele Banko,et al.  Headline Generation Based on Statistical Translation , 2000, ACL.

[30]  Hermann Ney,et al.  Improved Statistical Alignment Models , 2000, ACL.

[31]  Kevin Knight,et al.  A Syntax-based Statistical Translation Model , 2001, ACL.

[32]  Hongyan Jing,et al.  Using Hidden Markov Modeling to Decompose Human-Written Summaries , 2002, Computational Linguistics.

[33]  R. Schwartz,et al.  Automatic Headline Generation for Newspaper Stories , 2002 .

[34]  Daniel Marcu,et al.  A Noisy-Channel Model for Document Compression , 2002, ACL.

[35]  Daniel Marcu,et al.  Summarization beyond sentence extraction: A probabilistic approach to sentence compression , 2002, Artif. Intell..

[36]  Jason Eisner,et al.  Learning Non-Isomorphic Tree Mappings for Machine Translation , 2003, ACL.

[37]  Regina Barzilay,et al.  Sentence Alignment for Monolingual Comparable Corpora , 2003, EMNLP.

[38]  Daniel Gildea,et al.  Loosely Tree-Based Alignment for Machine Translation , 2003, ACL.

[39]  Hermann Ney,et al.  A Systematic Comparison of Various Statistical Alignment Models , 2003, CL.

[40]  Chris Quirk,et al.  Monolingual Machine Translation for Paraphrase Generation , 2004, EMNLP.

[41]  Richard M. Schwartz,et al.  BBN/UMD at DUC-2004: Topiary , 2004 .

[42]  Yücel Altunbasak,et al.  Protein secondary structure prediction with semi Markov HMMs , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[43]  D. Marcu,et al.  A Tree-Position Kernel for Document Compression , 2004 .