Segmenting broadcast news streams using lexical chains

In this paper we propose a course-grained NLP approach to text segmentation based on the analysis of lexical cohesion within text. Most work in this area has focused on the discovery of textual units that discuss subtopic structure within documents. In contrast our segmentation task requires the discovery of topical units of text i.e. distinct news stories from broadcast news programmes. Our system SeLeCT first builds a set of lexical chains, in order to model the discourse structure of the text. A boundary detector is then used to search for breaking points in this structure indicated by patterns of cohesive strength and weakness within the text. We evaluate this technique on a test set of concatenated CNN news story transcripts and compare it with an established statistical approach to segmentation called TextTiling.

[1]  Marti A. Hearst.  Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages , 1997, CL.

[2]  Regina Barzilay,et al.  Using Lexical Chains for Text Summarization , 1997 .

[3]  Graeme Hirst,et al.  Automatically generating hypertext by computing semantic similarity , 1997 .

[4]  Graeme Hirst,et al.  Lexical Cohesion Computed by Thesaural relations as an indicator of the structure of text , 1991, CL.

[5]  Mitchell P. Marcus,et al.  Topic segmentation: algorithms and applications , 1998 .

[6]  W. Bruce Croft,et al.  Text Segmentation by Topic , 1997, ECDL.

[7]  Hinrich Schütze,et al.  Ambiguity resolution in language learning , 1997 .

[8]  John D. Lafferty,et al.  Text Segmentation Using Exponential Models , 1997, EMNLP.

[9]  Jeffrey C. Reynar.  An Automatic Method of Finding Topic Boundaries , 1994, ACL.

[10]  Rebecca J. Passonneau,et al.  Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues , 1993, ACL.

[11]  Yiming Yang,et al.  Topic Detection and Tracking Pilot Study Final Report , 1998 .

[12]  Okumura Manabu,et al.  Word Sense Disambiguation and Text Segmentation Based on Lexical Cohesion , 1994, COLING.

[13]  Manabu Okumura,et al.  Word Sense Disambiguation and Text Segmentation Based on Lexical Cohesion , 1994, COLING.

[14]  Stefan Kaufmann.  Second‐Order Cohesion , 2000, Comput. Intell..

[15]  Alan F. Smeaton.  Content-based access to digital video: the Físchlár system and the TREC video track , 2001 .

[16]  Hideki Kozima,et al.  Text Segmentation Based on Similarity between Words , 1993, ACL.