Using word sense discrimination on historic document collections

Word sense discrimination is the first, important step towards automatic detection of language evolution within large, historic document collections. By comparing the found word senses over time, we can reveal and use important information that will improve understanding and accessibility of a digital archive. Algorithms for word sense discrimination have been developed while keeping today's language in mind and have thus been evaluated on well selected, modern datasets. The quality of the word senses found in the discrimination step has a large impact on the detection of language evolution. Therefore, as a first step, we verify that word sense discrimination can successfully be applied to digitized historic documents and that the results correctly correspond to word senses. Because accessibility of digitized historic collections is influenced also by the quality of the optical character recognition (OCR), as a second step we investigate the effects of OCR errors on word sense discrimination results. All evaluations in this paper are performed on The Times Archive, a collection of newspaper articles from 1785 - 1985.

[1]  B. Dorow.  A graph model for words and their meanings , 2006 .

[2]  A. Barabasi,et al.  Quantifying social group evolution , 2007, Nature.

[3]  Helmut Schmidt,et al.  Probabilistic part-of-speech tagging using decision trees , 1994 .

[4]  Philip Resnik,et al.  Using Information Content to Evaluate Semantic Similarity in a Taxonomy , 1995, IJCAI.

[5]  Dekang Lin,et al.  Automatic Retrieval and Clustering of Similar Words , 1998, ACL.

[6]  Ari Rappoport,et al.  Efficient Unsupervised Discovery of Word Categories Using Symmetric Patterns and High Frequency Words , 2006, ACL.

[7]  Martin Suter,et al.  Small World , 2002 .

[8]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[9]  Marti A. Hearst.  Automatic Acquisition of Hyponyms from Large Text Corpora , 1992, COLING.

[10]  Hinrich Schütze,et al.  Automatic Word Sense Discrimination , 1998, Comput. Linguistics.

[11]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[12]  Nina Tahmasebi,et al.  Automatic Detection of Terminology Evolution , 2009, OTM Workshops.

[13]  Olivier Ferret,et al.  Discovering word senses from a network of lexical cooccurrences , 2004, COLING.

[14]  Patrick Pantel,et al.  Discovering word senses from text , 2002, KDD.

[15]  Ted Pedersen,et al.  Distinguishing Word Senses in Untagged Text , 1997, EMNLP.

[16]  Thomas Risse,et al.  First Results on Detecting Term Evolutions , 2009 .

[17]  Norbert Fuhr,et al.  Retrieval in text collections with historic spelling using linguistic and spelling variants , 2007, JCDL '07.

[18]  Dominic Widdows,et al.  Discovering Corpus-Specific Word Senses , 2003, EACL.