A Measure of Similarity in Textual Data Using Spearman's Rank Correlation Coefficient

In the last decade, many diverse advances have occurred in the field of information extraction from data. Information extraction in its simplest form takes place in computing environments, where structured data can be extracted through a series of queries. The continuous expansion of quantities of data have therefore provided an opportunity for knowledge extraction (KE) from a textual document (TD). A typical problem of this kind is the extraction of common characteristics and knowledge from a group of TDs, with the possibility to group such similar TDs in a process known as clustering. In this paper we present a technique for such KE among a group of TDs related to the common characteristics and meaning of their content. Our technique is based on the Spearman's Rank Correlation Coefficient (SRCC), for which the conducted experiments have proven to be comprehensive measure to achieve a high-quality KE.

[1]  Martha Palmer,et al.  Verb Semantics and Lexical Selection , 1994, ACL.

[2]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[3]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[4]  Martin Ester,et al.  Frequent term-based text clustering , 2002, KDD.

[5]  K. Pearson.  VII. Note on regression and inheritance in the case of two parents , 1895, Proceedings of the Royal Society of London.

[6]  Carlo Strapparava,et al.  Corpus-based and Knowledge-based Measures of Text Semantic Similarity , 2006, AAAI.

[7]  Romaric Besançon,et al.  Text Mining, knowledge extraction from unstructured textual data , 1998 .

[8]  Christiane Fellbaum,et al.  Combining Local Context and Wordnet Similarity for Word Sense Identification , 1998 .

[9]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[10]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[11]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[12]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[13]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[14]  Michael E. Lesk,et al.  Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone , 1986, SIGDOC '86.