Lexical dynamics and conceptual change: Analyses and implications for information retrieval

One important aspect of a document’s context is the time at which it was written. We report here on analyses of formal (dissertation abstracts) and informal (discussion board postings) communications among academics within two separate disciplines. We focus on academic communications because these especially must be understood within the context of what has been said before, together with what is considered relevant and worth saying at the time of publication. All corpora include time-stamp information that allows temporal analysis of changing lexical frequencies across decades. Using techniques borrowed from time series analysis, we find distinct patterns of “rising” and “falling” bigram frequencies in both domains, and argue that this information can be exploited to improve retrieval of relevant documents.