Term Weighting in Short Documents for Document Categorization , Keyword Extraction and Query Expansion

This thesis focuses on term weighting in short documents. I propose weighting approaches for assessing the importance of terms for three tasks: (1) document categorization, which aims to classify documents such as tweets into categories, (2) keyword extraction, which aims to identify and extract the most important words of a document, and (3) keyword association modeling, which aims to identify links between keywords and use them for query expansion. As the focus of text mining is shifting toward datasets that hold usergenerated content, for example, social media, the type of data used in the text mining research is changing. The main characteristic of this data is its shortness. For example, a user status update usually contains less than 20 words. When using short documents, the biggest challenge in term weighting comes from the fact that most words of a document occur only once within the document. This is called hapax legomena and we call it Term Frequency = 1, or TF=1 challenge. As many traditional feature weighting approaches, such as Term Frequency Inverse Document Frequency, are based on the occurrence frequency of each word within a document, these approaches do not perform well with short documents. The first contribution of this thesis is a term weighting approach for doc-