An evaluation of text classification methods for literary study

Text classification methods have been evaluated on topic classification tasks. This thesis extends the empirical evaluation to emotion classification tasks in the literary domain. This study selects two literary text classification problems---the eroticism classification in Dickinson's poems and the sentimentalism classification in early American novels---as two cases for this evaluation. Both problems focus on identifying certain kinds of emotion---a document property other than topic. This study chooses two popular text classification algorithms---naive Bayes and Support Vector Machines (SVM), and three feature engineering options---stemming, stopword removal and statistical feature selection (Odds Ratio and SVM)---as the subjects of evaluation. This study aims to examine the effects of the chosen classifiers and feature engineering options on the two emotion classification problems, and the interaction between the classifiers and the feature engineering options. This thesis seeks empirical answers to the following research questions: (1) is SVM a better classifier than naive Bayes regarding classification accuracy, new literary knowledge discovery and potential for example-based retrieval? (2) is SVM a better feature selection method than Odds Ratio regarding feature reduction rate and classification accuracy improvement? (3) does stop word removal affect the classification performance? (4) does stemming affect the performance of classifiers and feature selection methods? Some of our conclusions are consistent with what are obtained in topic classification, such as Odds Ratio does not improve SVM performance and stop word removal might harm classification. Some conclusions contradict previous results, such as SVM does not beat naive Bayes in both cases. Some findings are new to this area---SVM and naive Bayes select top features in different frequency ranges; stemming might harm feature selection methods. These experiment results provide new insights to the relation between classification methods, feature engineering options and non-topic document properties. They also provide guidance for classification method selection in literary text classification applications.