论文信息 - A new weighting algorithm for linear classifier

A new weighting algorithm for linear classifier

In the domain of text categorization (TC), the TF (term frequency)* IDF (inverse document frequency) weighting algorithm and TF*IWF*IWF weighting algorithm are widely used. However, the two algorithms are too biased by the term frequency and neglect the imbalance between classes. In this paper, we propose a new weighting algorithm, which is named as TF (term frequency)*IWF (inverse word frequency)*IWF (inverse word frequency)*VE (variance and expectation). The new algorithm improves the TF*IWF*IWF weighting algorithm in both TF and VE. This paper compares the new algorithm with TF*IWF*IWF algorithm respectively in theory and experiment. From the preliminary experiment, we find that the F1-measure has been improved for 11.78%.

Chengqing Zong | Keli Chen

[1] Thorsten Joachims,et al. A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization , 1997, ICML.

[2] Roberto Basili,et al. Language sensitive text classification , 2000, RIAO.

[3] Yiming Yang,et al. An Evaluation of Statistical Approaches to Text Categorization , 1999, Information Retrieval.

[4] Fabrizio Sebastiani,et al. A Tutorial on Automated Text Categorisation , 2000 .

[5] Gerard Salton,et al. Experiments in Multi-Lingual Information Retrieval , 1972, Inf. Process. Lett..

[6] Yiming Yang,et al. A re-examination of text categorization methods , 1999, SIGIR '99.

[7] Youngjoong Ko,et al. Automatic Text Categorization by Unsupervised Learning , 2000, COLING.

[8] G Salton,et al. Developments in Automatic Text Retrieval , 1991, Science.

[9] James P. Callan,et al. Training algorithms for linear text classifiers , 1996, SIGIR '96.