An automatically constructed thesaurus for neural network based document categorization

This paper presents a method for computing a thesaurus from a text corpus, and combined with a revised back-propagation neural network (BPNN) learning algorithm for document categorization. Automatically constructed thesaurus is a data structure that accomplished by extracting the relatedness between words. Neural network is one of the efficient approaches for document categorization. However the conventional BPNN has the problems of slow learning and easy to involve into the local minimum. We use a revised algorithm to improve the conventional BPNN that can overcome these problems. A well constructed thesaurus has been recognized as valuable tool in the effective operation of document categorization, it overcome some problem for the document categorization based on bag of words which ignored the relationship between words. To investigate the effectiveness of our method, we conducted the experiments on the standard Reuter-21578. The experimental results show that the proposed model was able to achieve higher categorization effectiveness as measured by the precision, recall and F-measure.

[1]  De-shuang Huang,et al.  The optimization of radial basis probabilistic neural networks based on genetic algorithms , 2002, Proceedings of the International Joint Conference on Neural Networks, 2003..

[2]  Tai-Yue Wang,et al.  Fuzzy support vector machine for multi-class text categorization , 2007, Inf. Process. Manag..

[3]  Jason D. M. Rennie.  ifile: An Application of Machine Learning to E-Mail Filtering , 2000 .

[4]  Satarupa Banerjee,et al.  Text classification: A least square support vector machine approach , 2007, Appl. Soft Comput..

[5]  Charles B. Owen,et al.  Application of simulated annealing to the backpropagation model improves convergence , 1993, Defense, Security, and Sensing.

[6]  Vassilis P. Plagianakos,et al.  Training neural networks with threshold activation functions and constrained integer weights , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[7]  Hans-Peter Frei,et al.  Concept based query expansion , 1993, SIGIR.

[8]  David William Pearson,et al.  Applications of artificial neural networks , 1998 .

[9]  Adrian J. Shepherd,et al.  Second-order methods for neural networks - fast and reliable training methods for multi-layer perceptrons , 1997, Perspectives in neural computing.

[10]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[11]  Hu Yunfa,et al.  Using Maximum Entropy Model for Chinese Text Categorization , 2005 .

[12]  Nenghai Yu,et al.  Mutually beneficial learning with application to on-line news classification , 2007, PIKM '07.

[13]  Bin-Da Liu,et al.  A backpropagation algorithm with adaptive learning rate and momentum coefficient , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[14]  Jun'ichi Tsujii,et al.  Maximum Entropy Models with Inequality Constraints: A Case Study on Text Categorization , 2005, Machine Learning.

[15]  Verayuth Lertnattee,et al.  Class normalization in centroid-based text categorization , 2006, Inf. Sci..

[16]  Wei Wu,et al.  Deterministic convergence of an online gradient method for BP neural networks , 2005, IEEE Transactions on Neural Networks.

[17]  Alberto L. Sangiovanni-Vincentelli,et al.  Efficient Parallel Learning Algorithms for Neural Networks , 1988, NIPS.

[18]  Emile Fiesler,et al.  High-order and multilayer perceptron initialization , 1997, IEEE Trans. Neural Networks.

[19]  John F. Kolen,et al.  Backpropagation is Sensitive to Initial Conditions , 1990, Complex Syst..

[20]  Zhi-Hua Zhou,et al.  Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization , 2006, IEEE Transactions on Knowledge and Data Engineering.

[21]  Adrian J. Shepherd,et al.  Second-Order Methods for Neural Networks , 1997 .

[22]  Yiming Yang,et al.  An example-based mapping method for text categorization and retrieval , 1994, TOIS.

[23]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[24]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[25]  Chuanyi Ji,et al.  A unified approach on fast training of feedforward and recurrent networks using EM algorithm , 1998, IEEE Trans. Signal Process..

[26]  Michael K. Weir,et al.  A method for self-determination of adaptive learning rates in back propagation , 1991, Neural Networks.

[27]  Wen-Hung Yang,et al.  AN ELECTROMAGNETISM ALGORITHM OF NEURAL NETWORK ANALYSIS—AN APPLICATION TO TEXTILE RETAIL OPERATION , 2004 .

[28]  Zheng Tang,et al.  A modified error function for the backpropagation algorithm , 2004, Neurocomputing.

[29]  Yogesh Singh,et al.  An activation function adapting training algorithm for sigmoidal feedforward networks , 2004, Neurocomputing.

[30]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[31]  Wai Lam,et al.  Automatic Text Categorization and Its Application to Text Retrieval , 1999, IEEE Trans. Knowl. Data Eng..

[32]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[33]  Ángel F. Zazo Rodríguez,et al.  Reformulation of queries using similarity thesauri , 2005, Inf. Process. Manag..

[34]  Muh-Cherng Wu,et al.  An effective application of decision tree to stock trading , 2006, Expert Syst. Appl..

[35]  Tommy W. S. Chow,et al.  A weight initialization method for improving training speed in feedforward neural network , 2000, Neurocomputing.

[36]  Songbo Tan,et al.  An effective refinement strategy for KNN text classifier , 2006, Expert Syst. Appl..

[37]  Min-Soeng Kim,et al.  Nonlinear time series modelling and prediction using Gaussian RBF network with evolutionary structure optimisation , 2001 .

[38]  Hyung Jeong Yang,et al.  Hierarchical document categorization with k-NN and concept-based thesauri , 2006, Inf. Process. Manag..

[39]  Weiyi Liu,et al.  A fuzzy approach to classification of text documents , 2008, Journal of Computer Science and Technology.

[40]  Lourdes Araujo,et al.  Query Expansion with an Automatically Generated Thesaurus , 2006, IDEAL.