Combination of modified BPNN algorithms and an efficient feature selection method for text categorization

This paper proposes new modified methods for back propagation neural networks and uses semantic feature space to improve categorization performance and efficiency. The standard back propagation neural network (BPNN) has the drawbacks of slow learning and getting trapped in local minima, leading to a network with poor performance and efficiency. In this paper, we propose two methods to modify the standard BPNN and adopt the semantic feature space (SFS) method to reduce the number of dimensions as well as construct latent semantics between terms. The experimental results show that the modified methods enhanced the performance of the standard BPNN and were more efficient than the standard BPNN. The SFS method cannot only greatly reduce the dimensionality, but also enhances performance and can therefore be used to further improve text categorization systems precisely and efficiently.

[1]  Tai-Yue Wang,et al.  Fuzzy support vector machine for multi-class text categorization , 2007, Inf. Process. Manag..

[2]  Feng Yu,et al.  Text Classification Based on a Combination of Ontology with Statistical Method , 2006, 2006 International Conference on Machine Learning and Cybernetics.

[3]  Haym Hirsh,et al.  Using LSI for text classification in the presence of background text , 2001, CIKM '01.

[4]  Zhi-Hua Zhou,et al.  Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization , 2006, IEEE Transactions on Knowledge and Data Engineering.

[5]  Adrian J. Shepherd,et al.  Second-Order Methods for Neural Networks , 1997 .

[6]  Dik Lun Lee,et al.  Feature reduction for neural network based text categorization , 1999, Proceedings. 6th International Conference on Advanced Systems for Advanced Applications.

[7]  Yashwant Prasad Singh,et al.  Dynamic tunneling technique for efficient training of multilayer perceptrons , 1999, IEEE Trans. Neural Networks.

[8]  Cheng Hua Li,et al.  Text Categorization Based on Artificial Neural Networks , 2006, ICONIP.

[9]  Wei Wu,et al.  Deterministic convergence of an online gradient method for BP neural networks , 2005, IEEE Transactions on Neural Networks.

[10]  Bin-Da Liu,et al.  A backpropagation algorithm with adaptive learning rate and momentum coefficient , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[11]  De-shuang Huang,et al.  The structure optimization of radial basis probabilistic neural networks based on genetic algorithms , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[12]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[13]  Antônio de Pádua Braga,et al.  Improving neural networks generalization with new constructive and pruning methods , 2002, J. Intell. Fuzzy Syst..

[14]  Yiming Yang,et al.  An example-based mapping method for text categorization and retrieval , 1994, TOIS.

[15]  Alberto Tesi,et al.  On the Problem of Local Minima in Backpropagation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Hwee Tou Ng,et al.  Feature selection, perceptron learning, and a usability case study for text categorization , 1997, SIGIR '97.

[17]  Satarupa Banerjee,et al.  Text classification: A least square support vector machine approach , 2007, Appl. Soft Comput..

[18]  Vassilis P. Plagianakos,et al.  Training neural networks with threshold activation functions and constrained integer weights , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[19]  David William Pearson,et al.  Applications of artificial neural networks , 1998 .

[20]  Mark R. Lehto,et al.  Hybrid Singular Value Decomposition: A Model of Human Text Classification , 2007, HCI.

[21]  Muh-Cherng Wu,et al.  An effective application of decision tree to stock trading , 2006, Expert Syst. Appl..

[22]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[23]  Tommy W. S. Chow,et al.  A weight initialization method for improving training speed in feedforward neural network , 2000, Neurocomputing.

[24]  Alberto L. Sangiovanni-Vincentelli,et al.  Efficient Parallel Learning Algorithms for Neural Networks , 1988, NIPS.

[25]  John F. Kolen,et al.  Backpropagation is Sensitive to Initial Conditions , 1990, Complex Syst..

[26]  H. Tamura,et al.  An improved backpropagation algorithm to avoid the local minima problem , 2004, Neurocomputing.

[27]  Hyung Jeong Yang,et al.  Hierarchical document categorization with k-NN and concept-based thesauri , 2006, Inf. Process. Manag..

[28]  De-shuang Huang,et al.  The optimization of radial basis probabilistic neural networks based on genetic algorithms , 2002, Proceedings of the International Joint Conference on Neural Networks, 2003..

[29]  Wei-Ying Ma,et al.  Supervised latent semantic indexing for document categorization , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[30]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[31]  Adrian J. Shepherd,et al.  Second-order methods for neural networks - fast and reliable training methods for multi-layer perceptrons , 1997, Perspectives in neural computing.

[32]  Zheng Tang,et al.  A modified error function for the backpropagation algorithm , 2004, Neurocomputing.

[33]  Yogesh Singh,et al.  An activation function adapting training algorithm for sigmoidal feedforward networks , 2004, Neurocomputing.

[34]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[35]  Songbo Tan,et al.  An effective refinement strategy for KNN text classifier , 2006, Expert Syst. Appl..

[36]  Guy W. Mineau,et al.  Beyond TFIDF Weighting for Text Categorization in the Vector Space Model , 2005, IJCAI.

[37]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[38]  Min-Soeng Kim,et al.  Nonlinear time series modelling and prediction using Gaussian RBF network with evolutionary structure optimisation , 2001 .

[39]  Arjen van Ooyen,et al.  Improving the convergence of the back-propagation algorithm , 1992, Neural Networks.

[40]  Emile Fiesler,et al.  High-order and multilayer perceptron initialization , 1997, IEEE Trans. Neural Networks.

[41]  Padmini Srinivasan,et al.  Automatic Text Categorization Using Neural Networks , 1997 .

[42]  Andrew McCallum,et al.  A comparison of event models for naive bayes text classification , 1998, AAAI 1998.

[43]  Yiming Yang,et al.  Noise reduction in a statistical approach to text categorization , 1995, SIGIR '95.

[44]  Rich Caruana,et al.  Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping , 2000, NIPS.

[45]  Charles B. Owen,et al.  Application of simulated annealing to the backpropagation model improves convergence , 1993, Defense, Security, and Sensing.

[46]  Guo-An Chen,et al.  Acceleration of backpropagation learning using optimised learning rate and momentum , 1993 .

[47]  Michael K. Weir,et al.  A method for self-determination of adaptive learning rates in back propagation , 1991, Neural Networks.

[48]  Wen-Hung Yang,et al.  AN ELECTROMAGNETISM ALGORITHM OF NEURAL NETWORK ANALYSIS—AN APPLICATION TO TEXTILE RETAIL OPERATION , 2004 .