Comparing Neural Network Approach With N- Gram Approach For Text Categorization

This paper compares Neural network Approach with N-gram approach, for text categorization, and demonstrates that Neural Network approach is similar to the N-gram approach but with much less judging time. Both methods demonstrated here are aimed at language identification. The presence of particular characters, words and the statistical information of word lengths are used as a feature vector. In an identification experiment with Asian languages the neural network approach achieved 98% correct classification rate with 500 bytes, but it is five times faster than n-gram based approach. Keywords-N-Gram,Neural Network,LanguageIdentification,Text categorization I. INTRODUCTION As Internet services supersizing increasing in popularity, more and more languages are able to make their way online. In such a trend, a need exits for the rapid organizing of ever- expanding electronic documents. A well-trained librarian can easily identify the language of a book or a document, but it is not so easy when it presents online: there are so many documents in so many languages, and most of them cannot be passed immediately with a glance eye. Thus an automatically language identification system is needed to be built to take this task. Because of the sheer volume of documents to be handled, the categorization must be efficient, consuming as small storage and little processing time as possible. Text classification addresses the problem of assigning a given passage of text (or a document) to one or more predefined classes. This is an important area of information retrieval research that has been heavily investigated. A segmentation-based approach was compared with the non-segmentation-based approach. N-gram based approach is the most widely accepted one and it is proved to have a good performance. As crucial as accuracy, the speed of classification is also a crucial factor for a classifier in a huge volume of categorization environment. However, most of the authors did not provide any information on the speed of classification. In the following sections, t he comparison of the performances of categorization algorithms using Neural networks and N-gram based approach are discussed. It is demonstrated that the identification rate of Neural networks is similar to the corresponding Ngram approach but with much less judging time.