An Efficient Semantic Ranked Keyword Search of Big Data Using Map Reduce

Information retrieval is fast becoming the prevailing form of information access, surpassing traditional database style searching. Ontologies have become the tool of choice employed in many information retrieval systems and more prominently in semantic information retrieval. In order to overcome the disadvantages in key word based information retrieval systems, which transfer irrelevant information, ontology has been designed. A system with ontology mimics the real world, where every task is laced with certain meaning as this is basic idea behind knowledge processing. Hadoop, which is an open source frame work for storing and processing large datasets, is used for preprocessing the text documents. First, a set of text documents are considered. Preprocessing is performed on a large domain of data using Hadoop MapReduce. This includes the removal of the stop words along with stemming and excluding less frequency words. Despite this pre-processing, owing to the colossal number of index terms still floating in the considered domain data, the problem of high dimensionality is encountered. Therefore the dimensionality of such a group of terms is reduced by identifying it as a concept and those concepts can be viewed as a single dimension in a ontology based information retrieval system. Now ontology is constructed by assigning synonym set to each concept in this structure using tools like word net. Thus constructed ontology can be mapped on to the processed query which gives us the relevant information from the data pool considered.