An Efficient Data Access Method Exploiting Quadtrees on MapReduce Frameworks

Due to the advance of diverse techniques such as social networks and sensor networks, the volume of data to be processed has rapidly increased. After Google proposed the MapReduce framework which processes big data using large clusters of commodity machines, the MapReduce framework is considered as an effective processing paradigm for a massive data set. However, in the view of the performance, a problem of the MapReduce framework is that an efficient access method i.e., an index is not supported. Thus, whole data should be retrieved even though a user wants to access a small portion of data. In this paper, we propose an efficient method constructing quadtrees on the MapReduce framework. Our technique reduces the index construction time utilizing a sampling technique to partition a data set. In addition, using the constructed quadtree as well as the MapReduce framework, a subset of data to be retrieved is easily identified and is processed in parallel. Our experimental result demonstrates the efficiency of our proposed algorithm with diverse environments.

[1]  Feifei Li,et al.  Building Wavelet Histograms on Large Data in MapReduce , 2011, Proc. VLDB Endow..

[2]  Jay L. Devore,et al.  Probability and statistics for engineering and the sciences , 1982 .

[3]  David R. O'Hallaron,et al.  Materialized community ground models for large-scale earthquake simulation , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[4]  Naphtali Rishe,et al.  Experiences on Processing Spatial Data with MapReduce , 2009, SSDBM.

[5]  Farnoush Banaei Kashani,et al.  Voronoi-Based Geospatial Query Processing with MapReduce , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[6]  Marianne Winslett,et al.  Scientific and Statistical Database Management, 21st International Conference, SSDBM 2009, New Orleans, LA, USA, June 2-4, 2009, Proceedings , 2009, SSDBM.

[7]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[8]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[9]  Beng Chin Ooi,et al.  Gorder: An Efficient Method for KNN Join Processing , 2004, VLDB.

[10]  Yonggang Wang,et al.  Research and implementation on spatial data storage and operation based on Hadoop platform , 2010, 2010 Second IITA International Conference on Geoscience and Remote Sensing.

[11]  Kai Wang,et al.  Spatial Queries Evaluation with MapReduce , 2009, 2009 Eighth International Conference on Grid and Cooperative Computing.

[12]  Jeffrey Scott Vitter,et al.  Random sampling with a reservoir , 1985, TOMS.

[13]  Xiaqing Wu,et al.  Automatic alignment of large-scale aerial rasters to road-maps , 2007, GIS.

[14]  Ge Yu,et al.  Database systems for advanced applications: 16th international conference, DASFAA 2011 international workshops: GDB, SIM3, FlashDB, SNSMW, DaMEN, DQIS, Hong Kong, China, April 22-25, 2011, proceedings , 2011 .

[15]  Shuigeng Zhou,et al.  Adapting Skyline Computation to the MapReduce Framework: Algorithms and Experiments , 2011, DASFAA Workshops.

[16]  Xuan Song,et al.  Accelerating Spatial Data Processing with MapReduce , 2010, 2010 IEEE 16th International Conference on Parallel and Distributed Systems.

[17]  Douglas Comer,et al.  Ubiquitous B-Tree , 1979, CSUR.

[18]  R. Vershynin.  How Close is the Sample Covariance Matrix to the Actual Covariance Matrix? , 2010, 1004.3484.

[19]  Jon Louis Bentley,et al.  Quad trees a data structure for retrieval on composite keys , 1974, Acta Informatica.

[20]  Beng Chin Ooi,et al.  Efficient Processing of k Nearest Neighbor Joins using MapReduce , 2012, Proc. VLDB Endow..

[21]  Min Wang,et al.  Efficient Multi-way Theta-Join Processing Using MapReduce , 2012, Proc. VLDB Endow..