Disk-based shortest path discovery using distance index over large dynamic graphs

Abstract The persistent alternation of the internet world is changing networks rapidly. Shortest path discovery, especially over dynamic networks such as web page links, telephone or route networks, and ontologies, has received intense attention because of its importance for services in IoT. For example, when a new road is newly opened or becomes unavailable for any unexpected reason, the shortest paths must be recomputed. The system should respond promptly to its users with the updated recommended paths. In this paper, we propose a disk-based shortest path method that updates the shortest paths in a very large dynamic graph efficiently. The proposed method uses partial shortest paths as indices for efficient shortest path discovery. We classify the changes in the graph into four cases, such as the insertion or deletion of edges and the increase or decrease of edge weights. Our proposed strategy considers updating only the corresponding parts of the indices for each case. Our experiments on real-world dynamic datasets verify that the proposed framework updates the shortest paths 4 to 50 times faster than the existing type of framework.

[1]  Giuseppe F. Italiano,et al.  Incremental algorithms for minimal length paths , 1991, SODA '90.

[2]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[3]  Haixun Wang,et al.  Efficient subgraph search over large uncertain graphs , 2011, Proc. VLDB Endow..

[4]  Chen Wang,et al.  Scalable mining of large disk-based graph databases , 2004, KDD.

[5]  Aristides Gionis,et al.  Searching the wikipedia with contextual information , 2008, CIKM '08.

[6]  Sharma Chakravarthy,et al.  DB-FSG: An SQL-Based Approach for Frequent Subgraph Mining , 2008, DEXA.

[7]  Norbert Zeh,et al.  An External Memory Data Structure for Shortest Path Queries , 1999, COCOON.

[8]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[9]  Hengqing Tong,et al.  Dynamic Shortest Path Algorithm in Stochastic Traffic Networks Using PSO Based on Fluid Neural Network , 2011, J. Intell. Learn. Syst. Appl..

[10]  Ruth Nussinov,et al.  Structure and dynamics of molecular networks: A novel paradigm of drug discovery. A comprehensive review , 2012, Pharmacology & therapeutics.

[11]  Jon Kleinberg,et al.  Maximizing the spread of influence through a social network , 2003, KDD '03.

[12]  Jeffrey Xu Yu,et al.  Relational Approach for Shortest Path Discovery over Large Graphs , 2011, Proc. VLDB Endow..

[13]  Dong Xin,et al.  Fast personalized PageRank on MapReduce , 2011, SIGMOD '11.

[14]  Marlon Dumas,et al.  Fast fully dynamic landmark-based estimation of shortest path distances in very large graphs , 2011, CIKM '11.

[15]  Jon M. Kleinberg,et al.  Group formation in large social networks: membership, growth, and evolution , 2006, KDD '06.

[16]  Giuseppe F. Italiano,et al.  A new approach to dynamic all pairs shortest paths , 2004, JACM.

[17]  Jure Leskovec,et al.  Defining and evaluating network communities based on ground-truth , 2012, Knowledge and Information Systems.

[18]  Sibo Wang,et al.  Efficient single-source shortest path and distance queries on large graphs , 2013, KDD.

[19]  Aart J. C. Bik,et al.  Pregel: a system for large-scale graph processing , 2010, SIGMOD Conference.

[20]  Norbert Zeh,et al.  An External Memory Data Structure for Shortest Path Queries , 1999, COCOON.

[21]  Lenore Cowen,et al.  Compact roundtrip routing in directed networks , 2004, J. Algorithms.

[22]  Joseph Gonzalez,et al.  PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs , 2012, OSDI.

[23]  Shimon Even,et al.  Updating distances in dynamic graphs , 1985 .

[24]  Jure Leskovec,et al.  The dynamics of viral marketing , 2005, EC '06.

[25]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[26]  Jiawei Han,et al.  gSpan: graph-based substructure pattern mining , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[27]  Fang Wei TEDI: efficient shortest path query answering on graphs , 2010, SIGMOD 2010.

[28]  Jure Leskovec,et al.  Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters , 2008, Internet Math..

[29]  Christos Faloutsos,et al.  Graphs over time: densification laws, shrinking diameters and possible explanations , 2005, KDD '05.

[30]  Takuya Akiba,et al.  Fast exact shortest-path distance queries on large networks by pruned landmark labeling , 2013, SIGMOD '13.

[31]  Hans Rohnert,et al.  A Dynamization of the All Pairs Least Cost Path Problem , 1985, STACS.

[32]  Dorothea Wagner,et al.  Speed-Up Techniques for Shortest-Path Computations , 2007, STACS.

[33]  James Cheng,et al.  Efficient processing of distance queries in large graphs: a vertex cover approach , 2012, SIGMOD Conference.

[34]  Aristides Gionis,et al.  Fast shortest path distance estimation in large networks , 2009, CIKM.

[35]  Svetlana Bureeva,et al.  Network and pathway analysis of compound-protein interactions. , 2009, Methods in molecular biology.

[36]  Andrew V. Goldberg,et al.  Computing the shortest path: A search meets graph theory , 2005, SODA '05.

[37]  M. Newman,et al.  The structure of scientific collaboration networks. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[38]  Philip N. Klein,et al.  Faster Shortest-Path Algorithms for Planar Graphs , 1997, J. Comput. Syst. Sci..

[39]  Laks V. S. Lakshmanan,et al.  Efficient network aware search in collaborative tagging sites , 2008, Proc. VLDB Endow..

[40]  Jeffrey Xu Yu,et al.  Shortest Path Computing in Relational DBMSs , 2014, IEEE Transactions on Knowledge and Data Engineering.

[41]  Takuya Akiba,et al.  Shortest-path queries for complex networks: exploiting low tree-width outside the core , 2012, EDBT '12.

[42]  Sharma Chakravarthy,et al.  HDB-Subdue: A Scalable Approach to Graph Mining , 2009, DaWaK.

[43]  Berthier A. Ribeiro-Neto,et al.  Efficient search ranking in social networks , 2007, CIKM '07.

[44]  Joseph M. Hellerstein,et al.  Distributed GraphLab: A Framework for Machine Learning in the Cloud , 2012, Proc. VLDB Endow..