Distributed Computing With the Cloud

We investigate the effect of omnipresent cloud storage on distributed computing. We specify a network model with links of prescribed bandwidth that connect standard processing nodes, and, in addition, passive storage nodes. Each passive node represents a cloud storage system, such as Dropbox, Google Drive etc. We study a few tasks in this model, assuming a single cloud node connected to all other nodes, which are connected to each other arbitrarily. We give implementations for basic tasks of collaboratively writing to and reading from the cloud, and for more advanced applications such as matrix multiplication and federated learning. Our results show that utilizing node-cloud links as well as node-node links can considerably speed up computations, compared to the case where processors communicate either only through the cloud or only through the network links. We provide results for general directed graphs, and for graphs with ``fat'' links between processing nodes. For the general case, we provide optimal algorithms for uploading and downloading files using flow techniques. We use these primitives to derive algorithms for \emph{combining}, where every processor node has an input value and the task is to compute a combined value under some given associative operator. In the case of fat links, we assume that links between processors are bidirectional and have high bandwidth, and we give near-optimal algorithms for any commutative combining operator (such as vector addition). For the task of matrix multiplication (or other non-commutative combining operators), where the inputs are ordered, we present sharp results in the simple ``wheel'' network, where procesing nodes are arranged in a ring, and are all connected to a single cloud node.

[1]  Baruch Awerbuch,et al.  Sparse Partitions (Extended Abstract , 1990, FOCS 1990.

[2]  Roy Friedman,et al.  Hybrid Distributed Consensus , 2013, OPODIS.

[3]  R. Karp,et al.  LogP: towards a realistic model of parallel computation , 1993, PPOPP '93.

[4]  LiuYang,et al.  Federated learning for privacy-preserving AI , 2020 .

[5]  William J. Bolosky,et al.  Single Instance Storage in Windows , 2000 .

[6]  Seif Haridi,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[7]  Dutch T. Meyer,et al.  A study of practical deduplication , 2011, TOS.

[8]  Martin Skutella,et al.  Quickest Flows Over Time , 2007, SIAM J. Comput..

[9]  Trade-offs between Communication Throughput and Parallel Time , 1999, J. Complex..

[10]  Sriram V. Pemmaraju,et al.  Lessons from the Congested Clique applied to MapReduce , 2015, Theor. Comput. Sci..

[11]  D. T. Lee,et al.  On a Circle-Cover Minimization Problem , 1984, Inf. Process. Lett..

[12]  Marvin Theimer,et al.  Reclaiming space from duplicate files in a serverless distributed file system , 2002, Proceedings 22nd International Conference on Distributed Computing Systems.

[13]  Nathan Linial,et al.  Locality in Distributed Graph Algorithms , 1992, SIAM J. Comput..

[14]  Éva Tardos,et al.  Polynomial time algorithms for some evacuation problems , 1994, SODA '94.

[15]  Fabian Kuhn,et al.  Computing Shortest Paths and Diameter in the Hybrid Network Model , 2020, PODC.

[16]  Boaz Patt-Shamir,et al.  Minimum-Weight Spanning Tree Construction in O(log log n) Communication Rounds , 2005, SIAM J. Comput..

[17]  Martin Skutella,et al.  Solving Evacuation Problems Efficiently--Earliest Arrival Flows with Multiple Sources , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[18]  Keren Censor-Hillel,et al.  Distance Computations in the Hybrid Network Model via Oracle Simulations , 2020, STACS.

[19]  Rainer E. Burkard,et al.  The quickest flow problem , 1993, ZOR Methods Model. Oper. Res..

[20]  A. Kaufmann,et al.  Methods and models of operations research , 1963 .

[21]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[22]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[23]  Gad M. Landau,et al.  The power of multimedia: combining point-to point and multi-access networks , 1988, PODC '88.

[24]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[25]  Pierre Fraigniaud.  Distributed computational complexities: are you volvo-addicted or nascar-obsessed? , 2010, PODC '10.

[26]  Leslie G. Valiant,et al.  A bridging model for parallel computation , 1990, CACM.

[27]  David Peleg,et al.  Distributed Computing: A Locality-Sensitive Approach , 1987 .

[28]  Martin Skutella,et al.  An Introduction to Network Flows over Time , 2008, Bonn Workshop of Combinatorial Optimization.

[29]  Sergei Vassilvitskii,et al.  A model of computation for MapReduce , 2010, SODA '10.

[30]  Yossi Matias,et al.  Can shared-memory model serve as a bridging model for parallel computation? , 1997, SPAA '97.

[31]  Yossi Matias,et al.  Modeling Parallel Bandwidth: Local versus Global Restrictions , 1999, Algorithmica.

[32]  Christian Scheideler,et al.  Shortest Paths in a Hybrid Network Model , 2020, SODA.

[33]  L. Freeman.  Looking Beyond the Hype : Evaluating Data Deduplication Solutions , 2007 .