Synchronizing state with strong similarity between local and remote systems

Distributed systems such as a mobile device and its cloud storage have a strongly similar state: they are periodically synchronized, but evolve independently in between the synchronization events, sometimes in a disconnected manner that makes keeping the state consistent at both end difficult. This scenario appears in a wide range of situations, between a mobile device application and its server, in database situations. Another context is that of two routers which see similar traffic to populate their route tables and want to exchange these route tables. In all these cases, we need to periodically synchronize a local state and a remote state at the minimum possible cost in terms of bandwidth in between the two synchronization points. We consider a simple distributed source coding approach to the problem of efficient set reconciliation to support the exchange of this highly correlated caching information between a local and a remote system. We show theoretically that we can keep the amount of exchanged data proportional to the size of the cache difference; and we show by simulation that our method is practical to implement.

[1]  Yaron Minsky,et al.  Set reconciliation with nearly optimal communication complexity , 2003, IEEE Trans. Inf. Theory.

[2]  Van Jacobson,et al.  Networking named content , 2009, CoNEXT '09.

[3]  Zixiang Xiong,et al.  Distributed source coding for sensor networks , 2004, IEEE Signal Processing Magazine.

[4]  K. Ramchandran,et al.  Distributed source coding using syndromes (DISCUS): design and construction , 1999, Proceedings DCC'99 Data Compression Conference (Cat. No. PR00096).

[5]  Christof Fetzer,et al.  Bloom filter based routing for content-based publish/subscribe , 2008, DEBS.

[6]  Li Fan,et al.  Summary cache: a scalable wide-area Web cache sharing protocol , 1998, SIGCOMM '98.

[7]  Li Fan,et al.  Web caching and Zipf-like distributions: evidence and implications , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[8]  George Varghese,et al.  What's the difference? , 2011, SIGCOMM 2011.

[9]  R. A. McDonald,et al.  Noiseless Coding of Correlated Information Sources , 1973 .

[10]  Van Jacobson,et al.  Adaptive web caching: towards a new global caching architecture , 1998, Comput. Networks.

[11]  Kannan Ramchandran,et al.  Distributed source coding using syndromes (DISCUSS): design and construction , 1999 .

[12]  Alfredo Viola,et al.  Exact distribution of individual displacements in linear probing hashing , 2005, TALG.

[13]  Bernd Girod,et al.  Rate-adaptive codes for distributed source coding , 2006, Signal Process..

[14]  Karen R. Sollins,et al.  Arguments for an information-centric internetworking architecture , 2010, CCRV.

[15]  Scott Shenker,et al.  A data-oriented (and beyond) network architecture , 2007, SIGCOMM 2007.