On the Synchronization Bottleneck of OpenStack Swift-Like Cloud Storage Systems

As one type of the most popular cloud storage services, OpenStack Swift and its follow-up systems replicate each object across multiple storage nodes and leverage <italic>object sync protocols</italic> to achieve high reliability and <italic>eventual consistency</italic>. The performance of object sync protocols heavily relies on two key parameters: <inline-formula><tex-math notation="LaTeX">$r$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq1-2810179.gif"/></alternatives></inline-formula> (number of replicas for each object) and <inline-formula><tex-math notation="LaTeX">$n$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq2-2810179.gif"/></alternatives></inline-formula> (number of objects hosted by each storage node). In existing tutorials and demos, the configurations are usually <inline-formula> <tex-math notation="LaTeX">$r=3$</tex-math><alternatives><inline-graphic xlink:href="ruan-ieq3-2810179.gif"/> </alternatives></inline-formula> and <inline-formula><tex-math notation="LaTeX">$n<1,000$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq4-2810179.gif"/></alternatives></inline-formula> by default, and the sync process seems to perform well. However, we discover in data-intensive scenarios, e.g., when <inline-formula> <tex-math notation="LaTeX">$r>3$</tex-math><alternatives><inline-graphic xlink:href="ruan-ieq5-2810179.gif"/> </alternatives></inline-formula> and <inline-formula><tex-math notation="LaTeX">$n\gg 1,000$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq6-2810179.gif"/></alternatives></inline-formula>, the sync process is significantly delayed and produces massive network overhead, referred to as the <italic>sync bottleneck problem</italic>. By reviewing the source code of OpenStack Swift, we find that its object sync protocol utilizes a fairly simple and network-intensive approach to check the consistency among replicas of objects. Hence in a sync round, the number of exchanged hash values per node is <inline-formula><tex-math notation="LaTeX">$\Theta (n\times r)$</tex-math> <alternatives><inline-graphic xlink:href="ruan-ieq7-2810179.gif"/></alternatives></inline-formula>. To tackle the problem, we propose a lightweight and practical object sync protocol, <italic>LightSync</italic>, which not only remarkably reduces the sync overhead, but also preserves high reliability and eventual consistency. LightSync derives this capability from three novel building blocks: 1) <italic>Hashing of Hashes</italic>, which aggregates all the <inline-formula><tex-math notation="LaTeX">$h$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq8-2810179.gif"/></alternatives></inline-formula> hash values of each data partition into a single but representative hash value with the Merkle tree; 2) <italic>Circular Hash Checking</italic>, which checks the consistency of different partition replicas by only sending the aggregated hash value to the clockwise neighbor; and 3) <italic>Failed Neighbor Handling</italic>, which properly detects and handles node failures with moderate overhead to effectively strengthen the robustness of LightSync. The design of LightSync offers provable guarantee on reducing the per-node network overhead from <inline-formula><tex-math notation="LaTeX">$\Theta (n\times r)$</tex-math><alternatives><inline-graphic xlink:href="ruan-ieq9-2810179.gif"/></alternatives></inline-formula> to <inline-formula><tex-math notation="LaTeX">$\Theta (\frac{n}{h})$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq10-2810179.gif"/></alternatives></inline-formula>. Furthermore, we have implemented LightSync as an open-source patch and adopted it to OpenStack Swift, thus reducing the sync delay by up to 879 <inline-formula><tex-math notation="LaTeX">$\times$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq11-2810179.gif"/></alternatives></inline-formula> and the network overhead by up to 47.5<inline-formula><tex-math notation="LaTeX">$\times$</tex-math><alternatives> <inline-graphic xlink:href="ruan-ieq12-2810179.gif"/></alternatives></inline-formula>.