Fine-tuning the consistency-latency trade-off in quorum-replicated distributed storage systems

NoSQL storage systems are used extensively by web applications and provide an attractive alternative to conventional databases when the need for scalability outweighs the need for transactions. Several of these systems, notably Amazon's Dynamo and its open-source derivatives, provide quorum-based replication and present the application developer with a choice of multiple client-side "consistency levels" that determine the number of replicas accessed by reads and writes. This setting, in turn, affects both the latency and the consistency observed by the client application. Since using a fixed combination of read and write consistency levels for a given application provides only a limited number of discrete options for tuning the consistency-latency trade-off, we investigate techniques that allow more fine-grained tuning as may be required to support consistency guarantees through service level agreements (SLAs). We consider two such techniques, a novel technique that assigns the consistency level on a peroperation basis by choosing randomly between two options (e.g., weak vs. strong consistency) with a tunable probability, and a known technique that uses weak consistency and injects delays into storage operations artificially. We compare and contrast these two techniques experimentally against each other and against combinations of fixed consistency levels using Apache Cassandra deployed in Amazon's EC2 environment.