The CacheLib Caching Engine: Design and Experiences at Scale

Web services rely on caching at nearly every layer of the system architecture. Commonly, each cache is implemented and maintained independently by a distinct team and is highly specialized to its function. For example, an application-data cache would be independent from a CDN cache. However, this approach ignores the difficult challenges that different caching systems have in common, greatly increasing the overall effort required to deploy, maintain, and scale each cache. This paper presents a different approach to cache development, successfully employed at Facebook, which extracts a core set of common requirements and functionality from otherwise disjoint caching systems. CacheLib is a generalpurpose caching engine, designed based on experiences with a range of caching use cases at Facebook, that facilitates the easy development and maintenance of caches. CacheLib was first deployed at Facebook in 2017 and today powers over 70 services including CDN, storage, and application-data caches. This paper describes our experiences during the transition from independent, specialized caches to the widespread adoption of CacheLib. We explain how the characteristics of production workloads and use cases at Facebook drove important design decisions. We describe how caches at Facebook have evolved over time, including the significant benefits seen from deploying CacheLib. We also discuss the implications our experiences have for future caching design and research.

[1]  Xiaoning Ding,et al.  DULO: an effective buffer cache management scheme to exploit both temporal and spatial locality , 2005, FAST'05.

[2]  Bingsheng He,et al.  Tree indexing on solid state drives , 2010, Proc. VLDB Endow..

[3]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[4]  Sachin Katti,et al.  Cliffhanger: Scaling Performance Cliffs in Web Memory Caches , 2016, NSDI.

[5]  Eunji Lee,et al.  Preventing Fast Wear-out of Flash Cache with An Admission Control Policy , 2015 .

[6]  Michael J. Freedman,et al.  Hyperbolic Caching: Flexible Caching for Web Applications , 2017, USENIX ATC.

[7]  David A. Patterson,et al.  Computer Architecture: A Quantitative Approach , 1969 .

[8]  Willy Zwaenepoel,et al.  Size-aware Sharding For Improving Tail Latencies in In-memory Key-value Stores , 2018, NSDI.

[9]  Kannan Ramchandran,et al.  EC-Cache: Load-Balanced, Low-Latency Cluster Caching with Online Erasure Coding , 2016, OSDI.

[10]  Jacob Nelson,et al.  IncBricks: Toward In-Network Computation with an In-Network Cache , 2017, ASPLOS.

[11]  Eddie Kohler,et al.  Cache craftiness for fast multicore key-value storage , 2012, EuroSys '12.

[12]  Pradeep Dubey,et al.  Architecting to achieve a billion requests per second throughput on a single key-value store server platform , 2015, 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA).

[13]  Badrish Chandramouli,et al.  FASTER: A Concurrent Key-Value Store with In-Place Updates , 2018, SIGMOD Conference.

[14]  Mor Harchol-Balter,et al.  RobinHood: Tail Latency Aware Caching - Dynamic Reallocation from Cache-Rich to Cache-Poor , 2018, OSDI.

[15]  Michel Dubois,et al.  Cache replacement algorithms with nonuniform miss costs , 2006, IEEE Transactions on Computers.

[16]  Li Fan,et al.  Web caching and Zipf-like distributions: evidence and implications , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[17]  Kai Li,et al.  RIPQ: Advanced Photo Caching on Flash for Facebook , 2015, FAST.

[18]  Brad Fitzpatrick,et al.  Distributed caching with memcached , 2004 .

[19]  Yuanyuan Zhou,et al.  The Multi-Queue Replacement Algorithm for Second Level Buffer Caches , 2001, USENIX Annual Technical Conference, General Track.

[20]  Bruce M. Maggs,et al.  Globally Distributed Content Delivery , 2002, IEEE Internet Comput..

[21]  Ulrich Drepper,et al.  What Every Programmer Should Know About Memory , 2007 .

[22]  Douglas Comer,et al.  Ubiquitous B-Tree , 1979, CSUR.

[23]  Cory Hill,et al.  f4: Facebook's Warm BLOB Storage System , 2014, OSDI.

[24]  Nate Foster,et al.  NetCache: Balancing Key-Value Stores with Fast In-Network Caching , 2017, SOSP.

[25]  Timothy G. Armstrong,et al.  LinkBench: a database benchmark based on the Facebook social graph , 2013, SIGMOD '13.

[26]  Akanksha Jain,et al.  Back to the Future: Leveraging Belady's Algorithm for Improved Cache Replacement , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[27]  Roy Friedman,et al.  TinyLFU: A Highly Efficient Cache Admission Policy , 2014, PDP.

[28]  Daniel Sánchez,et al.  Talus: A simple way to remove cliffs in cache performance , 2015, 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA).

[29]  Alan L. Cox,et al.  GD-Wheel: a cost-aware replacement policy for key-value stores , 2015, EuroSys.

[30]  Hyeontaek Lim,et al.  MICA: A Holistic Approach to Fast In-Memory Key-Value Storage , 2014, NSDI.

[31]  Mor Harchol-Balter,et al.  Saving Cash by Using Less Cache , 2012, HotCloud.

[32]  Andrea C. Arpaci-Dusseau,et al.  The Unwritten Contract of Solid State Drives , 2017, EuroSys.

[33]  Ludmila Cherkasova,et al.  Improving WWW Proxies Performance with Greedy-Dual- Size-Frequency Caching Policy , 1998 .

[34]  Bin Fan,et al.  MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing , 2013, NSDI.

[35]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[36]  Mark Silberstein,et al.  GAIA: An OS Page Cache for Heterogeneous Systems , 2019, USENIX Annual Technical Conference.

[37]  Sachin Katti,et al.  Flashield: a Hybrid Key-value Cache that Controls Flash Write Amplification , 2019, NSDI.

[38]  Bruce M. Maggs,et al.  Algorithmic Nuggets in Content Delivery , 2015, CCRV.

[39]  Chandra Krintz,et al.  Cache-conscious data placement , 1998, ASPLOS VIII.

[40]  Roy Friedman,et al.  Adaptive Software Cache Management , 2018, Middleware.

[41]  Michael M. Swift,et al.  FlashTier: a lightweight, consistent and durable storage cache , 2012, EuroSys '12.

[42]  George Varghese,et al.  An Improved Construction for Counting Bloom Filters , 2006, ESA.

[43]  Tony Tung,et al.  Scaling Memcache at Facebook , 2013, NSDI.

[44]  Nir Shavit,et al.  Flat combining and the synchronization-parallelism tradeoff , 2010, SPAA '10.

[45]  Werner Vogels,et al.  Dynamo: amazon's highly available key-value store , 2007, SOSP.

[46]  Kai Li,et al.  Learning Relaxed Belady for Content Distribution Network Caching , 2020, NSDI.

[47]  Songqing Chen,et al.  The stretched exponential distribution of internet media access patterns , 2008, PODC '08.

[48]  Willy Zwaenepoel,et al.  KVell: the design and implementation of a fast persistent key-value store , 2019, SOSP.

[49]  Ramesh K. Sitaraman,et al.  Overlay Networks: An Akamai Perspective , 2014 .

[50]  Cheng Li,et al.  Pannier: Design and Analysis of a Container-Based Flash Cache for Compound Objects , 2017, ACM Trans. Storage.

[51]  Kay A. Robbins,et al.  Practical UNIX programming: a guide to concurrency, communication, and multithreading , 1995 .

[52]  Bin Fan,et al.  Small cache, big effect: provable load balancing for randomly partitioned cluster services , 2011, SoCC.

[53]  Thomas F. Wenisch,et al.  µTune: Auto-Tuned Threading for OLDI Microservices , 2018, OSDI.

[54]  Dennis Shasha,et al.  2Q: A Low Overhead High Performance Buffer Management Replacement Algorithm , 1994, VLDB.

[55]  Qian Li,et al.  Arachne: Core-Aware Thread Management , 2018, OSDI.

[56]  Ramez Elmasri,et al.  Fundamentals of Database Systems , 1989 .

[57]  Ramesh K. Sitaraman,et al.  Trade-offs in optimizing the cache deployments of CDNs , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[58]  Chris Sears.  The Elements of Cache Programming Style , 2000, Annual Linux Showcase & Conference.

[59]  Dan Feng,et al.  Improving flash-based disk cache with Lazy Adaptive Replacement , 2013, 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST).

[60]  Gabriel H. Loh,et al.  PIPP: promotion/insertion pseudo-partitioning of multi-core shared caches , 2009, ISCA '09.

[61]  John Allen,et al.  Scuba: Diving into Data at Facebook , 2013, Proc. VLDB Endow..

[62]  Gu-Yeon Wei,et al.  Profiling a warehouse-scale computer , 2015, 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA).

[63]  Bruce M. Maggs,et al.  Protecting Websites from Attack with Secure Delivery Networks , 2015, Computer.

[64]  Yan Zhang,et al.  Empirical evaluation of multi-level buffer cache collaboration for storage systems , 2005, SIGMETRICS '05.

[65]  Robbert van Renesse,et al.  An analysis of Facebook photo caching , 2013, SOSP.

[66]  Li Zhang,et al.  NVMcached: An NVM-based Key-Value Cache , 2016, APSys.

[67]  Bruce M. Maggs,et al.  An analysis of live streaming workloads on the internet , 2004, IMC '04.

[68]  Song Jiang,et al.  Workload analysis of a large-scale key-value store , 2012, SIGMETRICS '12.

[69]  Xiaozhou Li,et al.  DistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed Caching , 2019, FAST.

[70]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[71]  Ramesh K. Sitaraman,et al.  Footprint Descriptors: Theory and Practice of Cache Provisioning in a Global CDN , 2017, CoNEXT.

[72]  Hui Ding,et al.  TAO: Facebook's Distributed Data Store for the Social Graph , 2013, USENIX Annual Technical Conference.

[73]  Irfan Ahmad,et al.  Cache Modeling and Optimization using Miniature Simulations , 2017, USENIX Annual Technical Conference.

[74]  Jack Brimberg,et al.  Location and sizing of facilities on a line , 2001 .

[75]  Zongpeng Li,et al.  Youtube traffic characterization: a view from the edge , 2007, IMC '07.