Hot-patching a web server: A case study of ASAP code repair

Software updates are the current standard to respond to software bugs. The software developer provides an update fix that is then applied by the administrator: the binary is modified and the service is restarted. Restarting a service inevitably leads to downtime and service unavailability; in the case of a multithreaded installation of Apache, restart takes several seconds and depending on the load of the web server, several hundred or even thousand client requests will be rejected with an error. Given the cost of restarts, system administrators attempt to minimize the frequency of service restarts or postpone a restart until the next maintenance window. However, to ensure the integrity of the system, code repair must happen as soon as possible (ASAP). We describe here the effectiveness of an on-the-fly update system that provides ASAP repair by integrating dynamic patches with a sandbox based on dynamic binary translation. To investigate the feasibility of ASAP code repair, we analyze the software updates released for Apache 2.2 between Dec 1st, 2005 and Feb 18, 2013. The study shows that such a system allows patching 45 of 49 bugs at runtime. Of the 4 unpatchable bugs, 1 bug is not applicable to dynamic update mechanisms, and 3 bugs require a restart. Furthermore, a performance evaluation of the prototype implementation shows that our approach adds low execution overhead (below 7% for different configurations that request a 287kB file).

[1]  Jack W. Davidson,et al.  Safe virtual execution using software dynamic translation , 2002, 18th Annual Computer Security Applications Conference, 2002. Proceedings..

[2]  Haibo Chen,et al.  Live updating operating systems using virtualization , 2006, VEE '06.

[3]  Michael Hicks,et al.  Towards standardized benchmarks for Dynamic Software Updating systems , 2012, 2012 4th International Workshop on Hot Topics in Software Upgrades (HotSWUp).

[4]  Gustavo Alonso,et al.  Dynamic weaving for aspect-oriented programming , 2002, AOSD '02.

[5]  Ophir Frieder,et al.  Dynamic program updating: A software maintenance technique for minimizing software, downtime , 1989, J. Softw. Maintenance Res. Pract..

[6]  Gavin M. Bierman,et al.  Mutatis Mutandis: Safe and predictable dynamic software updating , 2007, TOPL.

[7]  Michael Hicks,et al.  Mutatis Mutandis : Safe and predictable dynamic software updating , 2007 .

[8]  M. Frans Kaashoek,et al.  Ksplice: automatic rebootless kernel updates , 2009, EuroSys '09.

[9]  Harish Patil,et al.  Pin: building customized program analysis tools with dynamic instrumentation , 2005, PLDI '05.

[10]  Bryan Ford,et al.  Vx32: Lightweight User-level Sandboxing on the x86 , 2008, USENIX Annual Technical Conference.

[11]  Andrew Schultz,et al.  OPUS: Online Patches and Updates for Security , 2005, USENIX Security Symposium.

[12]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[13]  Hovav Shacham,et al.  The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86) , 2007, CCS '07.

[14]  Thomas R. Gross,et al.  DynSec: On-the-fly Code Rewriting and Repair , 2013, HotSWUp.

[15]  Scott Nettles,et al.  Dynamic software updating , 2001, PLDI '01.

[16]  Bennet S. Yee,et al.  Native Client: A Sandbox for Portable, Untrusted x86 Native Code , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[17]  Iulian Neamtiu,et al.  Safe and timely updates to multi-threaded programs , 2009, PLDI '09.

[18]  Robert Wahbe,et al.  Efficient software-based fault isolation , 1994, SOSP '93.

[19]  Thomas R. Gross,et al.  Safe Loading - A Foundation for Secure Execution of Untrusted Programs , 2012, 2012 IEEE Symposium on Security and Privacy.

[20]  Derek Bruening,et al.  Secure Execution via Program Shepherding , 2002, USENIX Security Symposium.

[21]  Bernhard Plattner,et al.  Modelling the Security Ecosystem- The Dynamics of (In)Security , 2009, WEIS.

[22]  Stephen McCamant,et al.  Evaluating SFI for a CISC Architecture , 2006, USENIX Security Symposium.

[23]  Manuel Oriol,et al.  Practical dynamic software updating for C , 2006, PLDI '06.

[24]  Haibo Chen,et al.  POLUS: A POwerful Live Updating System , 2007, 29th International Conference on Software Engineering (ICSE'07).

[25]  Rida A. Bazzi,et al.  Immediate Multi-Threaded Dynamic Software Updates Using Stack Reconstruction , 2009, USENIX Annual Technical Conference.

[26]  Thomas R. Gross,et al.  Fine-grained user-space security through virtualization , 2011, VEE '11.