Minimizing Mean-Time-to-Recover in a Recursively Restartable Software System

This paper presents ideas on how to structure software systems for high availability by considering MTTR/MTTF characteristics of components in addition to the traditional criteria, such as functionality or state sharing. Recursive restartability (RR), a recently proposed technique for achieving high availability, exploits partial restarts at various levels within complex software infrastructures to recover from transient failures and rejuvenate software components. Here we refine the original proposal and apply the RR philosophy to Mercury, a COTS-based university satellite ground station that has been in operation for over 2 years. We develop four techniques for transforming component group boundaries such that time-to-repair is reduced, hence increasing system availability. We also further RR by defining the notions of an oracle, restart tree and restart group, while showing how to reason about system properties in terms of restart groups. From our experience with applying RR to Mercury, we draw design guidelines and lessons for the systematic application of recursive restartability to other software systems amenable to RR.