Recovery in Distributed Systems Using Optimistic Message Logging and Checkpointing

Message logging and checkpointing can provide fault tolerance in distributed systems in which all process communication is through messages. This paper presents a general model for reasoning about recovery in these systems. Using this model, we prove that the set of recoverable system states that have occurred during any single execution of the system forms a lattice, and that therefore, there is always a unique maximum recoverable system state, which never decreases. Based on this model, we present an algorithm for determining this maximum recoverable state and prove its correctness. Our algorithm utilizes all logged messages and checkpoints, and thus always finds the maximum recoverable state possible. Previous recovery methods using optimistic message logging and checkpointing have not considered the existing checkpoints, and thus may not find this maximum state. Furthermore, by utilizing the checkpoints, some messages received by a process before it was checkpointed may not need to be logged. Using our algorithm also adds less communication overhead to the system than do previous methods. Our model and algorithm can be used with any message logging protocol, whether pessimistic or optimistic, but their full generality is only required with

[1]  Brian Randell System structure for software fault tolerance , 1975 .

[2]  Wolfgang Graetsch,et al.  Fault tolerance under UNIX , 1989, TOCS.

[3]  David R. Jefferson,et al.  Virtual time , 1985, ICPP.

[4]  Robert E. Strom,et al.  Optimistic recovery in distributed systems , 1985, TOCS.

[5]  David L. Presotto,et al.  Publishing: a reliable broadcast communication mechanism , 1983, SOSP '83.

[6]  RICHARD KOO,et al.  Checkpointing and Rollback-Recovery for Distributed Systems , 1986, IEEE Transactions on Software Engineering.

[7]  Leslie Lamport,et al.  Distributed snapshots: determining global states of distributed systems , 1985, TOCS.

[8]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[9]  Phil Hontalas,et al.  Distributed Simulation and the Time Wrap Operating System. , 1987, SOSP 1987.

[10]  Barbara Liskov,et al.  Reliable object storage to support atomic actions , 1983, SOSP 1985.

[11]  David L. Russell,et al.  State Restoration in Systems of Communicating Processes , 1980, IEEE Transactions on Software Engineering.

[12]  A. Prasad Sistla,et al.  Efficient distributed recovery using message logging , 1989, PODC '89.

[13]  Fred B. Schneider,et al.  Implementing fault-tolerant services using the state machine approach: a tutorial , 1990, CSUR.

[14]  Richard D. Schlichting,et al.  Fail-stop processors: an approach to designing fault-tolerant computing systems , 1983, TOCS.

[15]  Butler W. Lampson,et al.  Crash Recovery in a Distributed Data Storage System , 1981 .

[16]  Anita Borg,et al.  A message system supporting fault tolerance , 1983, SOSP '83.

[17]  David B. Johnson,et al.  Sender-Based Message Logging , 1987 .

[18]  Bruce G. Lindsay,et al.  Transaction management in the R* distributed database management system , 1986, TODS.