A comparison of reliability estimation methods for binary systems

This paper uses a simulation-based approach to compare the predictive accuracy of five different methods for estimating the risk of failure for binary failure/no failure systems such as US strategic missiles, space launch vehicles, and security systems based on the results of a number of tests. This paper tests two Bayesian approaches, two classical (frequentist) approaches, and the method currently used the US Air Force Strategic Command (STRATCOM) to estimate the reliability of strategic nuclear missiles. First, test results are simulated based on an assumed underlying reliability profile. Then the system's reliability is estimated by each of the approaches using the simulated test results, and these estimates are compared with the assumed underlying reliability. Statistical procedures are used to compare the errors from the different methods. The results of this study show that the STRATCOM approach and a classical approach using only the test data from the current period are significantly less accurate than the other three methods and that the accuracy of the Bayesian methods depend on the prior density functions used. The results in this paper provide a quantitative assessment of the accuracy of the tested methods.