Reliability survey of military acquisition systems

At the request of Director, Operational Test and Evaluation (DOT&E), the Institute for Defense Analyses (IDA) has conducted annual reliability surveys of DoD programs under DOT&E oversight since 2009 to provide a continuing understanding of the extent to which military programs are implementing reliability-focused DoD policy guidance and assess whether the implementation of this guidance is leading to improved reliability. This paper provides an assessment of the survey results. Overall survey results support the understanding that systems with a comprehensive reliability growth program are mo re likely to meet reliability goals in testing. In particular, the results show the importance of establishing and meeting Reliability, Availability, and Maintainability (RAM) entrance criteria before proceeding to operational testing (OT). While many programs did not establish or meet RAM entrance criteria, those that did were far more likely to demonstrate reliability at or above the required value during OT. Examples of effective RAM entrance criteria include (1) demonstrating in the last developmental test event prior to the OT a reliability point estimate that is consistent with the reliability growth curve, and (2) for automated information systems and software-intensive sensor and weapons systems, ensuring that there are no open Category 1 or 2 deficiency reports prior to OT. There is also evidence that having intermediate goals linked to the reliability growth curve improves the chance of meeting RAM entrance criteria. The survey results also indicate that programs are increasingly incorporating reliability-focused policy guidance, but despite these policy implementation improvements, many programs still fail to reach reliability goals. In other words, the policies have not yet proven effective at improving reliability trends. The reasons programs fail to reach reliability goals include inadequate requirements, unrealistic assumptions, lack of a design for reliability effort, and failure to employ a comprehensive reliability growth process. Although the DoD is in a period of new policy that emphasizes good reliability growth principles, without a consistent implementation of those principles, the reliability trend will remain flat.