Impossible Certainty: Cost Risk Analysis for Air Force Systems

Abstract : This report is one of a series from a RAND *Project AIR FORCE* project, "The Cost of Future Military Aircraft: Historical Cost Estimating Relationships and Cost Reduction Initiatives." The purpose of the project is to improve the tools used to estimate the costs of future weapon systems. It focuses on how recent technical, management, and government policy changes affect cost. This report examines cost estimating risk analysis methods and recommends a policy prescription. The project was conducted within the RAND *Project AIR FORCE* Resource Management Program. The research is sponsored by the Principal Deputy, Office of the Assistant Secretary of the Air Force (Acquisition), Lt Gen John D.W. Corley. The project technical monitor is Jay Jordan, Technical Director of the Air Force Cost Analysis Agency. This report should interest government cost analysts, the military acquisition communities, and those concerned with current and future acquisition policies.

[1]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[2]  Peter W. G. Morris,et al.  The management of projects , 1994 .

[3]  G. K. Eagleson,et al.  Transformations for Smooth Regression Models with Multiplicative Errors , 1997 .

[4]  John Haigh,et al.  Probabilistic Risk Analysis: Foundations and Methods , 2003 .

[5]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[6]  Gary Klein,et al.  Sources of Power: How People Make Decisions , 2017 .

[7]  R. Hogarth,et al.  Ambiguity and Uncertainty in Probabilistic Inference. , 1985 .

[8]  Harry M. Markowitz,et al.  The Early History of Portfolio Theory: 1600–1960 , 1999 .

[9]  Paul McMahon,et al.  Risk Management Guide for DoD Acquisition , 1998 .

[10]  Division on Earth.  Risk Assessment in the Federal Government: Managing the Process , 1983 .

[11]  L. Gold,et al.  Ranking possible carcinogenic hazards. , 1987, Science.

[12]  B. Fischhoff,et al.  Rating the Risks , 1979 .

[13]  A. Tversky,et al.  Rational choice and the framing of decisions , 1990 .

[14]  Nancy Gaston,et al.  Guidelines: Writing for Adults with Limited Reading Skills. , 1988 .

[15]  Colin M. Sorrill.  Risk Analysis for Large Projects: Models, Methods and Cases , 1987 .

[16]  William Meckling,et al.  Predictability of the Costs, Time, and Success of Development , 1962 .

[17]  M. Degroot.  Optimal Statistical Decisions , 1970 .

[18]  A. Myrick Freeman,et al.  The benefits of environmental improvement , 1979 .

[19]  Michael R. Taylor,et al.  Application of risk assessment to food safety decision making. , 1983, Regulatory toxicology and pharmacology : RTP.

[20]  Tom DeMarco,et al.  Controlling Software Projects: Management, Measurement, and Estimates , 1986 .

[21]  Daniel M. Norton,et al.  An Analysis of Weapon System Cost Growth , 1993 .

[22]  Barney B. Roberts,et al.  Risk-Based Decision Support Techniques for Programs and Projects , 2003 .

[23]  Norman C. Rasmussen,et al.  The Application of Probabilistic Risk Assessment Techniques to Energy Technologies , 1981 .

[24]  James O. Berger,et al.  STATISTICAL DECISION THEORY: FOUNDATIONS, CONCEPTS, AND METHODS , 1984 .

[25]  B. Johnson.  Risk Communication: A Mental Models Approach , 2002 .

[26]  Daniel Zelterman,et al.  Bayesian Artificial Intelligence , 2005, Technometrics.

[27]  P Slovic,et al.  Informing and educating the public about risk. , 1986, Risk analysis : an official publication of the Society for Risk Analysis.

[28]  S. R. Jammalamadaka,et al.  Against the Gods: The Remarkable Story of Risk , 1999 .

[29]  Daniel M. Norton,et al.  The Defense System Cost Performance Database: Cost Growth Analysis Using Selected Acquisition Reports , 1996 .

[30]  D. Berry.  Statistical Decision Theory, Foundations, Concepts, and Methods , 1982 .

[31]  Rachelle D. Hollander,et al.  Acceptable Evidence: Science and Values in Risk Management , 1994 .

[32]  S. Sobel.  A COMPUTERIZED TECHNIQUE TO EXPRESS UNCERTAINTY IN ADVANCED SYSTEM COST ESTIMATES , 1965 .

[33]  A. Wildavsky.  No Risk Is the Highest Risk of All , 1979 .

[34]  Vincent T. Covello,et al.  Benefits assessment : the state of the art , 1986 .

[35]  Kristine M. Kuhn,et al.  The Relative Importance of Probabilities, Outcomes, and Vagueness in Hazard Risk Decisions , 1996 .

[36]  A. Roy.  SAFETY-FIRST AND HOLDING OF ASSETS , 1952 .

[37]  Ellen Coopersmith.  Making Decisions in the Oil and Gas Industry , 2001 .

[38]  D. Ellsberg.  Decision, probability, and utility: Risk, ambiguity, and the Savage axioms , 1961 .

[39]  R. Zerbe,et al.  Benefit-cost analysis in theory and practice , 2002 .

[40]  Tim L. Tinker,et al.  An evaluation primer on health risk communication programs and outcomes , 1997 .

[41]  Paul F. Dienemann,et al.  ESTIMATING COST UNCERTAINTY USING MONTE CARLO TECHNIQUES , 1966 .

[42]  Anthony O'Hagan,et al.  Eliciting expert beliefs in substantial practical applications , 1998 .

[43]  H. Sapolsky.  The Polaris System Development: Bureaucratic and Programmatic Success in Government , 1972 .

[44]  Robin M. Hogarth,et al.  Cognitive Processes and the Assessment of Subjective Probability Distributions , 1975 .

[45]  E. H. Conrow,et al.  Effective Risk Management: Some Keys to Success , 2003 .

[46]  Stephen A. Book,et al.  General-Error Regression for Deriving Cost-Estimating Relationships , 1997 .

[47]  Susan A. Resetar,et al.  Advanced Airframe Structural Materials: A Primer and Cost Estimating Methodology , 1991 .

[48]  Gene A. Venzke.  Implementation of Risk Assessment in the Total Risk Assessing Cost Estimate (Trace) , 1977 .

[49]  N A Connelly,et al.  Evaluating Risk Communication: Examining Target Audience Perceptions About Four Presentation Formats for Fish Consumption Health Advisory Information , 1998, Risk analysis : an official publication of the Society for Risk Analysis.

[50]  Fred Raymond.  Quantify Risk to Manage Cost and Schedule , 1999 .

[51]  Craig R. Fox,et al.  Ambiguity Aversion and Comparative Ignorance , 1995 .

[52]  M. Morris.  Understanding Risk - Informing Decisions in a Democratic Society , 1997 .

[53]  Daniel M Utech,et al.  Understanding Cost and Schedule Growth in Acquisition Programs , 1994 .

[54]  Shari Lawrence Pfleeger,et al.  Software Cost Estimation and Sizing Methods, Issues, and Guidelines , 2005 .

[55]  K. T. Wallenius.  Cost Uncertainty Assessment Methodology: A Critical Overview , 1985 .

[56]  Christopher W. Myers,et al.  Understanding cost growth and performance shortfalls in pioneer process plants , 1981 .

[57]  R. W. Hess,et al.  Aircraft Airframe Cost Estimating Relationships. Study Approach and Conclusions , 1987 .

[58]  John Birkler,et al.  Refueling and Complex Overhaul of the USS Nimitz (CVN 68): Lessons for the Future , 2003 .

[59]  Shlomo Zilberstein,et al.  Models of Bounded Rationality , 1995 .

[60]  David A. Bearden,et al.  Small-Satellite Costs , 2006 .

[61]  A. Tversky,et al.  On the elicitation of preferences for alternative therapies. , 1982, The New England journal of medicine.

[62]  D. Kahneman,et al.  Delusions of success. How optimism undermines executives' decisions. , 2003, Harvard business review.

[63]  Dietram A. Scheufele,et al.  Risk communication: A mental models approach , 2003 .

[64]  J. Kadane,et al.  Experiences in elicitation , 1998 .

[65]  Paul G Hough.  Pitfalls in Calculating Cost Growth from Selected Acquisition Reports , 1992 .

[66]  James J. Rooney,et al.  Root cause analysis for beginners , 2004 .

[67]  James E. Diekmann,et al.  Assessing Cost Uncertainty: Lessons from Environmental Restoration Projects , 1998 .

[68]  D. Vose.  Risk Analysis: A Quantitative Guide , 2000 .

[69]  William E. Kastenberg,et al.  Dealing with uncertainty arising out of probabilistic risk assessment , 1983 .

[70]  Lawrence J Klementowski,et al.  PERT/CPM and Supplementary Analytical Techniques. An Analysis of Aerospace Usage , 1978 .

[71]  Robert Perry,et al.  System Acquisition Strategies , 1971 .

[72]  S. Kelman,et al.  Cost-benefit analysis: an ethical critique. , 1981, Regulation.

[73]  David Novick,et al.  Reducing lead-time through improved technological forecasting: Some specific suggestions for more usefully formulated projections of technological availability☆ , 1969 .

[74]  Bill Shepherd,et al.  Managing Risk in a Program Office Environment , 2003 .

[75]  G. H. Fisher,et al.  Cost Considerations in Policy Analysis , 1975 .