Does Calibration Improve Prediction Accuracy

Department of Defense (DoD) and other government agencies. To better understand and control these costs, agencies often use parametric cost models for software development cost and schedule estimation. However, the accuracy of these models is poor when the default values embedded in the models are used [1]. Even after the software cost models are calibrated to DoD databases, most have been shown to be accurate to within only 25 percent of actual cost or schedule about half the time. For example, Robert Thibodeau [2] reported accuracy of early versions of the PRICES and SLIM models to be within 25 and 30 percent, respectively, on military ground programs. The IIT Research Institute [3] reported similar results on eight Ada programs, with the most accurate model at only 30 percent of actual cost or schedule, 62 percent of the time. Furthermore, the level of accuracy reported by these studies is likely overstated because most studies have failed to use holdout samples to validate the calibrated models. Instead of reserving a sample of the database for validation, the same data used to calibrate the models were used to assess accuracy [4]. In a study using 28 military ground software data points, Gerald Ourada [5] showed that failure to use a holdout sample overstates a model's accuracy. Half of the data was used to calibrate the Air Force's REVIC model. The remaining half was used to validate the calibrated model. REVIC was accurate to within 30 percent, 57 percent of the time on the calibration subset, but only 28 percent of the time on the validation subset. Validating on a holdout sample is clearly more relevant because new programs being estimated are, by definition, not in the calibration database. The purpose of this project was to calibrate and properly evaluate the accuracy of selected software cost estimation models using holdout samples. The expectation is that calibration improves the estimating accuracy of a model [6]. This paper describes the results of a long-term project at the Air Force Institute of Technology to calibrate and validate selected software cost estimation models. Two Air Force product centers provided software databases: the Space and Missile Systems Center (SMC), and the Electronic Systems Center (ESC). The project has been nicknamed the " Decalogue project " because 10 masters' theses extensively document the procedures and results of calibrating each software cost estimation model. The Decalogue project is organized into three …