An empirical validation of the relationship between the magnitude of relative error and project size

Cost estimates are important deliverables of a software project. Consequently, a number of cost prediction models have been proposed and evaluated. The common evaluation criteria have been MMRE, MdMRE and PRED(k). MRE is the basic metric in these evaluation criteria. The implicit rationale of using a relative error measure like MRE, rather than an absolute one, is presumably to have a measure that is independent of project size. We investigate if this implicit claim holds true for several data sets: Albrecht, Kemerer, Finnish, DMR and Accenture-ERP. The results suggest that MRE is not independent of project size. Rather, MRE is larger for small projects than for large projects. A practical consequence is that a project manager predicting a small project may falsely believe in a too low MRE. Vice versa when predicting a large project. For researchers, it is important to know that MMRE is not an appropriate measure of the expected MRE of small and large projects. We recommend therefore that the data set be partitioned into two or more subsamples and that MMRE is reported per subsample. In the long term, we should consider using other evaluation criteria.