Software Versioning and Quality Degradation? An Exploratory Study of the Evidence

We present a framework for measuring software quality using pricing and demand data, and empirical estimates that quantify the extent of quality degradation associated with software versioning. Using a 7-month, 108-product panel of software sales from Amazon.com, we document the extent to which quality varies across different software versions, estimating quality degradation that ranges from as little as 8% to as much as 56% below that of the corresponding flagship version. Consistent with prescriptions from the theory of vertical differentiation, we also find that an increase in the total number of versions is associated with an increase in the difference in quality between the highest and lowest quality versions, and a decrease in the quality difference between neighboring versions. We compare our estimates with those derived from two sets of subjective measures of quality, based on CNET editorial ratings and Amazon.com user reviews, and discuss competing interpretations of the significant differences that emerge from this comparison. As the first empirical study of software versioning that is based on both subjective and econometrically estimated measures of quality, this paper provides a framework for testing a wide variety of results that are based on related models of vertical differentiation, and its findings have important implications for studies that treat web-based user ratings as cardinal data.