Improvement Opportunities and Suggestions for Benchmarking

During the past 10 years, the amount of effort put on setting up benchmarking repositories has considerably increased at the organizational, national and even at international levels to help software managers to determine the performance of software activities and to make better software estimates. This has enabled a number of studies with an emphasis on the relationship between software product size, effort and cost drivers in order to either measure the average performance for similar software projects or to develop estimation models and then refine them using the collected data. However, despite these efforts, none of those methods are yet deemed to be universally applicable and there is still no agreement on which cost drivers are significant in the estimation process. This study discusses some of the possible reasons why in software engineering, practitioners and researchers have not yet been able to come up with reasonable and well quantified relationships between effort and cost drivers although considerable amounts of data on software projects have been collected. An improved classification of application types in benchmarking repositories is also proposed.

[1]  Ioannis Stamelos,et al.  Building a Cost Estimation Model Based on Categorical Data , 2001 .

[2]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[3]  R. C. Tausworthe,et al.  Deep space network software cost estimation model , 1981 .

[4]  Luigi Buglione,et al.  Strengthening CMMI Maturity Levels with a Quantitative Approach to Root-Cause Analysis , 2008 .

[5]  Robert E. Park,et al.  Software Size Measurement: A Framework for Counting Source Statements , 1992 .

[6]  S. Morasca,et al.  An empirical study of software productivity , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.

[7]  David N. Card,et al.  Benchmarking Software Organizations , 2001, IEEE Software.

[8]  Frank Bomarius,et al.  Get Your Experience Factory Ready for the Next Decade--Ten Years after "How to Build and Run One"-- , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[9]  T. Wright,et al.  Organizational Benchmarking Using the ISBSG Data Repository , 2001, IEEE Softw..

[10]  Barbara A. Kitchenham,et al.  An empirical analysis of software productivity over time , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[11]  Katrina Maxwell,et al.  Benchmarking Software-Development Productivity - Applied Research Results , 2000, IEEE Softw..

[12]  Ellis Horowitz,et al.  Software Cost Estimation with COCOMO II , 2000 .

[13]  Victor R. Basili,et al.  The software engineering laboratory - an operational software experience factory , 1992, International Conference on Software Engineering.

[14]  Building a software cost estimation model based on categorical data , 2001, Proceedings Seventh International Software Metrics Symposium.

[15]  Katrina Maxwell.  Collecting Data for Comparability: Benchmarking Software Development Productivity , 2001, IEEE Softw..

[16]  Bojan Cukic.  Guest Editor's Introduction: The Promise of Public Software Engineering Data Repositories , 2005, IEEE Softw..

[17]  Lawrence H. Putnam,et al.  A General Empirical Solution to the Macro Software Sizing and Estimating Problem , 1978, IEEE Transactions on Software Engineering.