An empirical review of software process assessments

Abstract In the recent past there has been a rapid increase in the use of process assessments in the software industry. Such assessments play an important role in initiating and tracking process improvements. In addition, there has been a proliferation of best practice models that are the basis for the assessments. The purpose of this chapter is to review the empirical evidence that exists to date on the efficacy of process assessments, with a specific focus on the CMM for software and the emerging ISO/IEC 15504 international standard. The available evidence is quite extensive, and supports the claims that assessments can be an effective tool for process improvement. We also highlight gaps in the literature where further work needs to be done.

[1]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[2]  Hoi K. Suen,et al.  Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment , 1985 .

[3]  Richard L. Nolan,et al.  Managing the computer resource , 1973, Commun. ACM.

[4]  Barry Boehm,et al.  The effects of software process maturity on software development effort , 1997 .

[5]  Shari Lawrence Pfleeger,et al.  Design and analysis in software engineering: the language of case studies and formal experiments , 1994, SOEN.

[6]  Bacmground Bootstrap Bootstrap: Europe's assessment method , 1993, IEEE Software.

[7]  Khaled El Emam,et al.  Empirical Studies of Software Process Assessment Methods * † , 1999 .

[8]  Hans-Jürgen Kugler,et al.  Bootstrap: fine-tuning process assessment , 1994, IEEE Software.

[9]  Watts S. Humphrey,et al.  Software process improvement at Hughes Aircraft , 1991, IEEE Software.

[10]  Watts S. Humphrey,et al.  Characterizing the software process: a maturity framework , 1988, IEEE Software.

[11]  Dennis R. Goldenson,et al.  SPICE: an empiricist's perspective , 1995, Proceedings of Software Engineering Standards Symposium.

[12]  Taratip Suwannasart,et al.  Developing a Testing Maturity Model : Part I , 1996 .

[13]  Warren Harrison,et al.  Empirical software engineering : an international journal , 1996 .

[14]  Khaled El Emam,et al.  A Method for Instrumenting Software Evolution Processes and An Example Application , 1994 .

[15]  Khaled El Emam,et al.  Costs and Benefits of Software Process Improvement , 1997 .

[16]  Khaled El Emam,et al.  The reliability of ISO/IEC PDTR 15504 assessments , 1997, Softw. Process. Improv. Pract..

[17]  Kenneth L. Kraemer,et al.  Evolution and organizational information systems: an assessment of Nolan's stage model , 1984, CACM.

[18]  Khaled El Emam,et al.  Estimating the extent of standards use: the case of ISO/IEC 15504 , 2000, J. Syst. Softw..

[19]  Dennis F. Galletta,et al.  Some Cautions on the Measurement of User Information Satisfaction , 1989 .

[20]  SIWUW A. Raghvan Diffusing Software-Engineering Methods , 2001 .

[21]  Robert M. Flowe,et al.  A Correlational Study of the SEI's Capability Maturity Model and Software Development Performance in DoD Contracts. , 1994 .

[22]  Richard L. Nolan Thoughts about the fifth stage , 1975, DATB.

[23]  Mark C. Paulk,et al.  The Evolation of the SEI's Capacity Maturity Model for Software , 1995, Softw. Process. Improv. Pract..

[24]  Watts S. Humphrey,et al.  A method for assessing the software engineering capability of contractors , 1987 .

[25]  Terry Bollinger,et al.  A critical look at software capability evaluations , 1991, IEEE Software.

[26]  Peter G. W. Keen,et al.  Decision support systems : an organizational perspective , 1978 .

[27]  James D. Herbsleb,et al.  Conceptual simplicity meets organizational complexity: case study of a corporate metrics program , 1998, Proceedings of the 20th International Conference on Software Engineering.

[28]  N. H. Madhavji,et al.  An effort estimation model for implementing ISO 9001 , 1995, Proceedings of Software Engineering Standards Symposium.

[29]  Blake Ives,et al.  An empirical study of the impact of user involvement on system usage and information satisfaction , 1986, CACM.

[30]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[31]  Raymond Dion,et al.  Process improvement and the corporate balance sheet , 1993, IEEE Software.

[32]  Khaled El Emam,et al.  Validating the ISO/IEC 15504 Measure of Software Requirements Analysis Process Capability , 2000, IEEE Trans. Software Eng..

[33]  Paul N. Finlay,et al.  Perceptions of the Benefits From the Introduction of CASE: An Empirical Study , 1994, MIS Q..

[34]  Judith G. Brodman,et al.  Return on Investment (ROI) from Software Process Improvement as Measured by US Industry , 1995, Softw. Process. Improv. Pract..

[35]  R. Dion,et al.  Elements of a process-improvement program (software quality) , 1992, IEEE Software.

[36]  Mark C. Paulk,et al.  Capability Maturity Model for Software , 2001 .

[37]  Donald H. Drury,et al.  An Empirical Assessment of the Stages of DP Growth , 1983, MIS Q..

[38]  Taratip Suwannasart,et al.  Developing a Testing Maturity Model, Part II , 1996 .

[39]  Capers Jones,et al.  Assessment and control of software risks , 1994, Yourdon Press Computing Series.

[40]  Karen D. Prenger,et al.  Costs and Benefits of Software Process Improvement , 1997 .

[41]  R. Zwick,et al.  Another look at interrater agreement. , 1988, Psychological bulletin.

[42]  George Dadoun ISO 9000: a requirement for doing business , 1992, CASCON.

[43]  Niv Ahituv,et al.  Principles of information systems for management , 1982 .

[44]  J. Neter,et al.  Applied linear statistical models : regression, analysis of variance, and experimental designs , 1974 .

[45]  J. Bach Enough About Process: What We Need are Heroes , 1995, IEEE Softw..

[46]  J. G. Brodman,et al.  What small businesses and small organizations say about the CMM , 1994, Proceedings of 16th International Conference on Software Engineering.

[47]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[48]  Robert C. Goldstein,et al.  The Stage Hypothesis and Data Administration: Some Contradictory Evidence , 1981, ICIS.

[49]  W. A. Scott,et al.  Reliability of Content Analysis ; The Case of Nominal Scale Cording , 1955 .

[50]  Khaled El Emam,et al.  Measuring the success of requirements engineering processes , 1995, Proceedings of 1995 IEEE International Symposium on Requirements Engineering (RE'95).

[51]  Tridas Mukhopadhyay,et al.  The effects of software processes on meeting targets and quality , 1995, Proceedings of the Twenty-Eighth Annual Hawaii International Conference on System Sciences.

[52]  Bob McFeeley,et al.  IDEAL: A User's Guide for Software Process Improvement. , 1996 .

[53]  Caroline D. Buchman Software process improvement at AlliedSignal Aerospace , 1996, Proceedings of HICSS-29: 29th Hawaii International Conference on System Sciences.

[54]  Khaled El Emam,et al.  Spice: The Theory and Practice of Software Process Improvement and Capability Determination , 1997 .

[55]  Steven L. Alter,et al.  Decision support systems : current practice and continuing challenges , 1980 .

[56]  Susan Rosenbaum,et al.  Software improvements in an international company , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[57]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[58]  Shari Lawrence Pfleeger,et al.  Understanding and improving technology transfer in software engineering , 1999, J. Syst. Softw..

[59]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[60]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[61]  James D. Herbsleb,et al.  After the Appraisal: A Systematic Survey of Process Improvement, its Benefits, and Factors that Influence Success. , 1995 .

[62]  David H. Kitson,et al.  An analysis of SEI software process assessment results: 1987-1991 , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[63]  Leon J. Osterweil,et al.  Proceedings of the 16th international conference on Software engineering , 1994, ICSE 1994.

[64]  Watts S. Humphrey,et al.  Comments on 'A Critical Look' , 1991, IEEE Softw..

[65]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[66]  Donna K. Dunaway,et al.  CMM-Based Appraisal for Internal Process Improvement (CBA IPI): Method Description , 1996 .

[67]  Khaled El Emam,et al.  Elements of Software Process Assessment & Improvement , 1999 .

[68]  James D. Herbsleb,et al.  Benefits of CMM-Based Software Process Improvement: Initial Results , 1994 .

[69]  Jane Siegel,et al.  Interim Profile: Development and Trial of a Method to Rapidly Measure Software Engineering Maturity Status , 1994 .

[70]  Iris Vessey,et al.  The Effect of User Involvement on System Success: A Contingency Approach , 1988, MIS Q..

[71]  JonesCapers The economics of software process improvement , 1996 .

[72]  Sree Nilakanta,et al.  Measurement: A blueprint for theory-building in MIS , 1994, Inf. Manag..

[73]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[74]  Shari Lawrence Pfleeger,et al.  Evaluating software engineering standards , 1994, Computer.

[75]  William J. Doll,et al.  The test-retest reliability of user involvement instruments , 1994, Inf. Manag..

[76]  F. Coallier,et al.  Risk management in software product procurement , 1999 .

[77]  Bill Curtis,et al.  A field study of the software design process for large systems , 1988, CACM.

[78]  Khaled El Emam,et al.  Validating the ISO/IEC 15504 measures of software development process capability , 2000, J. Syst. Softw..

[79]  Dennis R. Goldenson,et al.  Interrater agreement in SPICE-based assessments: some preliminary results , 1996, Proceedings of Software Process 1996.

[80]  James D. Herbsleb,et al.  Software quality and the Capability Maturity Model , 1997, CACM.

[81]  Izak Benbasat,et al.  Impact of Organizational Maturity on Information System Skill Needs , 1980, MIS Q..

[82]  Ronald A. Radice,et al.  A Programming Process Study , 1999, IBM Syst. J..

[83]  Deborah A. Boehm-Davis,et al.  Empirical Studies of Programmers: Sixth Workshop , 1996 .

[84]  D. W. Drew,et al.  Tailoring the software engineering Institute's (SEI) Capability Maturity Model (CMM) to a software sustaining engineering organization , 1992, Proceedings Conference on Software Maintenance 1992.

[85]  Lionel C. Briand,et al.  Assessor agreement in rating SPICE processes , 1996, Softw. Process. Improv. Pract..

[86]  Art Hersh,et al.  Where's the Return on Process Improvement? , 1993, IEEE Softw..

[87]  Roger Bate,et al.  A Systems Engineering Capability Maturity Model, Version 1.0. , 1994 .

[88]  Blake Ives,et al.  The measurement of user information satisfaction , 1983, CACM.

[89]  Bob Smith,et al.  Modelling the reliability of SPICE based assessments , 1997, Proceedings of IEEE International Symposium on Software Engineering Standards.

[90]  Bill Brykczynski,et al.  Policy Assessment for the Software Process Maturity Model , 1992 .

[91]  Malcolm Munro,et al.  Growth stages of end user computing , 1988, CACM.

[92]  Khaled El Emam,et al.  Cost implications of interrater agreement for software process assessments , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[93]  Kwasi Amoako-Gyampah,et al.  User involvement and user satisfaction: An exploratory contingency model , 1993, Inf. Manag..

[94]  R. Alpert,et al.  Communications Through Limited-Response Questioning , 1954 .

[95]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[96]  Izak Benbasat,et al.  A critque of the stage hypothesis: theory and empirical evidence , 1984, CACM.

[97]  Richard L. Nolan,et al.  Managing the data resource function , 1974 .

[98]  Brian Hollocks,et al.  Managing Management Information Systems , 1978 .

[99]  Khaled El Emam,et al.  A field study of requirements engineering practices in information systems development , 1995, Proceedings of 1995 IEEE International Symposium on Requirements Engineering (RE'95).

[100]  Jae-On Kim,et al.  Factor Analysis: Statistical Methods and Practical Issues , 1978 .

[101]  Hossein Saiedian,et al.  SEI Capability Maturity Model's Impact on Contractors , 1995, Computer.

[102]  David Rugg Using a capability evaluation to select a contractor , 1993, IEEE Software.

[103]  D P Hartmann,et al.  Considerations in the choice of interobserver reliability estimates. , 1977, Journal of applied behavior analysis.

[104]  Henry C. Lucas,et al.  The stage hypothesis and the s-curve: some contradictory evidence , 1977, CACM.

[105]  Bob Smith,et al.  Evaluating the interrater agreement of process capability ratings , 1997, Proceedings Fourth International Software Metrics Symposium.

[106]  Khaled El Emam,et al.  The Internal Consistency of the ISO/IEC 15504 Software Process Capability Scale , 1998, IEEE METRICS.

[107]  R. Linn Educational measurement, 3rd ed. , 1989 .

[108]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[109]  R. Nolan,et al.  Managing the Four Stages of EDP Growth , 1974 .

[110]  Werner Mellis,et al.  Success factors of organizational change in software process improvement , 1998, Softw. Process. Improv. Pract..

[111]  Izak Benbasat,et al.  Empirical Research in Information Systems: The Practice of Relevance , 1999, MIS Q..

[112]  Capers Jones,et al.  Gaps in SEI programs , 1995 .

[113]  Norman Fenton,et al.  Towards the evaluation of software engineering standards , 1993, Proceedings 1993 Software Engineering Standards Symposium.

[114]  Frank E. McGarry,et al.  Measuring the impacts individual process maturity attributes have on software products , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[115]  Khaled El Emam,et al.  An Empirical Evaluation of the Prospective International SPICE Standard , 1996, Softw. Process. Improv. Pract..

[116]  L. Cronbach Coefficient alpha and the internal structure of tests , 1951 .

[117]  Vijay Sethi,et al.  Construct Measurement in Information Systems Research: An Illustration in Strategic Systems , 1991 .

[118]  Mayuram S. Krishnan,et al.  Measuring Process Consistency: Implications for Reducing Software Defects , 1999, IEEE Trans. Software Eng..

[119]  Howard A. Rubin,et al.  Software process maturity: measuring its impact on productivity and quality , 1993, [1993] Proceedings First International Software Metrics Symposium.

[120]  Jean-François Grégoire,et al.  The stage hypothesis revisited: An EDP professionals' point of view , 1993, Inf. Manag..

[121]  Stephen M. Masters,et al.  CMM Appraisal Framework, Version 1.0 , 1995 .

[122]  John M. Ward,et al.  Principles of Information Systems Management , 1995 .

[123]  Sang-Hoon Kim,et al.  The relationship between procedural formalization in MIS development and MIS success: A contingent analysis , 1992, Inf. Manag..

[124]  Sasa M. Dekleva,et al.  Measuring Software Engineering Maturity: A Rasch Calibration , 1993, ICIS.

[125]  Richard Messnarz,et al.  Better Software Practice for Business Benefit: Principles and Experiences , 1999 .

[126]  S. B. Kiselev,et al.  The capability maturity model: guidelines for improving the software process , 1995 .

[127]  Richard H. Thayer,et al.  Software Engineering: A European Perspective , 1993 .

[128]  Judi Peterson,et al.  Software Metrics Capability Evaluation Guide , 1995 .

[129]  Fred D. Davis Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[130]  David Zubrow,et al.  Moving On Up: Data and Experience Doing CMM-Based Process Improvement , 1995 .