Reliability Measurement, Analysis, and Improvement for Large Software Systems

Abstracts This chapter surveys recent developments in software reliability engineering, and presents results in measuring test activities, analyzing product reliability, and identifying high risk areas for focused reliability improvement for large software systems. Environmental constraints and existing analysis techniques are carefully examined to select appropriate analysis techniques, models, implementation strategies, and support tools suitable for large software systems. The analysis and modeling activities include measure definition, data gathering, graphical test activity and defect tracking, overall quality assessment using software reliability growth models, and identification of problematic areas and risk management using tree-based reliability models. Various existing software tools have been adapted and integrated to support these analysis and modeling activities. This approach has been used in the testing phase of several large software products developed in the IBM Software Solutions Toronto Laboratory and was demon- strated to be effective and efficient. Various practical problems and solutions in implementing this strategy are also discussed.

[1]  J. Voas,et al.  Software Testability: The New Verification , 1995, IEEE Softw..

[2]  B. Boehm Software risk management: principles and practices , 1991, IEEE Software.

[3]  Peng Lu,et al.  An integrated environment for software reliability modeling , 1993, Proceedings of 1993 IEEE 17th International Computer Software and Applications Conference COMPSAC '93.

[4]  Walter Freiberger,et al.  Statistical Computer Performance Evaluation , 1972 .

[5]  William Farr,et al.  Software reliability modeling survey , 1996 .

[6]  Jeff Tian,et al.  Data partition based reliability modeling , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[7]  Marvin V. Zelkowitz,et al.  Complexity Measure Evaluation and Selection , 1995, IEEE Trans. Software Eng..

[8]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[9]  John D. Musa,et al.  A theory of software reliability and its application , 1975, IEEE Transactions on Software Engineering.

[10]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[11]  Allan R. Wilks,et al.  The new S language: a programming environment for data analysis and graphics , 1988 .

[12]  Z. Jelinski,et al.  Software reliability Research , 1972, Statistical Computer Performance Evaluation.

[13]  Willa K. Ehrlich,et al.  Advances in Software Reliability Engineering , 1996, Adv. Comput..

[14]  Elaine J. Weyuker,et al.  The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software , 1995, IEEE Trans. Software Eng..

[15]  Amrit L. Goel,et al.  Software Reliability Models: Assumptions, Limitations, and Applicability , 1985, IEEE Transactions on Software Engineering.

[16]  J. R. Brown,et al.  Testing for software reliability , 1975 .

[17]  Anneliese von Mayrhauser Software Engineering: Methods and Management , 1990 .

[18]  Peng Lu,et al.  Test-Execution-Based Reliability Measurement and Modeling for Large Commercial Software , 1995, IEEE Trans. Software Eng..

[19]  Adam A. Porter,et al.  Empirically guided software development using metric-based classification trees , 1990, IEEE Software.

[20]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[21]  Adam A. Porter,et al.  An improved classification tree analysis of high cost modules based upon an axiomatic definition of complexity , 1992, [1992] Proceedings Third International Symposium on Software Reliability Engineering.

[22]  Victor R. Basili,et al.  The TAME Project: Towards Improvement-Oriented Software Environments , 1988, IEEE Trans. Software Eng..

[23]  Bev Littlewood,et al.  A Bayesian Reliability Growth Model for Computer Software , 1973 .

[24]  William N. Venables,et al.  Modern Applied Statistics with S-Plus. , 1996 .

[25]  Jesse H. Poore,et al.  Markov analysis of software specifications , 1993, TSEM.

[26]  Trevor Hastie,et al.  Statistical Models in S , 1991 .

[27]  Jeff Tian,et al.  An Integrated Approach to Test Tracking and Analysis , 1996, J. Syst. Softw..

[28]  Jeff Tian,et al.  Test workload measurement and reliability analysis for large commercial software systems , 1997, Ann. Softw. Eng..

[29]  Simeon C. Ntafos,et al.  On Some Reliability Estimation Problems in Random and Partition Testing , 1993, IEEE Trans. Software Eng..

[30]  Norman F. Schneidewind,et al.  Software Reliability Model with Optimal Selection of Failure Data , 1993, IEEE Trans. Software Eng..

[31]  Hoang Pham Software Reliability , 1999 .

[32]  Marvin V. Zelkowitz Role of Verification in the Software Specification Process , 1993, Adv. Comput..

[33]  William S. Cleveland,et al.  Visualizing Data , 1993 .

[34]  Daryl Pregibon,et al.  Tree-based models , 1992 .

[35]  Vernon Rego,et al.  A case study to investigate sensitivity of reliability estimates to errors in operational profile , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[36]  Jeff Tian,et al.  Integrating Time Domain and Input Domain Analyses of Software Reliability Using Tree-Based Models , 1995, IEEE Trans. Software Eng..

[37]  Norman F. Schneidewind Analysis of error processes in computer software , 1975 .

[38]  W D Brooks,et al.  Analysis of Discrete Software Reliability Models. , 1980 .

[39]  Jeff Tian,et al.  Tool support for software measurement, analysis and improvement , 1997, J. Syst. Softw..

[40]  Adam A. Porter,et al.  Learning from Examples: Generation and Evaluation of Decision Trees for Software Resource Analysis , 1988, IEEE Trans. Software Eng..

[41]  Martin L. Shooman,et al.  Probabilistic Models for Software reliability Prediction , 1972, Statistical Computer Performance Evaluation.

[42]  Shari Lawrence Pfleeger,et al.  Software Quality: The Elusive Target , 1996, IEEE Softw..

[43]  Marvin V. Zelkowitz,et al.  SEL's Software Process Improvement Program , 1995, IEEE Softw..

[44]  George J. Schick,et al.  An Analysis of Competing Software Reliability Models , 1978, IEEE Transactions on Software Engineering.

[45]  Victor R. Basili,et al.  A reference architecture for the component factory , 1992, TSEM.

[46]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[47]  Watts S. Humphrey,et al.  Managing the software process , 1989, The SEI series in software engineering.

[48]  Michael R. Lyu,et al.  An empirical study of the correlation between code coverage and reliability estimation , 1996, Proceedings of the 3rd International Software Metrics Symposium.

[49]  Boris Beizer,et al.  Black Box Testing: Techniques for Functional Testing of Software and Systems , 1996, IEEE Software.

[50]  Alberto Pasquini,et al.  Sensitivity of reliability-growth models to operational profile errors vs. testing accuracy [software testing] , 1996, IEEE Trans. Reliab..

[51]  Eldred Nelson,et al.  Estimating software reliability from test data , 1978 .

[52]  Victor R. Basili,et al.  Developing Interpretable Models with Optimized Set Reduction for Identifying High-Risk Software Components , 1993, IEEE Trans. Software Eng..

[53]  Jeff Tian,et al.  Measurement and defect modeling for a legacy software system , 1995, Ann. Softw. Eng..

[54]  Ram Chillarege,et al.  Reliability Growth for Typed Defects , 1992 .

[55]  Shigeru Yamada,et al.  S-Shaped Reliability Growth Modeling for Software Error Detection , 1983, IEEE Transactions on Reliability.

[56]  Eugene Miya,et al.  On "Software engineering" , 1985, SOEN.

[57]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[58]  Victor R. Basili,et al.  Software Quality: An Overview from the Perspective of Total Quality Management , 1994, IBM Syst. J..

[59]  Robert L. Glass,et al.  Measuring software design quality , 1990 .

[60]  Jeff Tian,et al.  Exploratory analysis tools for tree-based models in software measurement and analysis , 1996, Proceedings of the Fourth International Symposium on Assessment of Software Tools.

[61]  Amrit L. Goel,et al.  Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures , 1979, IEEE Transactions on Reliability.

[62]  Marvin V. Zelkowitz Modeling Software Engineering Environment Capabilities , 1996, J. Syst. Softw..

[63]  Ray E. Schafer,et al.  Validation of Software Reliability Models , 1979 .

[64]  John F. Passafiume,et al.  Software testing and evaluation , 1987 .

[65]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.