Meta-learning based selection of software reliability models

The correct estimation of the software reliability level is fundamental to reduce efforts in the testing, maintenance and release activities. To help in this task, we find in the literature an increasing number of software reliability models (SRMs). However, none has proved to perform well considering different contexts. Due to this, the selection of the best model for a particular case is an important task. Most existing works on SRM selection need to test different models and decide based on how well the model fits the data and predicts the future events. Moreover, in general, they do not consider search-based models. Considering this fact, this paper introduces a Meta-learning approach for SRM selection. In such approach, some meta-features are used to indicate the best performing model. The approach is independent of the type of models to be selected, and can be used with different data mining algorithms. It includes the following activities: meta-knowledge extraction, meta-learning and classification. The activities meta-knowledge extraction and meta-learning are performed just once and generate a meta-classifier. Therefore, the meta-classifier is used to select the most adequate model for new projects (classification activity). The approach is evaluated in a set of experiments and the results do not show statistical difference between the Meta-learning approach and the choice of the best performing model. Otherwise, the results point out statistical difference between the Meta-learning approach and the choice of the worst performing model with a large stochastic difference according to the Vargha and Delaney Effect Size.

[1]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[2]  Eoca Raton.  SOFTWARE RELIABILITY MODEL SELECTION: A CASE STUDY , 1991 .

[3]  L. Darrell Whitley,et al.  Prediction of Software Reliability Using Connectionist Models , 1992, IEEE Trans. Software Eng..

[4]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[5]  Francisco de A. T. de Carvalho,et al.  A Modal Symbolic Classifier for selecting time series models , 2004, Pattern Recognit. Lett..

[6]  Teresa Bernarda Ludermir,et al.  Meta-learning approaches to selecting time series models , 2004, Neurocomputing.

[7]  Ganesh J. Pai.  A Survey of Software Reliability Models , 2013, ArXiv.

[8]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[9]  Paul B. Moranda,et al.  Predictions of software reliability during debugging , 1975 .

[10]  Stamatios V. Kartalopoulos,et al.  Understanding neural networks and fuzzy logic - basic concepts and applications , 1997 .

[11]  Aurora Trinidad Ramirez Pozo,et al.  Selecting software reliability models with a neural network meta classifier , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[12]  Michael R. Lyu,et al.  Handbook of software reliability engineering , 1996 .

[13]  Andreas S. Andreou,et al.  Maximum Entropy Discrimination Poisson Regression for Software Reliability Modeling , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Ricardo Vilalta,et al.  Metalearning - Applications to Data Mining , 2008, Cognitive Technologies.

[15]  Aurora Trinidad Ramirez Pozo,et al.  Exploring Genetic Programming and Boosting Techniques to Model Software Reliability , 2007, IEEE Transactions on Reliability.

[16]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[17]  Coral Calero,et al.  A Systematic Mapping Study of Software Reliability Modeling , 2014, Inf. Softw. Technol..

[18]  John D. Musa,et al.  A theory of software reliability and its application , 1975, IEEE Transactions on Software Engineering.

[19]  Shigeru Yamada,et al.  Software reliability modeling , 2014 .

[20]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[21]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[22]  R. Sitte.  Comparison of software-reliability-growth predictions: neural networks vs parametric-recalibration , 1999 .

[23]  Z. Jelinski,et al.  Software reliability Research , 1972, Statistical Computer Performance Evaluation.

[24]  Ritu Manderna.  Software Reliability Model Selection Criteria: A Literature Review , 2013 .

[25]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[26]  Muhammad Irfan Ullah,et al.  An approach for software reliability model selection , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[27]  Ricardo Vilalta,et al.  Meta-Learning - Concepts and Techniques , 2010, Data Mining and Knowledge Discovery Handbook.

[28]  John D. Musa,et al.  Software reliability - measurement, prediction, application , 1987, McGraw-Hill series in software engineering and technology.

[29]  Yumei Wu,et al.  Software reliability modeling based on SVM and virtual sample , 2013, 2013 Proceedings Annual Reliability and Maintainability Symposium (RAMS).

[30]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[31]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[32]  Lior Rokach,et al.  Data Mining And Knowledge Discovery Handbook , 2005 .

[33]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[34]  Alaa F. Sheta,et al.  Prediction of software reliability: a comparison between regression and neural network non-parametric models , 2001, Proceedings ACS/IEEE International Conference on Computer Systems and Applications.

[35]  Jongmoon Baik,et al.  Improving software reliability prediction through multi-criteria based dynamic model selection and combination , 2015, J. Syst. Softw..

[36]  Alex Alves Freitas,et al.  Contrasting meta-learning and hyper-heuristic research: the role of evolutionary algorithms , 2013, Genetic Programming and Evolvable Machines.

[37]  M. Topczewska,et al.  Data preprocessing in the classification of the imbalanced data , 2014 .

[38]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .

[39]  Vadlamani Ravi,et al.  A novel hybrid undersampling method for mining unbalanced datasets in banking and insurance , 2015, Eng. Appl. Artif. Intell..

[40]  Teresa Bernarda Ludermir,et al.  Selection of time series forecasting models based on performance information , 2004, Fourth International Conference on Hybrid Intelligent Systems (HIS'04).

[41]  BaikJongmoon,et al.  Improving software reliability prediction through multi-criteria based dynamic model selection and combination , 2015 .

[42]  山田 茂,et al.  Software reliability modeling : fundamentals and applications , 2014 .

[43]  Taghi M. Khoshgoftaar,et al.  Software reliability model selection: a cast study , 1991, Proceedings. 1991 International Symposium on Software Reliability Engineering.

[44]  A. Stuart,et al.  Non-Parametric Statistics for the Behavioral Sciences. , 1957 .

[45]  W. Afzal,et al.  Suitability of Genetic Programming for Software Reliability Growth Modeling , 2008, International Symposium on Computer Science and its Applications.