Exploring quality measures for the evaluation of feature models: a case study

Abstract Evaluating the quality of a feature model is essential to ensure that errors in the early stages do not spread throughout the Software Product Line (SPL). One way to evaluate the feature model is to use measures that could be associated with the feature model quality characteristics and their quality attributes. In this paper, we aim at investigating how measures can be applied to the quality assessment of SPL feature models. We performed an exploratory case study using the COfFEE maintainability measures catalog and the S.P.L.O.T. feature models repository. In order to support this case study, we built a dataset (denoted by MAcchiATO) containing the values of 32 measures from COfFEE for 218 software feature models, extracted from S.P.L.O.T. This research approach allowed us to explore three different data analysis techniques. First, we applied the Spearman’s rank correlation coefficient in order to identify relationships between the measures. This analysis showed that not all 32 measures in COfFEE are necessary to reveal the quality of a feature model and just 15 measures could be used. Next, the 32 measures in COfFEE were grouped by applying the Principal Component Analysis and a set of 9 new grouped measures were defined. Finally, we used the Tolerance Interval technique to define statistical thresholds for these 9 new grouped measures. So, our findings suggest that measures can be effectively used to support the quality evaluation of SPL feature models.

[1]  Goiuria Sagardui Mendieta,et al.  Quality aware software product line engineering , 2010, Journal of the Brazilian Computer Society.

[2]  Browne,et al.  Cross-Validation Methods. , 2000, Journal of mathematical psychology.

[3]  Dragan Gasevic,et al.  Decision support for the software product line domain engineering lifecycle , 2011, Automated Software Engineering.

[4]  Silvia Mara Abrahão,et al.  Gathering current knowledge about quality evaluation in software product lines , 2009, SPLC.

[5]  Juha Savolainen,et al.  Identifying improvement potential in evolving product line infrastructures: 3 case studies , 2012, SPLC '12.

[6]  Eduardo Figueiredo,et al.  A Method to Derive Metric Thresholds for Software Product Lines , 2015, 2015 29th Brazilian Symposium on Software Engineering.

[7]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[8]  Kyo Chul Kang,et al.  Feature-Oriented Domain Analysis (FODA) Feasibility Study , 1990 .

[9]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[10]  Coral Calero,et al.  Software reliability modeling based on ISO/IEC SQuaRE , 2016, Inf. Softw. Technol..

[11]  Goiuria Sagardui,et al.  Evaluation of Quality Attribute Variability in Software Product Families , 2008, 15th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ecbs 2008).

[12]  Douglas C. Schmidt,et al.  Evolving feature model configurations in software product lines , 2014, J. Syst. Softw..

[13]  Cláudia Maria Lima Werner,et al.  MobiLine: A Nested Software Product Line for the domain of mobile and context-aware applications , 2013, Sci. Comput. Program..

[14]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[15]  Goiuria Sagardui,et al.  Quality Assessment in Software Product Lines , 2008, ICSR 2008.

[16]  David A. Gustafson,et al.  Shotgun correlations in software measures , 1993, Softw. Eng. J..

[17]  Sungwon Kang,et al.  Extending ATAM to assess product line architecture , 2008, 2008 8th IEEE International Conference on Computer and Information Technology.

[18]  Douglas C. Schmidt,et al.  Automated diagnosis of feature model configurations , 2010, J. Syst. Softw..

[19]  Paul Clements,et al.  Software product lines - practices and patterns , 2001, SEI series in software engineering.

[20]  Age K. Smilde,et al.  Principal Component Analysis , 2003, Encyclopedia of Machine Learning.

[21]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[22]  Dietmar Pfahl,et al.  Reporting guidelines for controlled experiments in software engineering , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[23]  Leonor Barroca,et al.  Systematic Evaluation of Software Product Line Architectures , 2013, J. Univers. Comput. Sci..

[24]  H. Abdi,et al.  Principal component analysis , 2010 .

[25]  Barbara A. Kitchenham,et al.  A quantitative approach to monitoring software development , 1989, Softw. Eng. J..

[26]  Katharine G. Shepherd,et al.  Exploring the Research , 2017 .

[27]  Vojislav B. Misic,et al.  A holistic architecture assessment method for software product lines , 2007, Inf. Softw. Technol..

[28]  José Maria Monteiro,et al.  Analyzing the Feature Models Maintainability over their Evolution Process: An Exploratory Study , 2016, VaMoS.

[29]  Liming Zhu,et al.  Evaluating guidelines for reporting empirical software engineering studies , 2008, Empirical Software Engineering.

[30]  Goiuria Sagardui Mendieta,et al.  Quality Assessment in Software Product Lines , 2008, ICSR.

[31]  Kyo Chul Kang,et al.  Usage Context as Key Driver for Feature Selection , 2010, SPLC.

[32]  Sergio Segura,et al.  FAMA: Tooling a Framework for the Automated Analysis of Feature Models , 2007, VaMoS.

[33]  Vytautas Štuikys,et al.  MEASURING COMPLEXITY OF DOMAIN MODELS REPRESENTED BY FEATURE DIAGRAMS , 2015 .

[34]  Yuqing Lin,et al.  Quality attribute modeling and quality aware product configuration in software product lines , 2013, Software Quality Journal.

[35]  Alina A. von Davier,et al.  Cross-Validation , 2014 .

[36]  Eduardo Figueiredo,et al.  Defining metric thresholds for software product lines: a comparative study , 2015, SPLC.

[37]  Donald D. Cowan,et al.  S.P.L.O.T.: software product lines online tools , 2009, OOPSLA Companion.

[38]  Dragan Gasevic,et al.  Assessing the maintainability of software product line feature models using structural metrics , 2011, Software Quality Journal.

[39]  Sergio Segura,et al.  Automated analysis of feature models 20 years later: A literature review , 2010, Inf. Syst..

[40]  Yuqing Lin,et al.  Quality Attributes Assessment for Feature-Based Product Configuration in Software Product Line , 2010, 2010 Asia Pacific Software Engineering Conference.

[41]  José Maria Monteiro,et al.  Measures for Quality Evaluation of Feature Models , 2015, ICSR.

[42]  Eduardo Figueiredo,et al.  Detecting Code Smells in Software Product Lines -- An Exploratory Study , 2015, 2015 12th International Conference on Information Technology - New Generations.

[43]  Jonathon Shlens,et al.  A Tutorial on Principal Component Analysis , 2014, ArXiv.

[44]  C. Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[45]  M. Mari,et al.  Quality-Driven Architecture Design Method , 2002 .

[46]  Silvia Mara Abrahão,et al.  A systematic review of quality attributes and measures for software product lines , 2011, Software Quality Journal.

[47]  Kerstin Mueller,et al.  Software Product Line Engineering Foundations Principles And Techniques , 2016 .

[48]  Yucong Duan,et al.  Releasing the Power of Variability: Towards Constraint Driven Quality Assurance , 2013, 2013 Second IIAI International Conference on Advanced Applied Informatics.

[49]  Douglas G. Altman,et al.  Statistics with confidence: Confidence intervals and statistical guidelines . , 1990 .

[50]  Mustafa Degerli,et al.  Yazilim Muhendisligi ve Sistem Muhendisligi Sureclerinin Harmanlanmasi: ISO/IEC 12207: 2008'in Getirdikleri ve Degisikliklerin Etkilerine Yonelik Degerlendirmeler (Blending Software Engineering and Systems Engineering Processes: Changes Introduced by ISO/IEC 12207: 2008 and Evaluations on Effects of , 2017, UYMS.

[51]  Thorsten Berger,et al.  Towards system analysis with variability model metrics , 2014, VaMoS.

[52]  Steffen Thiel.  On the Definition of a Framework for an Architecting Process Supporting Product Family Development , 2001, PFE.

[53]  Silvia Mara Abrahão,et al.  A Multimodel for Integrating Quality Assessment in Model-Driven Engineering , 2012, 2012 Eighth International Conference on the Quality of Information and Communications Technology.

[54]  Evie McCrum-Gardner,et al.  Which is the correct statistical test to use? , 2008, The British journal of oral & maxillofacial surgery.

[55]  D. Janaki Ram,et al.  ReQuEst: Requirements-driven quality estimator , 2005, SOEN.

[56]  Lianping Chen,et al.  A systematic review of evaluation of variability management approaches in software product lines , 2011, Inf. Softw. Technol..

[57]  Goiuria Sagardui Mendieta,et al.  Model based analysis process for embedded software product lines , 2011, ICSSP '11.

[58]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.