A mapping study on design-time quality attributes and metrics

Support to the quality attribute (QA) & metric selection process.Maintainability is the most studied QA for most domains and development phases.Quality attributes are usually assessed through a correlation to a single metric.Metrics are validated in empirical settings and may lack theoretical validity. Developing a plan for monitoring software quality is a non-trivial task, in the sense that it requires: (a) the selection of relevant quality attributes, based on application domain and development phase, and (b) the selection of appropriate metrics to quantify quality attributes. The metrics selection process is further complicated due to the availability of various metrics for each quality attribute, and the constraints that impact metric selection (e.g., development phase, metric validity, and available tools). In this paper, we shed light on the state-of-research of design-time quality attributes by conducting a mapping study. We have identified 154 papers that have been included as primary studies. The study led to the following outcomes: (a) low-level quality attributes (e.g., cohesion, coupling, etc.) are more frequently studied than high-level ones (e.g., maintainability, reusability, etc.), (b) maintainability is the most frequently examined high-level quality attribute, regardless of the application domain or the development phase, (c) assessment of quality attributes is usually performed by a single metric, rather than a combination of multiple metrics, and (d) metrics are mostly validated in an empirical setting. These outcomes are interpreted and discussed based on related work, offering useful implications to both researchers and practitioners.

[1]  Soo Dong Kim,et al.  A framework for evaluating reusability of core asset in product line engineering , 2007, Inf. Softw. Technol..

[2]  Amar Ramdane-Cherif,et al.  ISO quality standards for measuring architectures , 2004, J. Syst. Softw..

[3]  Heiko Koziolek,et al.  Quantifying software architecture quality report on the first international workshop on software architecture metrics , 2014, SOEN.

[4]  Rachel Harrison,et al.  Experimental assessment of the effect of inheritance on the maintainability of object-oriented systems , 2000, J. Syst. Softw..

[5]  Doo-Hwan Bae,et al.  Measuring behavioral dependency for improving change-proneness prediction in UML-based design models , 2010, J. Syst. Softw..

[6]  Hans van Vliet,et al.  Software engineering - principles and practice , 1993 .

[7]  Danilo Caivano,et al.  Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments , 2016, Empirical Software Engineering.

[8]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[9]  George Valença,et al.  Accepted Manuscript Requirements Engineering for Software Product Lines: a Systematic Literature Review Accepted Manuscript Requirements Engineering for Software Product Lines: a Systematic Literature Review Accepted Manuscript , 2022 .

[10]  Mario Piattini,et al.  A systematic mapping study on serious game quality , 2014, EASE '14.

[11]  Eduardo Figueiredo,et al.  On the relationship of concern metrics and requirements maintainability , 2012, Inf. Softw. Technol..

[12]  Craig Larman,et al.  Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development (3rd Edition) , 1997 .

[13]  Letha H. Etzkorn,et al.  Coupling metrics for ontology-based system , 2006, IEEE Software.

[14]  Brian Berenbach,et al.  Metrics for model driven requirements development , 2006, ICSE '06.

[15]  Pearl Brereton,et al.  The impact of limited search procedures for systematic literature reviews — A participant-observer case study , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[16]  Yuming Zhou,et al.  The ability of object-oriented metrics to predict change-proneness: a meta-analysis , 2011, Empirical Software Engineering.

[17]  Shinji Kusumoto,et al.  Function point measurement from Web application source code based on screen transitions and database accesses , 2011, J. Syst. Softw..

[18]  Ronald D. Gordon,et al.  A Qualitative Justification for a Measure of Program Clarity , 1979, IEEE Transactions on Software Engineering.

[19]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[20]  Pierre N. Robillard,et al.  The interconnectivity metrics: A new metric showing how a program is organized , 1989, J. Syst. Softw..

[21]  Frank Elberzhager,et al.  A systematic mapping study on the combination of static and dynamic quality assurance techniques , 2012, Inf. Softw. Technol..

[22]  Paulo Borba,et al.  An Estimation Model for Test Execution Effort , 2007, ESEM 2007.

[23]  Jeffrey C. Carver,et al.  Modifiability measurement from a task complexity perspective: A feasibility study , 2009, ESEM 2009.

[24]  Shinji Kusumoto,et al.  Function point measurement from Java programs , 2002, ICSE '02.

[25]  Andreas Vogelsang,et al.  Are "Non-functional" Requirements really Non-functional? An Investigation of Non-functional Requirements in Practice , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[26]  Yves Le Traon,et al.  Measuring design testability of a UML class diagram , 2005, Information and Software Technology.

[27]  Yuanfang Cai,et al.  Measuring architecture quality by structure plus history analysis , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[28]  Deepak Goyal,et al.  A hierarchical model for object-oriented design quality assessment , 2015 .

[29]  Qinghua Zheng,et al.  Exploring community structure of software Call Graph and its applications in class cohesion measurement , 2015, J. Syst. Softw..

[30]  Mika Mäntylä,et al.  Using metrics in Agile and Lean Software Development - A systematic literature review of industrial studies , 2015, Inf. Softw. Technol..

[31]  Maurice H. Halstead,et al.  Elements of software science , 1977 .

[32]  Sallie M. Henry,et al.  Object-oriented metrics that predict maintainability , 1993, J. Syst. Softw..

[33]  Dietmar P. F. Möller,et al.  Systems and Software Engineering , 2016 .

[34]  Avinash C. Kak,et al.  Some structural measures of API usability , 2015, Softw. Pract. Exp..

[35]  Abdul Azim Abdul Ghani,et al.  A mapping study to investigate component-based software system metrics , 2013, J. Syst. Softw..

[36]  Tibor Gyimóthy,et al.  Using information retrieval based coupling measures for impact analysis , 2009, Empirical Software Engineering.

[37]  Kai-Yuan Cai,et al.  An analysis of research topics in software engineering - 2006 , 2008, J. Syst. Softw..

[38]  Tsong Yueh Chen,et al.  An assessment of systems and software engineering scholars and institutions (1993-1997) , 1997, J. Syst. Softw..

[39]  James M. Bieman,et al.  Measuring Design-Level Cohesion , 1998, IEEE Trans. Software Eng..

[40]  Mario Piattini,et al.  A Survey of Metrics for UML Class Diagrams , 2005, J. Object Technol..

[41]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[42]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[43]  K. Vairavan,et al.  An Experimental Study of Software Metrics for Real-Time Software , 1985, IEEE Transactions on Software Engineering.

[44]  Caspar Ryan,et al.  A Controlled Experiment for Evaluating the Impact of Coupling on the Maintainability of Service-Oriented Software , 2011, IEEE Transactions on Software Engineering.

[45]  Roberto da Silva Bigonha,et al.  Identifying thresholds for object-oriented software metrics , 2012, J. Syst. Softw..

[46]  T.C. Lethbridge,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK) and the Software Engineering Education Knowledge (SEEK) - a preliminary mapping , 2001, 10th International Workshop on Software Technology and Engineering Practice.

[47]  Sven Apel,et al.  Exploring Software Measures to Assess Program Comprehension , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[48]  Mehwish Riaz,et al.  A systematic review of software maintainability prediction and metrics , 2009, ESEM 2009.

[49]  Saeed Jalili,et al.  A clustering-based model for class responsibility assignment problem in object-oriented analysis , 2014, J. Syst. Softw..

[50]  Sandro Morasca,et al.  Defining and Validating Measures for Object-Based High-Level Design , 1999, IEEE Trans. Software Eng..

[51]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[52]  Richard H. Carver,et al.  An Evaluation of the MOOD Set of Object-Oriented Software Metrics , 1998, IEEE Trans. Software Eng..

[53]  Coral Calero,et al.  A Systematic Mapping Study of Software Reliability Modeling , 2014, Inf. Softw. Technol..

[54]  So Young Sohn,et al.  A strategic analysis for successful open source software utilization based on a structural equation model , 2008, J. Syst. Softw..

[55]  C. van Koten,et al.  An application of Bayesian network for predicting object-oriented software maintainability , 2006, Inf. Softw. Technol..

[56]  Danny Weyns,et al.  Variability in Software Systems—A Systematic Literature Review , 2014, IEEE Transactions on Software Engineering.

[57]  Taghi M. Khoshgoftaar,et al.  Measurement of data structure complexity , 1993, J. Syst. Softw..

[58]  Robert G. Reynolds Metrics to measure the complexity of partial programs , 1984, J. Syst. Softw..

[59]  Elaine J. Weyuker,et al.  Evaluating Software Complexity Measures , 2010, IEEE Trans. Software Eng..

[60]  Fannie Cobben,et al.  An empirical validation of R-indicators , 2008 .

[61]  Letha H. Etzkorn,et al.  An empirical study of object-oriented system evolution , 2000, Inf. Softw. Technol..

[62]  Michael English,et al.  Fine-Grained Software Metrics in Practice , 2007, ESEM 2007.

[63]  Mikael Lindvall,et al.  An Empirically-Based Process for Software Architecture Evaluation , 2004, Empirical Software Engineering.

[64]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[65]  Richard Torkar,et al.  Software fault prediction metrics: A systematic literature review , 2013, Inf. Softw. Technol..

[66]  Muhammad Ali Babar,et al.  Identifying relevant studies in software engineering , 2011, Inf. Softw. Technol..

[67]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[68]  Andreas Vogelsang,et al.  Are "Non-functional" Requirements really Non-functional? , 2017, Software Engineering.

[69]  eva Kühn,et al.  Automated measurement of API usability: The API Concepts Framework , 2015, Inf. Softw. Technol..

[70]  Pearl Brereton,et al.  Refining the systematic literature review process—two participant-observer case studies , 2010, Empirical Software Engineering.

[71]  Sérgio Soares,et al.  Aspect-oriented software maintenance metrics: A systematic mapping study , 2012, EASE.

[72]  Robert C. Martin Agile Software Development, Principles, Patterns, and Practices , 2002 .

[73]  Xavier Franch,et al.  Quality models for web services: A systematic mapping , 2014, Inf. Softw. Technol..

[74]  Trevor T. Moores Applying complexity measures to rule-based prolog programs , 1998, J. Syst. Softw..

[75]  Taghi M. Khoshgoftaar,et al.  Predictive Modeling Techniques of Software Quality from Software Measures , 1992, IEEE Trans. Software Eng..

[76]  Shari Lawrence Pfleeger,et al.  Software Quality: The Elusive Target , 1996, IEEE Softw..

[77]  Albert L. Baker,et al.  A comparison of measures of control flow complexity , 1979, COMPSAC.

[78]  Rachel Harrison,et al.  Reusability and maintainability in hypermedia applications for education , 1998, Inf. Softw. Technol..

[79]  Frank W. Calliss,et al.  An information-hiding metric , 1994, J. Syst. Softw..

[80]  Paul J. Layzell,et al.  Spatial complexity metrics: an investigation of utility , 2005, IEEE Transactions on Software Engineering.

[81]  Sajjad Mahmood,et al.  A complexity measure for UML component-based system specification , 2008 .

[82]  Paul D. Scott,et al.  Ranking reusability of software components using coupling metrics , 2007, J. Syst. Softw..

[83]  K. K. Aggarwal,et al.  Code and data spatial complexity: two important software understandability measures , 2003, Inf. Softw. Technol..

[84]  Martha Grabowski,et al.  Complexity in embedded intelligent real time systems , 1999, ICIS.

[85]  Lionel C. Briand,et al.  A Precise Method-Method Interaction-Based Cohesion Metric for Object-Oriented Classes , 2012, TSEM.

[86]  Barbara Kitchenham,et al.  What's up with software metrics? - A preliminary mapping study , 2010, J. Syst. Softw..

[87]  Akif Günes Koru,et al.  Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products , 2005, IEEE Transactions on Software Engineering.

[88]  Claes Wohlin,et al.  Empirical evidence on the link between object-oriented measures and external quality attributes: a systematic literature review , 2013, Empirical Software Engineering.

[89]  Ioannis Stamelos,et al.  Layer assessment of object-oriented software: A metric facilitating white-box reuse , 2013, J. Syst. Softw..

[90]  A. Jefferson Offutt,et al.  An Empirical Evaluation , 1994 .

[91]  Letha H. Etzkorn,et al.  A comparison of cohesion metrics for object-oriented systems , 2004, Inf. Softw. Technol..

[92]  James M. Bieman,et al.  Program slices as an abstraction for cohesion measurement , 1998, Inf. Softw. Technol..

[93]  Wei Li,et al.  Another metric suite for object-oriented programming , 1998, J. Syst. Softw..

[94]  Yves Le Traon,et al.  Usage and testability of AOP: An empirical study of AspectJ , 2013, Inf. Softw. Technol..

[95]  Lionel C. Briand,et al.  A Unified Framework for Coupling Measurement in Object-Oriented Systems , 1999, IEEE Trans. Software Eng..

[96]  Harpal S. Dhama Quantitative models of cohesion and coupling in software , 1995, J. Syst. Softw..

[97]  Stuart H. Zweben,et al.  Experimental evaluation of software design principles: An investigation into the effect of module coupling on system modifiability , 1984, J. Syst. Softw..

[98]  James M. Bieman,et al.  Cohesion and reuse in an object-oriented system , 1995, SSR '95.

[99]  Lionel C. Briand,et al.  Empirical Studies of Quality Models in Object-Oriented Systems , 2002, Adv. Comput..

[100]  Miroslaw Staron,et al.  Measuring the impact of changes to the complexity and coupling properties of automotive software systems , 2013, J. Syst. Softw..

[101]  R. Geoff Dromey,et al.  A Model for Software Product Quality , 1995, IEEE Trans. Software Eng..

[102]  Alain Abran,et al.  The Guide to the Software Engineering Body of Knowledge , 1999, IEEE Softw..

[103]  B. Föger The FIELD study , 2006, The Lancet.

[104]  Yinglong Ma,et al.  Semantic oriented ontology cohesion metrics for ontology-based systems , 2010, J. Syst. Softw..

[105]  Martin Hitz,et al.  Chidamber & Kemerer's Metrics Suite: a Measurement Theory Perspective , 1996 .

[106]  Mario Piattini,et al.  Metrics for data warehouse conceptual models understandability , 2007, Inf. Softw. Technol..

[107]  T. J. Emerson A discriminant metric for module cohesion , 1984, ICSE '84.

[108]  EMPIRICAL ASSESSMENT OF THE IMPACT OF MICROFINANCE ON QUALITY OF LIFE , 2014 .

[109]  Lutz Richter,et al.  A proposal for measuring the structural complexity of programs , 1990, J. Syst. Softw..

[110]  Maurice H. Halstead,et al.  Elements of software science (Operating and programming systems series) , 1977 .

[111]  Annabella Loconsole,et al.  Empirical studies on requirement management measures , 2004, Proceedings. 26th International Conference on Software Engineering.

[112]  Martin Shepperd,et al.  Early life-cycle metrics and software quality models. , 1990 .

[113]  Yuming Zhou,et al.  Predicting the maintainability of open source software using design metrics , 2008, Wuhan University Journal of Natural Sciences.

[114]  Walter Cazzola,et al.  A concern-oriented framework for dynamic measurements , 2015, Inf. Softw. Technol..

[115]  Hongyu Zhang,et al.  Measuring design complexity of semantic web ontologies , 2010, J. Syst. Softw..

[116]  Paul Clements,et al.  Software architecture in practice , 1999, SEI series in software engineering.

[117]  Amjed Tahir,et al.  A systematic mapping study on dynamic metrics and software quality , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[118]  Lionel C. Briand,et al.  A measurement framework for object-oriented software testability , 2005, Inf. Softw. Technol..