A systematic literature review of open source software quality assessment models

Background Many open source software (OSS) quality assessment models are proposed and available in the literature. However, there is little or no adoption of these models in practice. In order to guide the formulation of newer models so they can be acceptable by practitioners, there is need for clear discrimination of the existing models based on their specific properties. Based on this, the aim of this study is to perform a systematic literature review to investigate the properties of the existing OSS quality assessment models by classifying them with respect to their quality characteristics, the methodology they use for assessment, and their domain of application so as to guide the formulation and development of newer models. Searches in IEEE Xplore, ACM, Science Direct, Springer and Google Search is performed so as to retrieve all relevant primary studies in this regard. Journal and conference papers between the year 2003 and 2015 were considered since the first known OSS quality model emerged in 2003.ResultsA total of 19 OSS quality assessment model papers were selected. To select these models we have developed assessment criteria to evaluate the quality of the existing studies. Quality assessment models are classified into five categories based on the quality characteristics they possess namely: single-attribute, rounded category, community-only attribute, non-community attribute as well as the non-quality in use models. Our study reflects that software selection based on hierarchical structures is found to be the most popular selection method in the existing OSS quality assessment models. Furthermore, we found that majority (47%) of the existing models do not specify any domain of application.ConclusionsIn conclusion, our study will be a valuable contribution to the community and helps the quality assessment model developers in formulating newer models and also to the practitioners (software evaluators) in selecting suitable OSS in the midst of alternatives.

[1]  Reinhold Plösch,et al.  Operationalised product quality models and assessment: The Quamoco approach , 2014, Inf. Softw. Technol..

[2]  Luis Eduardo Mendoza,et al.  Quality Measurement Model for Analysis and Design Tools Based on FLOSS , 2008, 19th Australian Conference on Software Engineering (aswec 2008).

[3]  Klaas-Jan Stol,et al.  Challenges in using open source software in product development: a review of the literature , 2010, FLOSS '10.

[4]  Sanjay Misra,et al.  A Review of Models for Evaluating Quality in Open Source Software , 2013 .

[5]  Sung Yul Rhew,et al.  A Quality Model for Open Source Software Selection , 2007, Sixth International Conference on Advanced Language Processing and Web Information Technology (ALPIT 2007).

[6]  Min-Gyu Lee,et al.  Quality Evaluation Criteria Based on Open Source Mobile HTML5 UI Framework for Development of Cross-Platform , 2015 .

[7]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[8]  Vladimir Cretu,et al.  Towards a software quality assessment model based on open-source statical code analyzers , 2011, 2011 6th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI).

[9]  Marcus Ciolkowski,et al.  The QualOSS open source assessment model measuring the performance of open source communities , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[10]  Tristan Müller,et al.  How to choose a free and open source integrated library system , 2011, OCLC Syst. Serv..

[11]  David Mauricio,et al.  A Review of Software Quality Models for the Evaluation of Software Products , 2014, ArXiv.

[12]  Nadire Cavus.  The application of a multi-attribute decision-making algorithm to learning management systems evaluation , 2011, Br. J. Educ. Technol..

[13]  Timothy Lethbridge,et al.  A taxonomy of software types to facilitate search and evidence-based software engineering , 2008, CASCON '08.

[14]  Alberto Sillitti,et al.  Introducing the OpenSource Maturity Model , 2009, 2009 ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development.

[15]  Nakornthip Prompoon,et al.  Evaluating software quality in use using user reviews mining , 2013, The 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE).

[16]  Lerina Aversano,et al.  Applying EFFORT for Evaluating CRM Open Source Systems , 2011, PROFES.

[17]  Ali Idri,et al.  Predicting Software Product Quality: A Systematic Mapping Study , 2015, Computación y Sistemas.

[18]  Syahrul Fahmy,et al.  Evaluating the Quality of Software in e-Book Using the ISO 9126 Model , 2012 .

[19]  Mohamed Sarrab,et al.  Empirical study of open source software selection for adoption, based on software quality characteristics , 2014, Adv. Eng. Softw..

[20]  Sandro Morasca,et al.  The QualiSPo approach to OSS product quality evaluation , 2010, FLOSS '10.

[21]  Jean-Christophe Deprez,et al.  Comparing Assessment Methodologies for Free/Open Source Software: OpenBRR and QSOS , 2008, PROFES.

[22]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[23]  Ioannis Stamelos,et al.  Evaluating the Quality of Open Source Software , 2009, SQM@CSMR.

[24]  Ozan Çakir,et al.  A web-based decision support system for multi-criteria inventory classification using fuzzy AHP methodology , 2008, Expert Syst. Appl..

[25]  Arne-Kristian Groven,et al.  Free/Libre Open Source Quality Models-a comparison between two approaches , 2010 .

[26]  Raphaël Semeteys,et al.  Method for Qualification and Selection of Open Source Software , 2008 .

[27]  Oyvind Hauge,et al.  An empirical study on selection of Open Source Software - Preliminary results , 2009, 2009 ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development.

[28]  Kentaro Takeda,et al.  A Study on Maturity Model of Open Source Software Community to Estimate the Quality of Products , 2014, KES.

[29]  Sandro Morasca,et al.  A Survey on Open Source Software Trustworthiness , 2011, IEEE Software.

[30]  Ioannis Stamelos,et al.  The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation , 2008, OSS.

[31]  Sandro Morasca,et al.  An Investigation of the Users' Perception of OSS Quality , 2010, OSS.

[32]  Lerina Aversano,et al.  Quality evaluation of floss projects: Application to ERP systems , 2013, Inf. Softw. Technol..

[33]  Edmundas Kazimieras Zavadskas,et al.  State of art surveys of overviews on MCDM/MADM methods , 2014 .

[34]  Nadire Cavus,et al.  The evaluation of Learning Management Systems using an artificial intelligence fuzzy logic algorithm , 2010, Adv. Eng. Softw..

[35]  Elmar Jürgens,et al.  Software quality models: Purposes, usage scenarios and requirements , 2009, 2009 ICSE Workshop on Software Quality.

[36]  Chandrakumar Thangavel,et al.  Efficiency analysis of ERP projects—software quality perspective , 2015 .

[37]  Yong Hu,et al.  Systematic literature review of machine learning based software development effort estimation models , 2012, Inf. Softw. Technol..

[38]  Shenghuo Zhu,et al.  SumView: A Web-based engine for summarizing product reviews and customer opinions , 2013, Expert Syst. Appl..

[39]  Luiz Fernando Capretz,et al.  An open source usability maturity model (OS-UMM) , 2012, Comput. Hum. Behav..

[40]  Ali Idri,et al.  Evaluating Software Product Quality: A Systematic Mapping Study , 2014, 2014 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement.

[41]  Anna Grimán,et al.  Quality model for the selection of floss-based issue tracking system , 2008, ICSE 2008.

[42]  Richard Mathieu,et al.  The Application of DEA to Measure the Efficiency of Open Source Security Tool Production , 2007, AMCIS.

[43]  Sanjay Misra,et al.  Quantitative Quality Model for Evaluating Open Source Web Applications: Case Study of Repository Software , 2013, 2013 IEEE 16th International Conference on Computational Science and Engineering.