Open Source Software Evaluation, Selection, and Adoption: a Systematic Literature Review

Background. Open Source Software (OSS) is experiencing an increasing popularity both in industry and in academia. Aim. We investigated models for the selection, evaluation, and adoption of OSS, focusing on factors that affect most the evaluation of OSS. Method. We conducted a Systematic Literature Review of 262 studies published until the end of 2019, to understand whether OSS selection is still an interesting topic for researchers, and which factors are considered by stakeholders and are assessed by the available models. Result. We selected 60 primary studies: 20 surveys and 5 lessons learned studies elicited the motivations for OSS adoption; 35 papers proposed several OSS evaluation models focusing on different technical aspects. This Systematic Literature Review provides an overview of the available OSS evaluation methods, highlighting their limits and strengths, based on the wide range of technicalities and aspects explored by the selected primary studies. Conclusion. OSS producers can benefit from our results by checking if they are providing all the information commonly required by potential adopters. Users can learn how models work and which models cover the relevant characteristics of OSS they are most interested in.

[1]  Sandro Morasca,et al.  An empirical investigation of perceived reliability of open source Java programs , 2012, SAC '12.

[2]  Jong Bae Kim,et al.  A Study on Selection Process of Open Source Software , 2007, Sixth International Conference on Advanced Language Processing and Web Information Technology (ALPIT 2007).

[3]  Jason L. Dedrick,et al.  An exploratory study into open source platform adoption , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[4]  Marcus Ciolkowski,et al.  The QualOSS open source assessment model measuring the performance of open source communities , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[5]  Lakshmanan Ramanathan,et al.  An empirical investigation into the adoption of open source software in Information Technology outsourcing organizations , 2015, J. Syst. Inf. Technol..

[6]  Giancarlo Succi,et al.  A Cost Model of Open Source Software Adoption , 2009, Int. J. Open Source Softw. Process..

[7]  Marin Orlic,et al.  Criteria for evaluation of open source cloud computing solutions , 2011, Proceedings of the ITI 2011, 33rd International Conference on Information Technology Interfaces.

[8]  Qian Xu,et al.  OSSpal: Finding and Evaluating Open Source Software , 2017, OSS.

[9]  Sandro Morasca,et al.  Predicting OSS trustworthiness on the basis of elementary code assessment , 2010, ESEM '10.

[10]  Sandro Morasca,et al.  A Survey on the Importance of Some Economic Factors in the Adoption of Open Source Software , 2010, SERA.

[11]  Brian Fitzgerald,et al.  Commercial adoption of open source software: an empirical study , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[12]  Gang-Soo Lee,et al.  A Security Evaluation and Testing Methodology for Open Source Software Embedded Information Security System , 2005, ICCSA.

[13]  Robert D. Macredie,et al.  A theory-grounded framework of Open Source Software adoption in SMEs , 2011, Eur. J. Inf. Syst..

[14]  M. K. Mijinyawa,et al.  An Extended Framework for Evaluation of Open Source Software Adoption in Small Businesses , 2014 .

[15]  Saïd Assar,et al.  An empirical enquiry into the adoption of Open Source Software by individual users in France , 2009 .

[16]  Thomas Wieland,et al.  Evaluation criteria for free/open source software products based on project analysis , 2006, Softw. Process. Improv. Pract..

[17]  Yoshinobu Tamura,et al.  Reliability Modeling and Assessment for Open Source Cloud Software: A Stochastic Approach , 2014 .

[18]  Nina Helander,et al.  Adoption of Open Source Software and Software-as-a- Service Models in the Telecommunication Industry , 2011, ICSOB.

[19]  Sandro Morasca,et al.  Why Do Developers Adopt Open Source Software? Past, Present and Future , 2019, OSS.

[20]  Thomas Hess,et al.  Comparing the relative importance of evaluation criteria in proprietary and open‐source enterprise application software selection – a conjoint study of ERP and Office systems , 2011, Inf. Syst. J..

[21]  Reidar Conradi,et al.  Adoption of open source software in software-intensive organizations - A systematic literature review , 2010, Inf. Softw. Technol..

[22]  Alberto Siena,et al.  Modelling Risks in Open Source Software Component Selection , 2014, ER.

[23]  Sandro Morasca,et al.  Applying SCRUM in an OSS Development Process: An Empirical Evaluation , 2010, XP.

[24]  Alberto Sillitti,et al.  Introducing the OpenSource Maturity Model , 2009, 2009 ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development.

[25]  Sandro Morasca,et al.  T-DOC: A Tool for the Automatic Generation of Testing Documentation for OSS Products , 2010, OSS.

[26]  Witold Suryn,et al.  Method for the Evaluation of Open Source Software Quality from an IT Untrained User Perspective , 2014, C3S2E.

[27]  Davide Taibi,et al.  Can Opinion Mining Techniques Help to Select Open Source Software , 2016 .

[28]  Sandro Morasca,et al.  OSS-TMM: Guidelines for Improving the Testing Process of Open Source Software , 2011, Int. J. Open Source Softw. Process..

[29]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[30]  Sandro Morasca,et al.  Towards The Evaluation of OSS Trustworthiness: Lessons Learned From The Observation of Relevant OSS Projects , 2008, OSS.

[31]  Ioannis Stamelos,et al.  The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation , 2008, OSS.

[32]  Sandro Morasca,et al.  An Investigation of the Users' Perception of OSS Quality , 2010, OSS.

[33]  Sandro Morasca,et al.  OpenBQR: a framework for the assessment of OSS , 2007, OSS.

[34]  Sung Yul Rhew,et al.  A Quality Model for Open Source Software Selection , 2007, Sixth International Conference on Advanced Language Processing and Web Information Technology (ALPIT 2007).

[35]  Mohamed Sarrab,et al.  Empirical study of open source software selection for adoption, based on software quality characteristics , 2014, Adv. Eng. Softw..

[36]  Sandro Morasca,et al.  The QualiSPo approach to OSS product quality evaluation , 2010, FLOSS '10.

[37]  Diomidis Spinellis Code Documentation , 2010, IEEE Software.

[38]  Sandro Morasca,et al.  A Survey on Open Source Software Trustworthiness , 2011, IEEE Software.

[39]  Sihem Ben Sassi,et al.  Exploring information from OSS repositories and platforms to support OSS selection decisions , 2018, Inf. Softw. Technol..

[40]  Imed Hammouda,et al.  Open Source Systems: Enterprise Software and Solutions , 2018, IFIP Advances in Information and Communication Technology.

[41]  John Campbell,et al.  Enablers of Open Source Software adoption: A case study of APS organisations , 2012 .

[42]  David Lo,et al.  An Empirical Study of Adoption of Software Testing in Open Source Projects , 2013, 2013 13th International Conference on Quality Software.

[43]  Raphaël Semeteys,et al.  Method for Qualification and Selection of Open Source Software , 2008 .

[44]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[45]  Shuichi Takayama,et al.  Studying the Cost and Effectiveness of OSS Quality Assessment Models: An Experience Report of Fujitsu QNET , 2018, IEICE Trans. Inf. Syst..

[46]  Pearl Brereton,et al.  A systematic review of systematic review process research in software engineering , 2013, Inf. Softw. Technol..

[47]  Sandro Morasca,et al.  OP2A: How to Improve the Quality of the Web Portal of Open Source Software Products , 2011, WEBIST.

[48]  Yan Li,et al.  Open source software adoption: motivations of adopters and amotivations of non-adopters , 2011, DATB.

[49]  Syed Faisal Ahmed Bukhari,et al.  Open Source Software (OSS) Adoption Framework for Local Environment and its Comparison , 2009, SCSS.

[50]  Timo Koponen Evaluation Framework for Open Source Software Maintenance , 2006, 2006 International Conference on Software Engineering Advances (ICSEA'06).

[51]  Sandro Morasca,et al.  A Study on OSS Marketing and Communication Strategies , 2012, OSS.

[52]  Anne Immonen,et al.  Trustworthiness Evaluation and Testing of Open Source Components , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[53]  Andrea Back,et al.  The Influence of Risk Factors in Decision-Making Process for Open Source Software Adoption , 2016, Int. J. Inf. Technol. Decis. Mak..