Evaluating the Quality of Open Source Software

Traditionally, research on quality attributes was either kept under wraps within the organization that performed it, or carried out by outsiders using narrow, black-box techniques. The emergence of open source software has changed this picture allowing us to evaluate both software products and the processes that yield them. Thus, the software source code and the associated data stored in the version control system, the bug tracking databases, the mailing lists, and the wikis allow us to evaluate quality in a transparent way. Even better, the large number of (often competing) open source projects makes it possible to contrast the quality of comparable systems serving the same domain. Furthermore, by combining historical source code snapshots with significant events, such as bug discoveries and fixes, we can further dig into the causes and effects of problems. Here we present motivating examples, tools, and techniques that can be used to evaluate the quality of open source (and by extension also proprietary) software.

[1]  Stephen H. Kan,et al.  Metrics and Models in Software Quality Engineering , 1994, SOEN.

[2]  Andrew E. Johnson,et al.  Computer Vision on Mars , 2007, International Journal of Computer Vision.

[3]  Diomidis Spinellis,et al.  Code Quality: The Open Source Perspective , 2006 .

[4]  Diomidis Spinellis.  A tale of four kernels , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[5]  Ioannis Stamelos,et al.  Software Product and Process Assessment through Profile-Based Evaluation , 2003, Int. J. Softw. Eng. Knowl. Eng..

[6]  Christian Payne,et al.  On the security of open source software , 2002, Inf. Syst. J..

[7]  Niels Jørgensen,et al.  Putting it all in the trunk: incremental software development in the FreeBSD open source project , 2001, Inf. Syst. J..

[8]  Motoei Azuma.  SquaRE The next generation of the ISO/IEC 9126 and 14598 international standards series on software product quality , 2001 .

[9]  Michael Mitzenmacher,et al.  A Brief History of Generative Models for Power Law and Lognormal Distributions , 2004, Internet Math..

[10]  Diomidis Spinellis,et al.  Optimizing header file include directives , 2009, J. Softw. Maintenance Res. Pract..

[11]  Diomidis Spinellis.  Global software development in the freeBSD project , 2006, GSD '06.

[12]  D. Spinellis,et al.  How is open source affecting software development? , 2004, IEEE Software.

[13]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[14]  Sallie M. Henry,et al.  Software Structure Metrics Based on Information Flow , 1981, IEEE Transactions on Software Engineering.

[15]  H. D. Rombach,et al.  The Goal Question Metric Approach , 1994 .

[16]  Ioannis Stamelos,et al.  The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation , 2008, OSS.

[17]  MockusAudris,et al.  Report on MSR 2004 , 2005 .

[18]  Alexander K. Petrenko,et al.  Electronic Notes in Theoretical Computer Science , 2009 .

[19]  Joost Visser,et al.  Interpretation of Source Code Clusters in Terms of the ISO/IEC-9126 Maintainability Characteristics , 2008, 2008 12th European Conference on Software Maintenance and Reengineering.

[20]  Diomidis Spinellis,et al.  Refactoring--Does It Improve Software Quality? , 2007, Fifth International Workshop on Software Quality (WoSQ'07: ICSE Workshops 2007).

[21]  Alan Bundy,et al.  Constructing Induction Rules for Deductive Synthesis Proofs , 2006, CLASE.

[22]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[23]  George Candea,et al.  Recovery-oriented computing: building multitier dependability , 2004, Computer.

[24]  Paul W. Oman,et al.  Using metrics to evaluate software system maintainability , 1994, Computer.

[25]  Diomidis Spinellis,et al.  Global Analysis and Transformations in Preprocessed Languages , 2003, IEEE Trans. Software Eng..

[26]  M. Petró‐Turza,et al.  The International Organization for Standardization. , 2003 .

[27]  Diomidis Spinellis.  Code Quality: The Open Source Perspective (Effective Software Development Series) , 2006 .

[28]  Orest Pilskalns,et al.  Regression testing with UML software designs: A survey , 2009 .

[29]  Ioannis Stamelos,et al.  Code quality analysis in open source software development , 2002, Inf. Syst. J..

[30]  Diomidis Spinellis,et al.  Power laws in software , 2008, TSEM.

[31]  Tom Mens,et al.  A survey of software refactoring , 2004, IEEE Transactions on Software Engineering.

[32]  Ioannis Stamelos,et al.  Open source software development should strive for even greater code maintainability , 2004, CACM.

[33]  Brian Wildsmith,et al.  What a Tale , 1987 .

[34]  Shigeo Hirose,et al.  A New Approach to Humanitarian Demining , 2005, Auton. Robots.

[35]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[36]  Eirini Kalliamvakou,et al.  Mediterranean Conference on Information Systems ( MCIS ) 2009 Measuring Developer Contribution From Software Repository Data , 2017 .

[37]  Andrea Capiluppi,et al.  Detecting Agility of Open Source Projects Through Developer Engagement , 2008, OSS.