Empirical Validation of Component-based Software Systems Generation and Evaluation Approaches

Component-based software development needs to formalize a process of generation, evaluation and selection of Composite COTS-based Software Systems (CCSS), enabling software architects to make early decisions; the Azimut approach and its associated software tool were proposed to tackle this problem. This article presents an experimental study conduced to compare Azimut approach with a Systematized Ad-Hoc approach, regarding generated solutions quality, cost and effort. Results suggest that: (1) Azimut generate better quality solutions at lower cost, but not statistically significant, and (2) there is strong evidence showing that the effort required is higher than for Systematized Ad-Hoc approach; re-sampling methods (Bootstrap and Jackknife) were applied to reinforce these conclusions. Also this study serves as a framework for validating approaches, process and tools for generating and evaluating component-based software systems.

[1]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[2]  Santiago Comella-Dorda,et al.  A Process for COTS Software Product Evaluation , 2002, ICCBSS.

[3]  Hernán Astudillo,et al.  Multidimensional Catalogs for Systematic Exploration of Component-Based Design Spaces , 2006, IFIP Workshop on Advanced Software Engineering.

[4]  Chris Britton,et al.  IT Architectures and Middleware: Strategies for Building Large, Integrated Systems (2nd Edition) , 2000 .

[5]  Carme Quer,et al.  DesCOTS: a software system for selecting COTS components , 2004, Proceedings. 30th Euromicro Conference, 2004..

[6]  Neil A. M. Maiden,et al.  Acquiring COTS software selection requirements , 1998, Proceedings of IEEE International Symposium on Requirements Engineering: RE '98.

[7]  Andrew S. Tanenbaum,et al.  Distributed systems: Principles and Paradigms , 2001 .

[8]  Walter F. Tichy,et al.  A Controlled Experiment in Maintenance Comparing Design Patterns to Simpler Solutions , 2001, IEEE Trans. Software Eng..

[9]  Liming Zhu,et al.  Evaluating guidelines for empirical software engineering studies , 2006, ISESE '06.

[10]  Clemens Szyperski,et al.  Component software , 1997 .

[11]  Hernán Astudillo,et al.  Explicit Architectural Policies to Satisfy NFRs Using COTS , 2005, MoDELS Satellite Events.

[12]  Donald Firesmith,et al.  Specifying Reusable Security Requirements , 2004, J. Object Technol..

[13]  Reidar Conradi,et al.  A case study on building COTS-based system using aspect-oriented programming , 2005, SAC '05.

[14]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[15]  Marco Torchiano,et al.  Overlooked aspects of COTS-based development , 2004, IEEE Software.

[16]  Reidar Conradi,et al.  An empirical study of variations in COTS-based software development processes in the Norwegian IT industry , 2004, 10th International Symposium on Software Metrics, 2004. Proceedings..

[17]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[18]  Lionel C. Briand.  A Critical Analysis of Empirical Research in Software Testing , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[19]  Marco Torchiano,et al.  An empirical study on decision making in off-the-shelf component-based development , 2006, ICSE '06.

[20]  Carina Frota Alves,et al.  Investigating Conflicts in Cots Decision-Making , 2003, Int. J. Softw. Eng. Knowl. Eng..

[21]  H. Urkowitz,et al.  Tests of distribution-free non-Gaussian noise removal in spectral analysis of meteorological radar echo , 1994, Proceedings of IGARSS '94 - 1994 IEEE International Geoscience and Remote Sensing Symposium.