Predicting software complexity by means of evolutionary testing

One characteristic that impedes software from achieving good levels of maintainability is the increasing complexity of software. Empirical observations have shown that typically, the more complex the software is, the bigger the test suite is. Thence, a relevant question, which originated the main research topic of our work, has raised: "Is there a way to correlate the complexity of the test cases utilized to test a software product with the complexity of the software under test?". This work presents a new approach to infer software complexity with basis on the characteristics of automatically generated test cases. From these characteristics, we expect to create a test case profile for a software product, which will then be correlated to the complexity, as well as to other characteristics, of the software under test. This research is expected to provide developers and software architects with means to support and validate their decisions, as well as to observe the evolution of a software product during its life-cycle. Our work focuses on object-oriented software, and the corresponding test suites will be automatically generated through an emergent approach for creating test data named as Evolutionary Testing.

[1]  Mohammad Zulkernine,et al.  Can complexity, coupling, and cohesion metrics be used as early indicators of vulnerabilities? , 2010, SAC '10.

[2]  Oliver Bühler,et al.  Evolutionary functional testing , 2008, Comput. Oper. Res..

[3]  Matthias Grochtmann,et al.  Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing , 1998, Real-Time Systems.

[4]  Lionel C. Briand,et al.  A Unified Framework for Coupling Measurement in Object-Oriented Systems , 1999, IEEE Trans. Software Eng..

[5]  David B. Fogel,et al.  Evolutionary Computation: The Fossil Record , 1998 .

[6]  Motoei Azuma,et al.  Software management practice and metrics in the European community and Japan: Some results of a survey , 1994, J. Syst. Softw..

[7]  Sallie M. Henry,et al.  Software Structure Metrics Based on Information Flow , 1981, IEEE Transactions on Software Engineering.

[8]  B. H. Yin,et al.  The establishment and use of measures to evaluate the quality of software designs , 1978, SIGMETRICS Perform. Evaluation Rev..

[9]  Richard H. Carver,et al.  An Evaluation of the MOOD Set of Object-Oriented Software Metrics , 1998, IEEE Trans. Software Eng..

[10]  David Lorge Parnas,et al.  Software aging , 1994, Proceedings of 16th International Conference on Software Engineering.

[11]  Francisco Fernández de Vega,et al.  eCrash: a framework for performing evolutionary testing on third-party Java components , 2007 .

[12]  Joachim Wegener,et al.  Evaluating Evolutionary Testability with Software-Measurements , 2004, GECCO.

[13]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[14]  T. Vernazza,et al.  Defining metrics for software components , 2002 .

[15]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[16]  Dejun Mu,et al.  QoS Guarantee in Tomcat Web Server: A Feedback Control Approach , 2011, 2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery.

[17]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[18]  George E. Stark,et al.  Using metrics in management decision making , 1994, Computer.

[19]  L. G. Samaraweera,et al.  Using test case metrics to predict code quality and effort , 1996, SOEN.

[20]  Chen Fu,et al.  Exception-Chain Analysis: Revealing Exception Handling Architecture in Java Server Applications , 2007, 29th International Conference on Software Engineering (ICSE'07).

[21]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[22]  Alice T. Lee,et al.  Software analysis handbook: Software complexity analysis and software reliability estimation and prediction , 1994 .

[23]  Tracy Hall,et al.  Measuring OO systems: a critical analysis of the MOOD metrics , 1999, Proceedings Technology of Object-Oriented Languages and Systems. TOOLS 29 (Cat. No.PR00275).

[24]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[25]  Una-May O'Reilly,et al.  Genetic Programming II: Automatic Discovery of Reusable Programs. , 1994, Artificial Life.

[26]  Y. L. Traon,et al.  From genetic to bacteriological algorithms for mutation‐based testing , 2005, Softw. Test. Verification Reliab..

[27]  Fahim Arif,et al.  Analysis of object oriented complexity and testability using object oriented design metrics , 2010, NSEC '10.

[28]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[29]  Gordon B. Davis,et al.  Software Development Practices, Software Complexity, and Software Maintenance Performance: a Field Study , 1998 .

[30]  Arie van Deursen,et al.  An empirical study into class testability , 2006, J. Syst. Softw..

[31]  David P. Darcy,et al.  Managerial Use of Metrics for Object-Oriented Software: An Exploratory Analysis , 1998, IEEE Trans. Software Eng..

[32]  Maarten van Someren,et al.  Evolutionary testing of object-oriented software , 2010, SAC '10.

[33]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[34]  Chris F. Kemerer,et al.  Towards a metrics suite for object oriented design , 2017, OOPSLA '91.

[35]  David Probert,et al.  Release Management in Free Software Projects: Practices and Problems , 2007, OSS.

[36]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[37]  Frank Tsui,et al.  Analysis of software cohesion attribute and test case development complexity , 2011, ACM-SE '11.

[38]  Lionel C. Briand,et al.  A Unified Framework for Cohesion Measurement in Object-Oriented Systems , 2004, Empirical Software Engineering.

[39]  Christopher C. Skiscim,et al.  Optimization by simulated annealing: A preliminary computational study for the TSP , 1983, WSC '83.

[40]  Atam P. Dhawan,et al.  Genetic Algorithms as Global Random Search Methods: An Alternative Perspective , 1995, Evolutionary Computation.

[41]  Jan Mendling,et al.  A Discourse on Complexity of Process Models , 2006, Business Process Management Workshops.

[42]  Arie van Deursen,et al.  Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining , 2008, Empirical Software Engineering.

[43]  Rajesh Vasa,et al.  Evolution of cyclomatic complexity in object oriented software , 2003 .

[44]  Muthu Ramachandran,et al.  Software Metrics Evaluation Based on Entropy , 2010, ArXiv.

[45]  Stefan Wappler,et al.  Using evolutionary algorithms for the unit testing of object-oriented software , 2005, GECCO '05.

[46]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[47]  Ian Sommerville,et al.  System Configuration Management, ICSE'96 SCM-6 Workshop, Berlin, Germany, March 25-26, 1996, Proceedings , 1996, System Configuration Management.

[48]  Joachim Wegener,et al.  A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints , 2004, Real-Time Systems.

[49]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[50]  Herbert E. Dunsmore,et al.  A Software Metrics Survey , 1987 .

[51]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[52]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[53]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[54]  Tanja E. J. Vos,et al.  Search-based testing, the underlying engine of Future Internet testing , 2011, 2011 Federated Conference on Computer Science and Information Systems (FedCSIS).

[55]  Volker Claus,et al.  Evolutionary Algorithms and their Application , 1996 .

[56]  Mourad Badri,et al.  Exploring Empirically the Relationship between Lack of Cohesion in Object-oriented Systems and Coupling and Size , 2010, ICSOFT.

[57]  James E. Tomayko,et al.  The structural complexity of software an experimental test , 2005, IEEE Transactions on Software Engineering.

[58]  Norman Wilde,et al.  Maintenance Support for Object-Oriented Programs , 1992, IEEE Trans. Software Eng..

[59]  Robert V. Binder,et al.  Design for testability in object-oriented systems , 1994, CACM.

[60]  Joachim Wegener,et al.  Evaluation of Different Fitness Functions for the Evolutionary Testing of an Autonomous Parking System , 2004, GECCO.

[61]  Will Venters,et al.  Software engineering: theory and practice , 2006 .

[62]  Lionel C. Briand,et al.  Stress testing real-time systems with genetic algorithms , 2005, GECCO '05.

[63]  Lionel C. Briand,et al.  Empirical Studies of Quality Models in Object-Oriented Systems , 2002, Adv. Comput..

[64]  Mark Harman,et al.  Improving Evolutionary Testing By Flag Removal , 2002, GECCO.

[65]  Fernando Brito e Abreu,et al.  Using OCL to Formalize Object-Oriented Design Metrics Definitions , 2002 .

[66]  Carol Withrow,et al.  Error density and size in Ada software , 1990, IEEE Software.

[67]  Lionel C. Briand,et al.  Using genetic algorithms and coupling measures to devise optimal integration test orders , 2002, SEKE '02.

[68]  Günther Ruhe,et al.  Search Based Software Engineering , 2013, Lecture Notes in Computer Science.

[69]  Witold Pedrycz,et al.  An Empirical Exploration of the Distributions of the Chidamber and Kemerer Object-Oriented Metrics Suite , 2004, Empirical Software Engineering.

[70]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[71]  Mark Lorenz,et al.  Object-oriented software metrics - a practical guide , 1994 .

[72]  Jordi Torres,et al.  Complete instrumentation requirements for performance analysis of Web based technologies , 2003, 2003 IEEE International Symposium on Performance Analysis of Systems and Software. ISPASS 2003..

[73]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[74]  Paul D. Scott,et al.  New Coupling and Cohesion Metrics for Evaluation of Software Component Reusability , 2008, 2008 The 9th International Conference for Young Computer Scientists.

[75]  Chris F. Kemerer,et al.  Reliability of function points measurement: a field experiment , 2015, CACM.

[76]  Ina Schieferdecker,et al.  Improving evolutionary class testing in the presence of non-public methods , 2007, ASE.

[77]  Fernando Brito e Abreu,et al.  Object-Oriented Software Engineering: Measuring and Controlling the Development Process , 1994 .

[78]  Tom Mens,et al.  Challenges in software evolution , 2005, Eighth International Workshop on Principles of Software Evolution (IWPSE'05).

[79]  Fabrizio Riguzzi.  A Survey of Software Metrics , 2002 .

[80]  C. R. K. Reddy,et al.  Metrics for object oriented design software systems : a survey , 2010 .

[81]  John A. Clark,et al.  Automated test‐data generation for exception conditions , 2000, Softw. Pract. Exp..

[82]  T.C. Lethbridge,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK) and the Software Engineering Education Knowledge (SEEK) - a preliminary mapping , 2001, 10th International Workshop on Software Technology and Engineering Practice.

[83]  Bertrand Meyer,et al.  ARTOO , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[84]  Joachim Wegener,et al.  Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing , 2002, GECCO.

[85]  Jazi Eko Istiyanto,et al.  Modularity Index Metrics for Java-Based Open Source Software Projects , 2013, ArXiv.

[86]  Sandeep Purao,et al.  Product metrics for object-oriented systems , 2003, CSUR.

[87]  Peter Nordin,et al.  Genetic programming - An Introduction: On the Automatic Evolution of Computer Programs and Its Applications , 1998 .

[88]  Brian Henderson-Sellers,et al.  Object-Oriented Metrics , 1995, TOOLS.

[89]  Elaine J. Weyuker,et al.  Evaluating Software Complexity Measures , 2010, IEEE Trans. Software Eng..

[90]  Norman E. Fenton,et al.  Measurement : A Necessary Scientific Basis , 2004 .

[91]  Sallie M. Henry,et al.  Object-oriented metrics that predict maintainability , 1993, J. Syst. Softw..

[92]  R. C. Jain,et al.  Metrics for Test Case Design in Test Driven Development , 2010 .

[93]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..

[94]  Frederick T. Sheldon,et al.  Metrics for maintainability of class inheritance hierarchies , 2002, J. Softw. Maintenance Res. Pract..

[95]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[96]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[97]  Ron Weber,et al.  Toward a Theory of the Deep Structure of Information Systems , 1990, ICIS.

[98]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[99]  Ron Weber,et al.  Research on Structured Programming: An Empiricist's Evaluation , 1984, IEEE Transactions on Software Engineering.

[100]  Thomas Bäck,et al.  A Survey of Evolution Strategies , 1991, ICGA.

[101]  Nachiappan Nagappan,et al.  A software testing and reliability early warning (strew) metric suite , 2005 .

[102]  Peter P. Puschner,et al.  Testing the results of static worst-case execution-time analysis , 1998, Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279).

[103]  Meir M. Lehman,et al.  On understanding laws, evolution, and conservation in the large-program life cycle , 1984, J. Syst. Softw..

[104]  Joachim Wegener,et al.  Evaluating evolutionary testability for structure-oriented testing with software measurements , 2008, Appl. Soft Comput..

[105]  Jainendra K. Navlakha.  A Survey of System Complexity Metrics , 1987, Comput. J..

[106]  Martin J. Shepperd,et al.  Search Heuristics, Case-based Reasoning And Software Project Effort Prediction , 2002, GECCO.

[107]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[108]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[109]  Victor R. Basili,et al.  Software errors and complexity: an empirical investigation , 1993 .

[110]  Francisco Fernández de Vega,et al.  Test Case Evaluation and Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software , 2009, Inf. Softw. Technol..

[111]  Huan Li,et al.  Dynamical evolution analysis of the object-oriented software systems , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).