Exact scalable sensitivity analysis for the next release problem

The nature of the requirements analysis problem, based as it is on uncertain and often inaccurate estimates of costs and effort, makes sensitivity analysis important. Sensitivity analysis allows the decision maker to identify those requirements and budgets that are particularly sensitive to misestimation. However, finding scalable sensitivity analysis techniques is not easy because the underlying optimization problem is NP-hard. This article introduces an approach to sensitivity analysis based on exact optimization. We implemented this approach as a tool, OATSAC, which allowed us to experimentally evaluate the scalability and applicability of Requirements Sensitivity Analysis (RSA). Our results show that OATSAC scales sufficiently well for practical applications in Requirements Sensitivity Analysis. We also show how the sensitivity analysis can yield insights into difficult and otherwise obscure interactions between budgets, requirements costs, and estimate inaccuracies using a real-world case study.

[1]  Rolf Ernst,et al.  Applying Sensitivity Analysis in Real-Time Distributed Systems , 2005, IEEE Real-Time and Embedded Technology and Applications Symposium.

[2]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[3]  Jerffeson Teixeira de Souza,et al.  The Human Competitiveness of Search Based Software Engineering , 2010, 2nd International Symposium on Search Based Software Engineering.

[4]  Andrew Smith,et al.  Optimized staffing for product releases and its application at Chartwell Technology , 2008, J. Softw. Maintenance Res. Pract..

[5]  Jerffeson Teixeira de Souza,et al.  Ten Years of Search Based Software Engineering: A Bibliometric Analysis , 2011, SSBSE.

[6]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[7]  Tim Menzies,et al.  Optimizing requirements decisions with keys , 2008, PROMISE '08.

[8]  Klaudia Frankfurter Computers And Intractability A Guide To The Theory Of Np Completeness , 2016 .

[9]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[10]  Ramón Alvarez-Valdés,et al.  A scatter search algorithm for project scheduling under partially renewable resources , 2006, J. Heuristics.

[11]  G. Nemhauser,et al.  Discrete Dynamic Programming and Capital Allocation , 1969 .

[12]  David Pisinger,et al.  Where are the hard knapsack problems? , 2005, Comput. Oper. Res..

[13]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[14]  Yuanyuan Zhang,et al.  “Fairness Analysis” in Requirements Assignments , 2008, 2008 16th IEEE International Requirements Engineering Conference.

[15]  Günther Ruhe,et al.  Lightweight Replanning of Software Product Releases , 2006, 2006 International Workshop on Software Product Management (IWSPM'06 - RE'06 Workshop).

[16]  Liming Zhu,et al.  Tradeoff and Sensitivity Analysis in Software Architecture Evaluation Using Analytic Hierarchy Process , 2005, Software Quality Journal.

[17]  Günther Ruhe,et al.  Optimized Resource Allocation for Software Release Planning , 2009, IEEE Transactions on Software Engineering.

[18]  Giuliano Antoniol,et al.  Search-based techniques applied to optimization of project planning for a massive maintenance project , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[19]  Betty H. C. Cheng,et al.  Research Directions in Requirements Engineering , 2007, Future of Software Engineering (FOSE '07).

[20]  G. Carmichael,et al.  Direct and adjoint sensitivity analysis of chemical kinetic systems with KPP: Part I—theory and software tools , 2003, Atmospheric Environment.

[21]  Jon C. Helton,et al.  Survey of sampling-based methods for uncertainty and sensitivity analysis , 2006, Reliab. Eng. Syst. Saf..

[22]  Shin Yoo,et al.  Search based data sensitivity analysis applied to requirement engineering , 2009, GECCO.

[23]  Dietmar Pfahl,et al.  Trade-off Analysis for Requirements Selection , 2003, Int. J. Softw. Eng. Knowl. Eng..

[24]  Des Greer,et al.  Software release planning: an evolutionary and iterative approach , 2004, Inf. Softw. Technol..

[25]  D. Hamby A review of techniques for parameter sensitivity analysis of environmental models , 1994, Environmental monitoring and assessment.

[26]  R. Levine,et al.  A Sensitivity Analysis of Cross-Country Growth Regressions , 1991 .

[27]  Victor J. Rayward-Smith,et al.  The next release problem , 2001, Inf. Softw. Technol..

[28]  Des Greer,et al.  Quantitative studies in software release planning under risk and resource constraints , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[29]  Martin J. Shepperd,et al.  Software project economics: a roadmap , 2007, Future of Software Engineering (FOSE '07).

[30]  Enrique Alba,et al.  Software project management with GAs , 2007, Inf. Sci..

[31]  Tim Menzies,et al.  Converging on the optimal attainment of requirements , 2002, Proceedings IEEE Joint International Conference on Requirements Engineering.

[32]  René Beier,et al.  Random knapsack in expected polynomial time , 2003, STOC '03.

[33]  H Christopher Frey,et al.  OF SENSITIVITY ANALYSIS , 2001 .

[34]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[35]  Giovanni Denaro,et al.  ACM Transactions on Software Engineering and Methodology : Volume 22, Nomor 4, 2013 , 2014 .

[36]  Tim Menzies,et al.  Experiences using Visualization Techniques to Present Requirements, Risks to Them, and Options for Risk Mitigation , 2006, 2006 First International Workshop on Requirements Engineering Visualization (REV'06 - RE'06 Workshop).

[37]  David S. Rosenblum,et al.  Sensitivity analysis for a scenario-based reliability prediction model , 2005, WADS@ICSE.

[38]  Aurora Trinidad Ramirez Pozo,et al.  Search Based Software Engineering: Review and analysis of the field in Brazil , 2013, J. Syst. Softw..

[39]  Gene Hou,et al.  Overview of Sensitivity Analysis and Shape Optimization for Complex Aerodynamic Configurations , 1999 .

[40]  Robert E. Bixby,et al.  MIP: Theory and Practice - Closing the Gap , 1999, System Modelling and Optimization.

[41]  S. Wagner,et al.  Global Sensitivity Analysis of Predictor Models in Software Engineering , 2007, Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007).

[42]  Stefan Wagner An Approach to Global Sensitivity Analysis: FAST on COCOMO , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[43]  Yuanyuan Zhang,et al.  Search Based Requirements Optimisation: Existing Work and Challenges , 2008, REFSQ.

[44]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[45]  David S. Johnson,et al.  `` Strong '' NP-Completeness Results: Motivation, Examples, and Implications , 1978, JACM.

[46]  Yuanyuan Zhang,et al.  The multi-objective next release problem , 2007, GECCO '07.

[47]  Giuliano Antoniol,et al.  A robust search-based approach to project management in the presence of abandonment, rework, error and uncertainty , 2004, 10th International Symposium on Software Metrics, 2004. Proceedings..

[48]  Sjaak Brinkkemper,et al.  Flexible Release Composition using Integer Linear Programming , 2004 .

[49]  Martin S. Feather,et al.  4.6.1 Combining Heuristic Search, Visualization and Data Mining for Exploration of System Design Spaces , 2004 .

[50]  Günther Ruhe,et al.  Hybrid Intelligence in Software Release Planning , 2004, Int. J. Hybrid Intell. Syst..

[51]  Mark Harman,et al.  Search Based Approaches to Component Selection and Prioritization for the Next Release Problem , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[52]  Claes Wohlin,et al.  An evaluation of methods for prioritizing software requirements , 1998, Inf. Softw. Technol..

[53]  Günther Ruhe,et al.  Bi-objective release planning for evolving software systems , 2007, ESEC-FSE '07.

[54]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[55]  Sjaak Brinkkemper,et al.  Integrated Requirement Selection and Scheduling for the Release Planning of a Software Product , 2007, REFSQ.

[56]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[57]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[58]  John A. Clark,et al.  Dynamic adaptive Search Based Software Engineering , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.