A case study on testing, commissioning, and operation of very-large-scale software systems

An increasing number of software systems today are very-large-scale software systems (VLSS) with system-of-systems (SoS) architectures. Due to their heterogeneity and complexity VLSS are difficult to understand and analyze, which results in various challenges for development and evolution. Existing software engineering processes, methods, and tools do not sufficiently address the characteristics of VLSS. Also, there are only a few empirical studies on software engineering for VLSS. We report on results of an exploratory case study involving engineers and technical project managers of an industrial automation VLSS for metallurgical plants. The paper provides empirical evidence on how VLSS are tested, commissioned, and operated in practice. The paper discusses practical challenges and reports industrial requirements regarding process and tool support. In particular, software processes and tools need to provide general guidance at the VLSS level as well as specific methods and tools for systems that are part of the VLSS. Processes and tools need to support multi-disciplinary engineering across system boundaries. Furthermore, managing variability and evolution is success-critical in VLSS verification and validation.

[1]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[2]  Stefan Biffl,et al.  Semantic Integration of Software and Systems Engineering Environments , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[3]  Jo Ann Lane,et al.  Systems of systems test and evaluation challenges , 2010, 2010 5th International Conference on System of Systems Engineering.

[4]  Klaus Marius Hansen,et al.  Software ecosystems - A systematic literature review , 2013, J. Syst. Softw..

[5]  CPPParser Noa,et al.  Visual Exploration of Large-Scale System Evolution , 2008 .

[6]  Douglas C. Schmidt,et al.  Ultra-Large-Scale Systems: The Software Challenge of the Future , 2006 .

[7]  Barry W. Boehm,et al.  Some future trends and implications for systems and software engineering processes , 2006, Syst. Eng..

[8]  Paul Grünbacher,et al.  A systematic review and an expert survey on capabilities supporting multi product lines , 2012, Inf. Softw. Technol..

[9]  Paul Grünbacher,et al.  Evolving systems of systems: industrial challenges and research perspectives , 2013, SESoS.

[10]  Tuomo Kähkönen,et al.  Agile Methods for Large Organizations - Building Communities of Practice , 2004, Agile Development Conference.

[11]  Klaus Pohl,et al.  Software Product Line Engineering - Foundations, Principles, and Techniques , 2005 .

[12]  Jean Bézivin,et al.  On the Need for Megamodels , 2004, OOPSLA 2004.

[13]  Aldo Dagnino,et al.  Agile Software Development in Large Organizations , 2004, Computer.

[14]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[15]  Barry Boehm,et al.  The Changing Nature of Software Evolution , 2010 .

[16]  Dewayne E. Perry,et al.  Challenges in Evolving a Large Scale Software Product , 1998 .

[17]  Mark W. Maier.  Architecting Principles for Systems‐of‐Systems , 1996 .

[18]  Philippe Kruchten,et al.  Using grounded theory to study the experience of software development , 2011, Empirical Software Engineering.

[19]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[20]  Meir M. Lehman,et al.  Laws of Software Evolution Revisited , 1996, EWSPT.

[21]  Alessandro Orso,et al.  Scaling regression testing to large software systems , 2004, SIGSOFT '04/FSE-12.

[22]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[23]  Barry Boehm,et al.  21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems , 2006 .

[24]  Mary Beth Chrissis,et al.  CMMI for Development: Guidelines for Process Integration and Product Improvement , 2011 .

[25]  J. Knottnerus,et al.  Real world research. , 2010, Journal of clinical epidemiology.

[26]  Christine Nadel,et al.  Case Study Research Design And Methods , 2016 .

[27]  Klaus Pohl,et al.  Software product line testing , 2006, CACM.

[28]  Stig Larsson,et al.  Integrating business and software development models , 2002, IEEE Software.

[29]  Bernd Brügge,et al.  User involvement in software evolution practice: A case study , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[30]  Stefan Biffl,et al.  Integrating Production Automation Expert Knowledge Across Engineering Domains , 2011, Int. J. Distributed Syst. Technol..

[31]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[32]  Munindar P. Singh,et al.  Service-Oriented Computing: Key Concepts and Principles , 2005, IEEE Internet Comput..

[33]  Barry W. Boehm,et al.  New processes for new horizons: the incremental commitment model , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[34]  Khaled El Emam,et al.  Spice: The Theory and Practice of Software Process Improvement and Capability Determination , 1997 .

[35]  James Bret Michael,et al.  Architectural framework for a system-of-systems , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[36]  C. Brodsky.  The Discovery of Grounded Theory: Strategies for Qualitative Research , 1968 .

[37]  Barry W. Boehm.  Perspectives [The changing nature of software evolution; The inevitability of evolution] , 2010, IEEE Softw..