Scenario-based assessment of nonfunctional requirements

This paper describes a method and a tool for validating nonfunctional requirements in complex socio-technical systems. The system requirements analyzer (SRA) tool validates system reliability and operational performance requirements using scenario-based testing. Scenarios are transformed into sequences of task steps and the reliability of human agents performing tasks with computerized technology is assessed using Bayesian belief network (BN) models. The tool tests system performance within an envelope of environmental variations and reports the number of tests that pass a benchmark threshold. The tool diagnoses problematic areas in scenarios representing pathways through system models, assists in the identification of their causes, and supports comparison of alternative requirements specifications and system designs. It is suitable for testing socio-technical systems where operational scenarios are sequential and deterministic, in domains where designs are incrementally modified so set up costs of the BNs can be defrayed over multiple tests.

[1]  Mojgan Kowsari,et al.  A Validation System for Object Oriented Specifications of Information Systems , 1997, ADBIS.

[2]  Alistair Sutcliffe,et al.  Assessing the reliability of socio‐technical systems , 2003 .

[3]  Jens Rasmussen,et al.  Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  Erik Hollnagel,et al.  Cognitive reliability and error analysis method , 1998 .

[5]  Alistair G. Sutcliffe,et al.  Assessing the usability and efficiency of Design Rationale , 1997, INTERACT.

[6]  Shailey Minocha,et al.  An impact analysis method for safety-critical user interface design , 1999, TCHI.

[7]  Alistair G. Sutcliffe,et al.  Scenario advisor tool for requirements engineering , 2005, Requirements Engineering.

[8]  Norman E. Fenton,et al.  Software metrics: successes, failures and new directions , 1999, J. Syst. Softw..

[9]  K. D. Duncan,et al.  TASK ANALYSIS AND TRAINING DESIGN. , 1967 .

[10]  J. Shaoul.  Human Error , 1973, Nature.

[11]  J Rasmussen,et al.  Human error and the problem of causality in analysis of accidents. , 1990, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[12]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[13]  Shailey Minocha,et al.  Supporting Scenario-Based Requirements Engineering , 1998, IEEE Trans. Software Eng..

[14]  Alistair Sutcliffe,et al.  2.5.1 Assessing the Reliability of Socio‐technical Systems , 2002 .

[15]  Andrew Gemino,et al.  Empirical comparisons of animation and narration in requirements validation , 2004, Requirements Engineering.

[16]  Shailey Minocha,et al.  Human errors and system requirements , 1999, Proceedings IEEE International Symposium on Requirements Engineering (Cat. No.PR00188).

[17]  Gary Herrin,et al.  A guide to practical human reliability assessment , 1996 .

[18]  Pericles Loucopoulos,et al.  Visualisation of conceptual specifications , 1994, Inf. Syst..

[19]  Constance L. Heitmeyer,et al.  A strategy for efficiently verifying requirements , 2003, ESEC/FSE-11.

[20]  Constance L. Heitmeyer,et al.  A Strategy for Efficiently Verifying Requirements Specifications Using Composition and Invariants , 2003 .

[21]  John Mylopoulos,et al.  From object-oriented to goal-oriented requirements analysis , 1999, CACM.

[22]  Klaus Pohl,et al.  Requirements Elicitation and Validation with Real World Scenes , 1998, IEEE Trans. Software Eng..

[23]  Brian A. Nixon,et al.  Management of Performance Requirements for Information Systems , 2000, IEEE Trans. Software Eng..

[24]  Jeffrey D. Smith,et al.  An evaluation of inquiry-based requirements analysis for an Internet service , 1995, Proceedings of 1995 IEEE International Symposium on Requirements Engineering (RE'95).

[25]  Neil A. M. Maiden,et al.  Requirements engineering tools go mobile , 2004, Proceedings. 26th International Conference on Software Engineering.

[26]  Alistair G. Sutcliffe,et al.  Human-centred requirements engineering , 2004, Proceedings. 12th IEEE International Requirements Engineering Conference, 2004..

[27]  Axel van Lamsweerde,et al.  Goal-oriented requirements enginering: a roundtrip from research to practice [enginering read engineering] , 2004, Proceedings. 12th IEEE International Requirements Engineering Conference, 2004..

[28]  Annie I. Antón,et al.  A Representational Framework for Scenarios of System Use , 1998, Requirements Engineering.

[29]  A. van Lamsweerde.  Goal-oriented requirements enginering: a roundtrip from research to practice [enginering read engineering] , 2004 .

[30]  Martin Neil,et al.  Building large-scale Bayesian networks , 2000, The Knowledge Engineering Review.

[31]  Alistair G. Sutcliffe,et al.  Validating functional system requirements with scenarios , 2002, Proceedings IEEE Joint International Conference on Requirements Engineering.

[32]  Hong Zhu,et al.  Scenario Analysis in an Automated Tool for Requirements Engineering , 2000, Requirements Engineering.

[33]  Norman E. Fenton,et al.  A Critique of Software Defect Prediction Models , 1999, IEEE Trans. Software Eng..

[34]  Kenji Takahashi,et al.  Inquiry-based requirements analysis , 1994, IEEE Software.

[35]  V. Lalioti.  Animation for validation of business system specifications , 1997, Proceedings of the Thirtieth Hawaii International Conference on System Sciences.

[36]  Constance L. Heitmeyer,et al.  Applying the SCR requirements method to a weapons control panel: an experience report , 1998, FMSP '98.

[37]  E. N. Corlett,et al.  Handbook of human reliability analysis with emphasis on nuclear power plant applications: A.D. Swain and H.E. Guttmann Draft report for interim use and comment. Office of Nuclear Regulatory Research, US Regulatory Commission, Washington DC 20555, pp 440 , 1981 .

[38]  David Wright,et al.  Assesing dependability of safety critical systems using diverse evidence , 1998, IEE Proc. Softw..

[39]  John S. Anderson,et al.  Using scenarios in deficiency-driven requirements engineering , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[40]  Christopher Exton,et al.  A Java 3D implementation of a geon based visualisation tool for UML , 2003, PPPJ.

[41]  Robert W. Bailey,et al.  Human performance engineering: A guide for system designers , 1982 .

[42]  A. D. Swain,et al.  Handbook of human-reliability analysis with emphasis on nuclear power plant applications. Final report , 1983 .

[43]  David Chenho Kung,et al.  Status report: requirements engineering , 1993, IEEE Software.

[44]  Marco Pistore,et al.  Model checking early requirements specifications in Tropos , 2001, Proceedings Fifth IEEE International Symposium on Requirements Engineering.

[45]  Alistair G. Sutcliffe,et al.  The domain theory - patterns for knowledge and software reuse , 2002 .

[46]  Peter A. Wieringa,et al.  PREHEP: human error probability based process unit selection , 2001, IEEE Trans. Syst. Man Cybern. Syst..

[47]  Axel van Lamsweerde,et al.  Handling Obstacles in Goal-Oriented Requirements Engineering , 2000, IEEE Trans. Software Eng..

[48]  Julio Cesar Sampaio do Prado Leite,et al.  Nonfunctional requirements: from elicitation to conceptual models , 2004, IEEE Transactions on Software Engineering.

[49]  Mary Beth Rosson,et al.  Requirements Development in Scenario-Based Design , 1998, IEEE Trans. Software Eng..

[50]  Colette Rolland,et al.  Guiding Goal Modeling Using Scenarios , 1998, IEEE Trans. Software Eng..

[51]  Reiko Heckel,et al.  Model-Based Verification and Validation of Properties , 2003, UNIGRA.

[52]  Bev Littlewood,et al.  Software Reliability and Metrics , 2007 .

[53]  Jerzy W. Rozenblit,et al.  Test scenario generation from a structured requirements specification , 1999, Proceedings ECBS'99. IEEE Conference and Workshop on Engineering of Computer-Based Systems.

[54]  Norman E. Fenton,et al.  Software Measurement: Uncertainty and Causal Modeling , 2002, IEEE Softw..

[55]  Debra J. Richardson,et al.  Constructing Bayesian-network models of software testing and maintenance uncertainties , 1997, 1997 Proceedings International Conference on Software Maintenance.

[56]  Axel van Lamsweerde,et al.  Goal-Oriented Requirements Engineering: A Guided Tour , 2001, RE.

[57]  Erik Hollnagel,et al.  Human Reliability Analysis: Context and Control , 1994 .

[58]  Johannes RyserÊÊÊÊÊÊÊÊÊÊÊÊÊÊMartin Glinz,et al.  A Scenario-Based Approach to Validating and Testing Software Systems Using Statecharts , 1999 .

[59]  Nancy G. Leveson.  Intent Specifications: An Approach to Building Human-Centered Specifications , 2000, IEEE Trans. Software Eng..

[60]  T.M. Duffy,et al.  Scenario-Based Design: Envisioning Work and Technology in System Development [Book Review] , 1996, IEEE Transactions on Professional Communication.

[61]  Haralampos Karanikas,et al.  Evaluation of the SRA Tool Using Data Mining Techniques , 2003, CAiSE Short Paper Proceedings.

[62]  A. Davis,et al.  Giving voice to requirements engineering , 1994, IEEE Software.

[63]  Anders P. Ravn,et al.  From Safety Analysis to Software Requirements , 1998, IEEE Trans. Software Eng..

[64]  John Mylopoulos,et al.  Representing and Using Nonfunctional Requirements: A Process-Oriented Approach , 1992, IEEE Trans. Software Eng..

[65]  Colin Potts,et al.  ScenIC: a strategy for inquiry-driven requirements determination , 1999, Proceedings IEEE International Symposium on Requirements Engineering (Cat. No.PR00188).

[66]  Norman E. Fenton,et al.  Software Metrics: A Rigorous Approach , 1991 .

[67]  Alistair Sutcliffe,et al.  A Taxonomy of Error Types for Failure Analysis and Risk Assessment , 1998, Int. J. Hum. Comput. Interact..

[68]  Erik Hollnagel,et al.  The Phenotype of Erroneous Actions , 1993, Int. J. Man Mach. Stud..

[69]  James T. Reason,et al.  Managing the risks of organizational accidents , 1997 .

[70]  Peter Neumann,et al.  Safeware: System Safety and Computers , 1995, SOEN.