Experiments in Automated Identification of Ambiguous Natural-Language Requirements

Recent research indicates that a significant proportion of the failures observed in operational space mission systems can be traced back to ambiguous requirements. Frequently, the functionality and behavior specified by the defective requirement is insufficiently detailed (e.g., information about the expected operational context is missing) or the requirement is phrased so as to permit multiple interpretations. Recent analysis of several thousand anomalies observed in operational space missions shows that about 20% are due to either operator error or faulty procedures. Again, many of these anomalies can be traced back to requirements exhibiting the same types of ambiguities. Since a typical space mission is specified by several thousand requirements, a significant amount of effort may need to be expended to discover ambiguous requirements. Many development organizations still conduct this type of discovery manually, after the requirements have been specified – in addition to requiring significant effort, manual analysis is also error-prone. Previous work has shown the utility of machine learners and simple natural language processing techniques (e.g., parts of speech tagging) for supporting the identification of a relatively small number of temporal requirements within a large body of system requirements. We extend this work to the problem of identifying the ambiguous requirements within the complete set of requirements for the flight and ground components of a recently-launched space mission. Initial results indicate that these techniques can be effective in automatically identifying ambiguous requirements. Unlike heuristic techniques, new learning models for identifying ambiguous requirements can be developed as mission requirements and vocabularies evolve.

[1]  Beatrice Santorini.  Part-of-speech tagging guidelines for the penn treebank project , 1990 .

[2]  Ram Chillarege,et al.  Defect type and its impact on the growth curve (software development) , 1991, [1991 Proceedings] 13th International Conference on Software Engineering.

[3]  William Wilson,et al.  Automated quality analysis of Natural Language requirement specifications , 1996 .

[4]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[5]  Thorsten Brants.  TnT - Statistical Part-of-Speech Tagging , 2000 .

[6]  Thorsten Brants,et al.  TnT – A Statistical Part-of-Speech Tagger , 2000, ANLP.

[7]  George S. Avrunin,et al.  PROPEL: an approach supporting property elucidation , 2002, ICSE '02.

[8]  R. Lutz,et al.  Orthogonal defect classification for projects , 2003 .

[9]  Gerard J. Holzmann,et al.  The SPIN Model Checker - primer and reference manual , 2003 .

[10]  George S. Avrunin,et al.  From Natural Language Requirements to Rigorous Property Specifications , 2003 .

[11]  Jane Huffman Hayes,et al.  Improving requirements tracing via information retrieval , 2003, Proceedings. 11th IEEE International Requirements Engineering Conference, 2003..

[12]  David R. Karger,et al.  Tackling the Poor Assumptions of Naive Bayes Text Classifiers , 2003, ICML.

[13]  Erik Kamsties,et al.  From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity , 2003 .

[14]  Shin Nakajima,et al.  The SPIN Model Checker : Primer and Reference Manual , 2004 .

[15]  Jane Huffman Hayes,et al.  Helping analysts trace requirements: an objective look , 2004, Proceedings. 12th IEEE International Requirements Engineering Conference, 2004..

[16]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[17]  George S. Avrunin,et al.  User guidance for creating precise and accessible property specifications , 2006, SIGSOFT '06/FSE-14.

[18]  Forrest Shull,et al.  Inspecting the History of Inspections: An Example of Evidence-Based Technology Diffusion , 2008, IEEE Software.

[19]  Allen P. Nikora,et al.  Improving the Accuracy of Space Mission Software Anomaly Frequency Estimates , 2009, 2009 Third IEEE International Conference on Space Mission Challenges for Information Technology.

[20]  Allen P. Nikora,et al.  Automated Identification of LTL Patterns in Natural Language Requirements , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[21]  Kishor S. Trivedi,et al.  An empirical investigation of fault types in space mission system software , 2010, 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN).