Assessing the Performance of Automated Model Extraction Rules

Automated Model Extraction Rules take as input requirements (in natural language) to generate domain models. Despite the existing work on these rules, there is a lack of evaluations in industrial settings. To address this gap, we conduct an evaluation in an industrial context, reporting the extraction rules that are triggered to create a model from requirements and their frequency. We also assess the performance in terms of recall, precision and F-measure of the generated model compared to the models created by domain experts of our industrial partner. Results enable us to identify new research directions to push forward automated model extraction rules: the inclusion of new knowledge sources as input for the extraction rules, and the development of specific experiments to evaluate the understanding of the generated models.

[1]  Rodina Ahmad,et al.  Class Diagram Extraction from Textual Requirements Using Natural Language Processing (NLP) Techniques , 2010, 2010 Second International Conference on Computer Research and Development.

[2]  José Ambrosio Toval Álvarez,et al.  A systematic review of UML model consistency management , 2009, Inf. Softw. Technol..

[3]  Atul Gupta,et al.  Identifying domain elements from textual specifications , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[4]  Vincenzo Gervasi,et al.  On the Systematic Analysis of Natural Language Requirements with CIRCE , 2006, Automated Software Engineering.

[5]  Andrea De Lucia,et al.  On the role of the nouns in IR-based traceability recovery , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[6]  Juan Llorens Morillo,et al.  Requirements Verification in the Industry , 2011, CSDM.

[7]  Sjaak Brinkkemper,et al.  Automated Extraction of Conceptual Models from User Stories via NLP , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[8]  Lionel C. Briand,et al.  aToucan: An Automated Framework to Derive UML Analysis Models from Use Case Models , 2015, TSEM.

[9]  Paul Vickers,et al.  Parsed use case descriptions as a basis for object-oriented class model generation , 2011, J. Syst. Softw..

[10]  Lionel C. Briand,et al.  A systematic review of transformation approaches between user requirements and analysis models , 2011, Requirements Engineering.

[11]  Pamela Zave.  Classification of research efforts in requirements engineering , 1997, ACM Comput. Surv..

[12]  Mehrdad Sabetzadeh,et al.  Extracting domain models from natural-language requirements: approach and industrial evaluation , 2016, MoDELS.

[13]  Stefania Gnesi,et al.  Using NLP to Detect Requirements Defects: An Industrial Experience in the Railway Domain , 2017, REFSQ.

[14]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[15]  Ratna Sanyal,et al.  Semi-automatic generation of UML models from natural language requirements , 2011, ISEC.

[16]  Henda Hajjami Ben Ghézala,et al.  Automatic builder of class diagram (ABCD): an application of UML generation from functional requirements , 2016, Softw. Pract. Exp..

[17]  S. Abirami,et al.  Conceptual modeling of natural language functional requirements , 2014, J. Syst. Softw..

[18]  Mehrdad Sabetzadeh,et al.  Automated Checking of Conformance to Requirements Templates Using Natural Language Processing , 2015, IEEE Transactions on Software Engineering.

[19]  Nenad Medvidovic,et al.  Reducing Ambiguities in Requirements Specifications Via Automatically Created Object-Oriented Models , 2008, Monterey Workshop.

[20]  Felice Dell'Orletta,et al.  Measuring and Improving the Completeness of Natural Language Requirements , 2014, REFSQ.