On the effect of using SysML requirement diagrams to comprehend requirements: results from two controlled experiments

We carried out a controlled experiment and an external replication to investigate whether the use of requirement diagrams of the System Modeling Language (SysML) helps in the comprehensibility of requirements. The original experiment was conducted at the University of Basilicata in Italy with Bachelor students, while its replication was executed at the University of Gothenburg in Sweden with Bachelor and Master students. A total of 87 participants took part in the experiment and its replication. The achieved results indicated that the comprehension of requirements is statistically significant when requirements specification documents include requirement diagrams without any impact on the time to accomplish comprehension tasks. On the basis of our results, we also present and discuss possible implications from the practitioner and researcher perspectives.

[1]  Forrest Shull,et al.  The empirical investigation of Perspective-Based Reading , 1995, Empirical Software Engineering.

[2]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[3]  Ralph R. Young.  Effective Requirements Practices , 2001 .

[4]  Winifred Menezes,et al.  Marketing Technology to Software Practitioners , 2000, IEEE Softw..

[5]  Jeffrey C. Carver,et al.  Issues in using students in empirical studies in software engineering education , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[6]  Silvia Mara Abrahão,et al.  Assessing the Effectiveness of Sequence Diagrams in the Comprehension of Functional Requirements: Results from a Family of Five Experiments , 2013, IEEE Transactions on Software Engineering.

[7]  Bernd Bruegge,et al.  Object-Oriented Software Engineering: Using UML, Patterns and Java, Second Edition , 2003 .

[8]  Pearl Brereton,et al.  Empirical evidence about the UML: a systematic literature review , 2011, Softw. Pract. Exp..

[9]  C.J.H. Mann,et al.  A Practical Guide to SysML: The Systems Modeling Language , 2009 .

[10]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[11]  Ralf H. Reussner,et al.  A controlled experiment to evaluate how styles affect the understandability of requirements specifications , 2003, Inf. Softw. Technol..

[12]  H. Levene.  Robust tests for equality of variances , 1961 .

[13]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[14]  Natalia Juristo Juzgado,et al.  Analysis of the influence of communication between researchers on experiment replication , 2006, ISESE '06.

[15]  Dietmar Pfahl,et al.  Reporting Experiments in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[16]  Diane K. Michelson,et al.  Applied Statistics for Engineers and Scientists , 2001, Technometrics.

[17]  Jeffrey Parsons,et al.  How UML is used , 2006, CACM.

[18]  Marco Torchiano,et al.  How Developers' Experience and Ability Influence Web Application Comprehension Tasks Supported by UML Stereotypes: A Series of Four Experiments , 2010, IEEE Transactions on Software Engineering.

[19]  Jeffrey C. Carver,et al.  Knowledge-Sharing Issues in Experimental Software Engineering , 2004, Empirical Software Engineering.

[20]  Jan Verelst,et al.  The Influence of the Level of Abstraction on the Evolvability of Conceptual Models of Information Systems , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[21]  Sanford Friedenthal,et al.  A Practical Guide to SysML: The Systems Modeling Language , 2008 .

[22]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[23]  Mehrdad Sabetzadeh,et al.  A SysML-based approach to traceability management and design slicing in support of safety certification: Framework, tool support, and case studies , 2012, Inf. Softw. Technol..

[24]  Yvonne Rogers,et al.  External cognition: how do graphical representations work? , 1996, Int. J. Hum. Comput. Stud..

[25]  Liming Zhu,et al.  Evaluating guidelines for reporting empirical software engineering studies , 2008, Empirical Software Engineering.

[26]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[27]  Victor R. Basili,et al.  The TAME Project: Towards Improvement-Oriented Software Environments , 1988, IEEE Trans. Software Eng..

[28]  Mehrdad Sabetzadeh,et al.  Traceability and SysML design slices to support safety inspections: A controlled experiment , 2014, TSEM.

[29]  Giuseppe Scanniello,et al.  On the impact of UML analysis models on source-code comprehensibility and modifiability , 2014, ACM Trans. Softw. Eng. Methodol..

[30]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[31]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[32]  Paul D. Ellis,et al.  The essential guide to effect sizes : statistical power, meta-analysis, and the interpretation of research results , 2010 .

[33]  Siw Elisabeth Hove,et al.  The impact of UML documentation on software maintenance: an experimental evaluation , 2006, IEEE Transactions on Software Engineering.

[34]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[35]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[36]  Neil A. Ernst,et al.  A Framework for Empirical Evaluation of Model Comprehensibility , 2007, International Workshop on Modeling in Software Engineering (MISE'07: ICSE Workshop 2007).

[37]  Giuseppe Scanniello,et al.  Investigating the Role of UML in the Software Modeling and Maintenance - A Preliminary Industrial Survey , 2010, ICEIS.

[38]  Tore Dybå,et al.  A systematic review of effect size in software engineering experiments , 2007, Inf. Softw. Technol..

[39]  S. Shapiro,et al.  An analysis of variance test for normality ( complete samp 1 es ) t , 2007 .

[40]  Mehrdad Sabetzadeh,et al.  SafeSlice: a model slicing and design safety inspection tool for SysML , 2011, ESEC/FSE '11.