An Empirical Investigation to Understand the Difficulties and Challenges of Software Modellers When Using Modelling Tools

Software modelling is a challenging and error-prone task. Existing Model-Driven Engineering (MDE) tools provide modellers with little aid, partly because tool providers have not investigated users' difficulties through empirical investigations such as field studies. This paper presents the results of a two-phase user study to identify the most prominent difficulties that users might face when developing UML Class and State-Machine diagrams using UML modelling tools. In the first phase, we identified the preliminary modelling challenges by analysing 30 Class and State-Machine models that were previously developed by students as a course assignment. The result of the first phase helped us design the second phase of our user study where we empirically investigated different aspects of using modelling tools: the tools' effectiveness, users' efficiency, users' satisfaction, the gap between users' expectation and experience, and users' cognitive difficulties. Our results suggest that users' greatest difficulties are in (1) remembering contextual information and (2) identifying and fixing errors and inconsistencies.

[1]  Margaret M. Burnett,et al.  A practical guide to controlled experiments of software engineering tools with human participants , 2013, Empirical Software Engineering.

[3]  Timothy Lethbridge,et al.  A Survey of Tool Use in Modeling Education , 2017, 2017 ACM/IEEE 20th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[4]  J. Shaoul.  Human Error , 1973, Nature.

[5]  Jason E. Robbins,et al.  Cognitive support, UML adherence, and XMI interchange in Argo/UML , 2000, Inf. Softw. Technol..

[6]  Sahar Kokaly,et al.  User Experience for Model-Driven Engineering: Challenges and Future Directions , 2017, 2017 ACM/IEEE 20th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[7]  Constance L. Heitmeyer,et al.  Automated consistency checking of requirements specifications , 1996, TSEM.

[8]  Miroslaw Staron,et al.  An empirical study on using stereotypes to improve understanding of UML models , 2004, Proceedings. 12th IEEE International Workshop on Program Comprehension, 2004..

[9]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[10]  Keng Siau,et al.  Can UML Be Simplified? Practitioner Use of UML in Separate Domains , 2007, EMMSAD.

[11]  Maria Kutar,et al.  A Comparison of Empirical Study and Cognitive Dimensions Analysis in the Evaluation of UML Diagrams , 2002, PPIG.

[12]  James H. Hill,et al.  Measuring and Reducing Modeling Effort in Domain-Specific Modeling Languages with Examples , 2011, 2011 18th IEEE International Conference and Workshops on Engineering of Computer-Based Systems.

[13]  Alexander Egyed,et al.  Instant consistency checking for the UML , 2006, ICSE.

[14]  Michel R. V. Chaudron,et al.  In practice: UML software architecture and design description , 2006, IEEE Software.

[15]  James H. Hill,et al.  Proactive modeling: auto-generating models from their semantics and constraints , 2012, DSM '12.

[16]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[17]  Kelly Caine,et al.  Understanding Your Users: A Practical Guide to User Research Methods , 2015 .

[18]  Horst Lichter,et al.  Enabling Model Recommenders for Command-Enabled Editors , 2013, MDEBE@MoDELS.

[19]  D. Cheriton,et al.  An Experimental Investigation on Understanding the Difficulties and Challenges of Software Modellers When Using Modelling Tools , 2018 .

[20]  Håkan Burden,et al.  Industrial Adoption of Model-Driven Engineering: Are the Tools Really the Problem? , 2013, MoDELS.

[21]  Miguel A. Fernández,et al.  MDE Adoption in Industry: Challenges and Success Criteria , 2009, MoDELS.

[22]  Wolfgang Emmerich,et al.  Flexible consistency checking , 2003, TSEM.

[23]  Michel R. V. Chaudron,et al.  An empirical assessment of completeness in UML designs , 2004, ICSE 2004.

[24]  Nuno Jardim Nunes,et al.  The Meta Sketch Editor , 2006, CADUI.

[25]  James H. Hill,et al.  Proactive modeling: a new model intelligence technique , 2017, Software & Systems Modeling.

[26]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[27]  Krzysztof Czarnecki,et al.  Guided development with multiple domain-specific languages , 2007, MODELS'07.

[28]  Mark Rouncefield,et al.  Empirical assessment of MDE in industry , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[29]  Nuno Jardim Nunes,et al.  Hammering Models: Designing Usable Modeling Tools , 2011, INTERACT.

[30]  Michal Antkiewicz,et al.  Effects of using examples on structural model comprehension: a controlled experiment , 2014, ICSE.

[31]  Dimitris I. Rigas,et al.  How effective is it to design by voice , 2007 .

[32]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[33]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[34]  Benoît Combemale,et al.  The Relevance of Model-Driven Engineering Thirty Years from Now , 2014, MoDELS.

[35]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[36]  Mik Kersten.  Focusing knowledge work with task context , 2007 .

[37]  Broderick Crawford,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2007 .

[38]  Pauline A. Smith.  Towards a Practical Measure of Hypertext Usability , 1996, Interact. Comput..

[39]  Anna Zamansky,et al.  Cognitive factors in inconsistency management , 2015, 2015 IEEE 23rd International Requirements Engineering Conference (RE).

[40]  Filippo Ricca,et al.  Who Knows/Uses What of the UML: A Personal Opinion Survey , 2014, MoDELS.

[41]  Stuart Kent,et al.  Model Driven Engineering , 2002, IFM.

[42]  Monique Snoeck,et al.  Consistency by Construction: The Case of MERODE , 2003, ER.