A CSCW Requirements Engineering CASE Tool: Development and usability evaluation

Abstract Context CSRML Tool 2012 is a Requirements Engineering CASE Tool for the Goal-Oriented Collaborative Systems Requirements Modeling Language (CSRML). Objective This work aims at evaluating the usability of CSRML Tool 2012, thus identifying any possible usability flaw to be solved in the next releases of the application, as well as giving a general overview on how to develop a DSL tool similar to the one evaluated in this work by means of Visual Studio Modelling SDK. Method In this evaluation, which was reported by following the ISO/IEC 25062:2006 Common Industry Format for usability tests, 28 fourth-course Computer Science students took part. They were asked to carry out a series of modifications to an incomplete CSRML requirements specification. Usability was assessed by measuring the task’s completion rate, the elapsed time, number of accesses to the help system of the tool and the instructor’s verbal assistance. The participants’ arousal and pleasantness were assessed by analyzing both facial expressions and a USE questionnaire. Results In spite of obtaining high usability levels, the test outcome revealed some usability flaws that should be addressed. Conclusion The important lessons learnt from this evaluation are also applicable to the success of other usability tests as well as to the development of new CASE tools.

[1]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[2]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[3]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[4]  Keyvan Nayyeri.  Professional Visual Studio Extensibility , 2008 .

[5]  Clarence A. Ellis,et al.  Groupware: some issues and experiences , 1991, CACM.

[6]  Charles M. Grinstead,et al.  Introduction to probability , 1999, Statistics for the Behavioural Sciences.

[7]  Pieter Desmet,et al.  Measuring Emotion: Development and Application of an Instrument to Measure Emotional Responses to Products , 2005, Funology.

[8]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[9]  Benoit M. Macq,et al.  Usability evaluation of virtual reality interaction techniques for positioning and manoeuvring in reduced, manipulation-oriented environments , 2010, The Visual Computer.

[10]  Jonathan Lee,et al.  Structuring requirement specifications with goals , 2001, Inf. Softw. Technol..

[11]  John N. Tsitsiklis,et al.  Introduction to Probability , 2002 .

[12]  Eric Yu,et al.  Non-Functional Requirements Elicitation , 2019, International Journal of Recent Technology and Engineering.

[13]  Mauricio Camargo,et al.  A situation model to support awareness in collaborative design , 2013, Int. J. Hum. Comput. Stud..

[14]  Steve Cook,et al.  Domain-Specific Development with Visual Studio DSL Tools , 2007 .

[15]  Yun Yang,et al.  Using an experimental study to develop group awareness support for real-time distributed collaborative writing , 2006, Inf. Softw. Technol..

[16]  Silvia Mara Abrahão,et al.  Usability evaluation methods for the web: A systematic mapping study , 2011, Inf. Softw. Technol..

[17]  Aurora Vizcaíno,et al.  Requirements engineering tools: Capabilities, survey and assessment , 2012, Inf. Softw. Technol..

[18]  Y. Arai,et al.  Cultural differences , 2000 .

[19]  W. Keith Edwards,et al.  Verbal coordination in first person shooter games , 2012, CSCW.

[20]  Nicu Sebe,et al.  UX_Mate: from facial expressions to UX evaluation , 2012, DIS '12.

[21]  John Mylopoulos,et al.  A Requirements-Driven Development Methodology , 2001, CAiSE.

[22]  Mordechai Ben-Menachem,et al.  Writing effective use cases , 2001, SOEN.

[23]  Danilo Caivano,et al.  Usability evaluation: a survey of software development organizations , 2011, SEKE.

[24]  François Sainfort,et al.  Evaluating usability of a commercial electronic health record: A case study , 2008, Int. J. Hum. Comput. Stud..

[25]  Silvia Mara Abrahão,et al.  Evaluating requirements modeling methods based on user perceptions: A family of experiments , 2011, Inf. Sci..

[26]  Mali Senapathi,et al.  Journal of Information Technology Education a Framework for the Evaluation of Case Tool Learnability in Educational Environments , 2022 .

[27]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[28]  David Budgen,et al.  CASE tool evaluation: experiences from an empirical study , 2003, J. Syst. Softw..

[29]  Yijun Yu,et al.  OpenOME: An Open-source Goal and Agent-Oriented Model Drawing and Analysis Tool , 2011, iStar.

[30]  Kraig Finstad.  The system usability scale and non-native English speakers , 2006 .

[31]  Yutaka Matsushita,et al.  A company-office system "Valentine" providing informal communication and personal space based on 3D virtual space and avatars , 1999, Inf. Softw. Technol..

[32]  Andrew Sears,et al.  Heuristic Walkthroughs: Finding the Problems Without the Noise , 1997, Int. J. Hum. Comput. Interact..

[33]  Elena Navarro,et al.  CSRML: A Goal-Oriented Approach to Model Requirements for Collaborative Systems , 2011, ER.

[34]  Tanja E. J. Vos,et al.  An empirical approach for evaluating the usability of model-driven tools , 2013, Sci. Comput. Program..

[35]  Elena Navarro,et al.  An empirical evaluation of requirement engineering techniques for collaborative systems , 2011, EASE.

[36]  Monica Landoni,et al.  Is Google the answer? A study into usability of search engines , 2007 .

[37]  Asimina Vasalou,et al.  Cultural differences, experience with social networks and the nature of "true commitment" in Facebook , 2010, Int. J. Hum. Comput. Stud..

[38]  Gina Trapani,et al.  The Complete Guide to Google Wave , 2010 .

[39]  Elena Navarro,et al.  Comparing Goal-Oriented Approaches to Model Requirements for CSCW , 2011, ENASE.

[40]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[41]  Jeffrey C. Carver,et al.  A systematic literature review to identify and classify software requirement errors , 2009, Inf. Softw. Technol..

[42]  Carl Gutwin,et al.  A Descriptive Framework of Workspace Awareness for Real-Time Groupware , 2002, Computer Supported Cooperative Work (CSCW).

[43]  David Eichelberger,et al.  Handbook Of Psychological Testing , 2016 .

[44]  Dik J. Hermes,et al.  Usability of optically simulated haptic feedback , 2008, Int. J. Hum. Comput. Stud..

[45]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[46]  Xavier Franch,et al.  Making Explicit Some Implicit i* Language Decisions , 2011, ER.

[47]  M. Roseman,et al.  A usability study of awareness widgets in a shared workspace groupware system , 1996, CSCW '96.

[48]  Xavier Franch,et al.  J-PRiM: A Java Tool for a Process Reengineering i* Methodology , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[49]  Elena Navarro,et al.  Analyzing the understandability of Requirements Engineering languages for CSCW systems: A family of experiments , 2012, Inf. Softw. Technol..

[50]  Richard C. Gronback.  Eclipse Modeling Project: A Domain-Specific Language Toolkit , 2009 .

[51]  Alistair G. Sutcliffe,et al.  Investigating user experience in Second Life for collaborative learning , 2012, Int. J. Hum. Comput. Stud..

[52]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[53]  Peter Zielczynski.  Requirements Management Using IBM Rational RequisitePro , 2007 .

[54]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[55]  Mervyn A. Jack,et al.  A usability comparison of three alternative message formats for an SMS banking service , 2008, Int. J. Hum. Comput. Stud..

[56]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[57]  Mario Piattini,et al.  Assessing the understandability of UML statechart diagrams with composite states—A family of empirical studies , 2009, Empirical Software Engineering.

[58]  Kjeld Schmidt,et al.  Computer Supported Cooperative Work: New challenges to systems design , 1999 .

[59]  J. Russell.  A circumplex model of affect. , 1980 .

[60]  William G. Cochran,et al.  Experimental Designs, 2nd Edition , 1950 .

[61]  A. Baron.  Experimental Designs , 1990, The Behavior analyst.