Evaluation of an Integrated Tool Environment for Experimentation in DSL Engineering

Domain specific languages (DSL) are a popular means for providing customized solutions to a certain problem domain. So far, however, language workbenches lack sufficient built-in features in providing decision support when it comes to language design and improvement. Controlled experiments can provide data-driven decision support for both, researchers and language engineers, for comparing different languages or language features. This paper provides an evaluation of an integrated end-to-end tool environment for performing controlled experiments in DSL engineering. The experimentation environment is presented by a running example from engineering domain specific languages for acceptance testing. The tool is built on and integrated into the Meta Programming System (MPS) language workbench. For each step of an experiment the language engineer is supported by suitable DSLs and tools all within the MPS platform. The evaluation, from the viewpoint of the experiments subject, is based on the technology acceptance model (TAM). Results reveal that the subjects found the DSL experimentation environment intuitive and easy to use.

[1]  Rogério Eduardo Garcia,et al.  An Ontology for Controlled Experiments on Software Engineering , 2008, SEKE.

[2]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[3]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[4]  Harvey Siy,et al.  An Ontology to Support Empirical Studies in Software Engineering , 2009, 2009 International Conference on Computing, Engineering and Information.

[5]  Walter F. Tichy,et al.  Hints for Reviewing Empirical Work in Software Engineering , 2000, Empirical Software Engineering.

[6]  Fred D. Davis.  User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts , 1993, Int. J. Man Mach. Stud..

[7]  Tore Dybå,et al.  Conducting realistic experiments in software engineering , 2002, Proceedings International Symposium on Empirical Software Engineering.

[8]  Michael Felderer,et al.  Manual test case derivation from UML activity diagrams and state machines: A controlled experiment , 2015, Inf. Softw. Technol..

[9]  Uirá Kulesza,et al.  A Model-Driven Approach to Specifying and Monitoring Controlled Experiments in Software Engineering , 2013, PROFES.

[10]  Austen Rainer,et al.  Case Study Research in Software Engineering - Guidelines and Examples , 2012 .

[11]  Rini van Solingen,et al.  Goal Question Metric (GQM) Approach , 2002 .

[12]  Jordi Cabot,et al.  Corpus-based analysis of domain-specific languages , 2013, Software & Systems Modeling.

[13]  Jeffrey C. Carver,et al.  Assessing the Frequency of Empirical Evaluation in Software Modeling Research , 2011, EESSMod.

[14]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[15]  Miguel Goulão,et al.  Quality in use of domain-specific languages: a case study , 2011, PLATEAU '11.

[16]  André L. M. Santos,et al.  ESEML: empirical software engineering modeling language , 2012, DSM '12.

[17]  Juan de Lara,et al.  Engaging End-Users in the Collaborative Development of Domain-Specific Modelling Languages , 2013, CDVE.

[18]  Miguel Goulão,et al.  Do Software Languages Engineers Evaluate their Languages? , 2011, CIbSE.

[19]  Uirá Kulesza,et al.  Automated Support for Controlled Experiments in Software Engineering: A Systematic Review (S) , 2013, SEKE.

[20]  Miguel Goulão,et al.  Evaluating the Usability of Domain-Specific Languages , 2013 .

[21]  Uirá Kulesza,et al.  Assessing and Evolving a Domain Specific Language for Formalizing Software Engineering Experiments: An Empirical Study , 2014, Int. J. Softw. Eng. Knowl. Eng..

[22]  Marjan Mernik,et al.  Domain-Specific Languages: A Systematic Mapping Study , 2016, Inf. Softw. Technol..

[23]  Ruth Breu,et al.  Is business domain language support beneficial for creating test case specifications: A controlled experiment , 2016, Inf. Softw. Technol..

[24]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[25]  Guilherme Horta Travassos,et al.  Model-based testing approaches selection for software projects , 2009, Inf. Softw. Technol..

[26]  O. John,et al.  Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German , 2007 .

[27]  Forrest Shull,et al.  An Environment for Conducting Families of Software Engineering Experiments , 2008, Adv. Comput..

[28]  Ruth Breu,et al.  An integrated tool environment for experimentation in domain specific language engineering , 2016, EASE.

[29]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .