A framework for qualitative assessment of domain-specific languages

Domain-specific languages (DSLs) are used for improving many facets of software development, but whether and to what extent this aim is achieved is an important issue that must be addressed. This paper presents a proposal for a Framework for Qualitative Assessment of DSLs (FQAD). FQAD is used for determining the perspective of the evaluator, understanding the goal of the assessment and selecting fundamental DSL quality characteristics to guide the evaluator in the process. This framework adapts and integrates the ISO/IEC 25010:2011 standard, CMMI maturity level evaluation approach and the scaling approach used in DESMET into a perspective-based assessment. A detailed list of domain-specific language quality characteristics is elaborated, and a novel assessment method is proposed. Two case studies through which FQAD is matured and evaluated are reported. The case studies have shown that stakeholders find the FQAD process beneficial.

[1]  Pierre Boulet,et al.  Evaluation of Modeling Tools Adaptation , 2012 .

[2]  James W. Howatt,et al.  A project-based approach to programming language evaluation , 1995, SIGP.

[3]  Peter Philp,et al.  Furniture of the World , 1974 .

[4]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[5]  David A. Watt,et al.  Programming language design concepts , 2004 .

[6]  Mark van den Brand,et al.  Using Metrics for Assessing the Quality of ASF+SDF Model Transformations , 2009, ICMT@TOOLS.

[7]  Ulrich Frank,et al.  Domain-Specific Modeling Languages: Requirements Analysis and Design Guidelines , 2013, Domain Engineering, Product Lines, Languages, and Conceptual Models.

[8]  Pedro Hugo do Nascimento Gabriel,et al.  Software languages engineering: experimental evaluation , 2010 .

[9]  Izak Benbasat,et al.  The Case Research Strategy in Studies of Information Systems , 1987, MIS Q..

[10]  Aruna Raja,et al.  Domain Specific Languages , 2010 .

[11]  Allen S. Lee.  A Scientific Methodology for MIS Case Studies , 1989, MIS Q..

[12]  Paul Hudak,et al.  Building domain-specific embedded languages , 1996, CSUR.

[13]  Daniel L. Moody,et al.  The “Physics” of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering , 2009, IEEE Transactions on Software Engineering.

[14]  Alain Abran,et al.  DSML success factors and their assessment criteria , 2008 .

[15]  Juha-Pekka Tolvanen,et al.  Domain-Specific Modeling: Enabling Full Code Generation , 2008 .

[16]  Arie van Deursen,et al.  Domain-Specific Languages in Practice: A User Study on the Success Factors , 2009, MoDELS.

[17]  Parastoo Mohagheghi,et al.  Definitions and approaches to model quality in model-based software development - A review of literature , 2009, Inf. Softw. Technol..

[18]  M. D. Myers,et al.  An introduction to qualitative research in information systems , 2002 .

[19]  Gul Tokdemir.  AN ASSESSMENT MODEL FOR WEB-BASED INFORMATION SYSTEM EFFECTIVENESS: SEWISS , 2010 .

[20]  D. Naiditch.  Selecting a programming language for your project , 1998 .

[21]  Gabor Karsai,et al.  Design Guidelines for Domain Specific Languages , 2014, ArXiv.

[22]  Parastoo Mohagheghi,et al.  A Multi-dimensional Framework for Characterizing Domain Specific Languages , 2007 .

[23]  Shari Lawrence Pfleeger,et al.  Principles of survey research part 6: data analysis , 2003, SOEN.

[24]  Kim S. Cameron,et al.  Critical questions in assessing organizational effectiveness , 1980 .

[25]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[26]  J. Krakower,et al.  Assessing Organizational Effectiveness: Considerations and Procedures. , 1985 .

[27]  Steven Kelly,et al.  Worst Practices for Domain-Specific Modeling , 2009, IEEE Software.

[28]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[29]  Ruven E. Brooks,et al.  Studying programmer behavior experimentally: the problems of proper methodology , 1980, CACM.

[30]  Richard F. Paige,et al.  Principles for modeling language design , 2000, Inf. Softw. Technol..

[31]  Ron Weber,et al.  On the ontological expressiveness of information systems analysis and design grammars , 1993, Inf. Syst. J..

[32]  R. Paige,et al.  Requirements for Domain-Specific Languages , 2006 .

[33]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[34]  J. F. Overbeek.  Meta Object Facility (MOF): investigation of the state of the art , 2006 .

[35]  Barbara Kitchenham,et al.  DESMET: a methodology for evaluating software engineering methods and tools , 1997 .

[36]  Juha Kärnä,et al.  Evaluating the Use of Domain-Specific Modeling in Practice , 2009 .

[37]  Kevin R. Parker,et al.  A Formal Language Selection Process for Introductory Programming Courses , 2006, J. Inf. Technol. Educ..

[38]  R. Henry,et al.  Principles of survey research. , 2018, Family practice research journal.

[39]  Uday P. Khedker.  What Makes a Good Programming Language ? , 2008 .

[40]  John B. Black,et al.  Software psychology: The need for an interdisciplinary program , 1986, Proceedings of the IEEE.

[41]  Bonnie Kaplan,et al.  Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study , 1988, MIS Q..

[42]  John Krogstie,et al.  Evaluating UML using a generic quality framework , 2003 .

[43]  Clifford A. Shaffer,et al.  Building modeling tools that support verification, validation, and testing for the domain expert , 2005, Proceedings of the Winter Simulation Conference, 2005..

[44]  Mark Strembeck,et al.  An approach for the systematic development of domain-specific languages , 2009 .

[45]  Ali Mili,et al.  An empirical study of programming language trends , 2005, IEEE Software.

[46]  Kevin R. Parker,et al.  Criteria for the selection of a programming language for introductory courses , 2006, Int. J. Knowl. Learn..

[47]  B. Flyvbjerg.  Five Misunderstandings About Case-Study Research , 2006, 1304.1186.

[48]  Janne Merilinna,et al.  Comparison between different abstraction level programming: Experiment definition and initial results , 2007, OOPSLA 2007.

[49]  Nuno Oliveira,et al.  Comparing general-purpose and domain-specific languages: An empirical study , 2010, Comput. Sci. Inf. Syst..

[50]  Shari Lawrence Pfleeger,et al.  Principles of survey research: part 1: turning lemons into lemonade , 2001, SOEN.

[51]  Sevg Özkan,et al.  PB-ISAM: A PROCESS-BASED FRAMEWORK FOR INFORMATION SYSTEMS EFFECTIVENESS ASSESSMENT IN ORGANISATIONAL CONTEXTS , 2006 .

[52]  Ron Weber,et al.  On the deep structure of information systems , 1995, Inf. Syst. J..

[53]  M Mernik,et al.  When and how to develop domain-specific languages , 2005, CSUR.

[54]  Mark Strembeck,et al.  An approach for the systematic development of domain‐specific languages , 2009, Softw. Pract. Exp..

[55]  Aniruddha S. Gokhale,et al.  Domain-Specific Modeling , 2008, Handbook of Dynamic System Modeling.

[56]  Peter J. Clarke,et al.  Measuring the effort for creating and using domain-specific models , 2010, DSM '10.

[57]  Jonathan Sprinkle,et al.  Heterogeneous multi-core systems: UML profiles vs. DSM approaches , 2012, DSM '12.