Evaluating the characteristics of a non-standardised Model Requirements Analysis (MRA) for the development of policy impact assessment tools

The aim of this paper is to provide a critical analysis of the strengths and weaknesses of a non-standardised Model Requirements Analysis (MRA) used for the purpose of developing the Sustainability Impact Assessment Tool (SIAT). By 'non-standardised' we mean not strictly following a published MRA method. The underlying question we are interested in addressing is how non-standardised methods, often employed in research driven projects, compare to defined methods with more standardised structure, with regards their ability to capture model requirements effectively, and with regards their overall usability. Through describing and critically assessing the specific features of the non-standardised MRA employed, the ambition of this paper is to provide insights useful for impact assessment tool (IAT) development. Specifically, the paper will (i) characterise kinds of user requirements relevant to the functionality and design of IATs; (ii) highlight the strengths and weaknesses of non-standardised MRA for user requirements capture, analysis and reflection in the context of IAT; (iii) critically reflect on the process and outcomes of having used a non-standardised MRA in comparison with other more standardised approaches. To accomplish these aims, we first review methods available for IAT development before describing the SIAT development process, including the MRA employed. Major strengths and weaknesses of the MRA method are then discussed in terms of user identification and characterisation, organisational characterisation and embedding, and ability to capture design options for ensuring usability and usefulness. A detailed assessment on the structural differences of MRA with two advanced approaches (Integrated DSS design and goal directed design) and their role in performance of the MRA tool is used to critique the approach employed. The results show that MRA is able to bring thematic integration, establish system performance and technical thresholds as well as detailing quality and transparency guidelines. Nevertheless the discussion points out to a number of deficiencies in application - (i) a need to more effectively characterise potential users, and; (ii) a need to better foster communication among the distinguished roles in the development process. If addressed these deficiencies, SIAT non-standardised MRA could have brought out better outcomes in terms of tool usability and usefulness, and improved embedding of the tool into conditions of targeted end-users. Strengths and weaknesses of non-standard Model Requirements Analysis (MRA) analysed.The exercise revealed insights on user requirements of impact assessment tools.Major ones are thematic integration, quality assurance, technical performance and transparency.Offers comparison with goal directed design and Integrated Design and Development approach.Differentiates the approaches based on user characterisation, communication, usability and organisational embedding.

[1]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[2]  Bill Moggridge,et al.  Designing interactions , 2006 .

[3]  Jochen Hinkel,et al.  The PIAM approach to modular integrated assessment modelling , 2009, Environ. Model. Softw..

[4]  M. Pérez-Soba,et al.  Sustainability Impact Assessment of land use policies , 2008 .

[5]  Helene Hembrooke,et al.  Activity-Centered Design: An Ecological Approach to Designing Smart Tools and Usable Systems , 2004 .

[6]  Jan Gulliksen,et al.  Key principles for user-centred systems design , 2003, Behav. Inf. Technol..

[7]  Radhika Santhanam,et al.  An empirical investigation of ODSS impact on individuals and organizations , 2000, Decis. Support Syst..

[8]  Craig A. Aumann,et al.  Constructing model credibility in the context of policy appraisal , 2011, Environ. Model. Softw..

[9]  Anthony J. Jakeman,et al.  A methodology for the design and development of integrated models for policy support , 2011, Environ. Model. Softw..

[10]  Miquel Sànchez-Marrè,et al.  Environmental Decision Support Systems , 2001 .

[11]  Charles Richter,et al.  A review of the state of the practice in requirements modeling , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[12]  Rudolf Vetschera.  Decision Support Systems in Networked Organizations , 1997 .

[13]  Marco A. Janssen,et al.  The roles of computer models in the environmental policy life cycle , 2002 .

[14]  Anthony J. Jakeman,et al.  Environmental decision support systems (EDSS) development - Challenges and best practices , 2011, Environ. Model. Softw..

[15]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[16]  P. Jeffrey,et al.  On the Design of Computer-Based Models for Integrated Environmental Science , 2005, Environmental management.

[17]  Katharina Helming,et al.  Ex ante impact assessment of land use changes in European regions — the SENSOR approach , 2008 .

[18]  R. Brouwer,et al.  Integrated ecological, economic and social impact assessment of alternative flood control policies in the Netherlands , 2004 .

[19]  Katharina Helming,et al.  Transfer into decision support: The Sustainability Impact Assessment Tool (SIAT) , 2008 .

[20]  Donald G. Marquis,et al.  Successful industrial innovations : a study of factors underlying innovation in selected firms , 1969 .

[21]  Anthony Jakeman.  Environmental modelling, software and decision support : state of the art and new perspectives , 2008 .

[22]  Katharina Helming,et al.  Land use functions — a multifunctionality approach to assess the impact of land use changes on land use sustainability , 2008 .

[23]  Gitte Lindgaard,et al.  User Needs Analysis and requirements engineering: Theory and practice , 2006, Interact. Comput..

[24]  Marco Torchiano,et al.  Talking tests: an empirical assessment of the role of fit acceptance tests in clarifying requirements , 2007, IWPSE '07.

[25]  B. Nykvist,et al.  The use and non-use of policy appraisal tools in public policy making: an analysis of three European countries and the European Union , 2008 .

[26]  D. E. Reeve,et al.  Gis, Organisations and People: A Socio-Technical Approach , 1999 .

[27]  D. Norse,et al.  Links between science and policy making , 2000 .

[28]  Andrew P. Sage.  Decision support systems engineering , 1991 .

[29]  Alan Cooper,et al.  About Face 3: the essentials of interaction design , 1995 .

[30]  Hedwig van Delden,et al.  Integration of multi-scale dynamic spatial models of socio-economic and physical processes for river basin management , 2007, Environ. Model. Softw..

[31]  Adrie J. M. Beulens,et al.  A methodology to support multidisciplinary model-based water management , 2007, Environ. Model. Softw..

[32]  Anthony J. Jakeman,et al.  Progress in integrated assessment and modelling , 2002, Environ. Model. Softw..

[33]  Ralph R. Young.  Effective Requirements Practices , 2001 .

[34]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[35]  Brian S. McIntosh,et al.  A review of the factors which influence the use and usefulness of information systems , 2009, Environ. Model. Softw..

[36]  Brian S. McIntosh,et al.  Organisational drivers for, constraints on and impacts of decision and information support tool use in desertification policy and management , 2011, Environ. Model. Softw..

[37]  João Araújo,et al.  An evolutionary model of requirements correctness with early aspects , 2007, IWPSE '07.

[38]  M. Hemmati.  Multi-stakeholder Processes for Governance and Sustainability: Beyond Deadlock and Conflict , 2002 .

[39]  Young ae Hahn,et al.  Activity Centered Design: An Ecological Approach to Designing Smart Tools and Usable Systems , 2005 .

[40]  Philippe Kruchten,et al.  The Rational Unified Process: An Introduction , 1998 .

[41]  Dongwen Wang,et al.  Using multi-perspective methodologies to study users' interactions with the prototype front end of a guideline-based decision support system for diabetic foot care , 2009, Int. J. Medical Informatics.