Usability evaluation of an interactive decision support system for user-guided design of scenarios of watershed conservation practices

This paper evaluated user interaction with the graphical user interface of WRESTORE, an environmental decision support system (EDSS) for watershed planning that utilizes user ratings of design alternatives to guide optimization. The following usability metrics were collected for stakeholders as well as surrogates (who are often used for DSS development due to time or cost constraints): task times across sequential sessions, percentage of time spent and of mouse clicking in different areas of interest, and trends in self-reported user confidence levels. Task times conformed to theoretical models of learning curves. Stakeholders, however, spent 15% more time and made 14% more mouse clicks in information gathering areas than surrogates. Confidence levels increased over time in 67% of stakeholders, but only in 29% of surrogates. Relationships between time spent or mouse clicking events and confidence level trends indicated that confidence ratings increased over time for users who conducted more information gathering. This study highlights the importance of increasing user interactions within information gathering areas of an interface to improve user-guided search, and suggests that developers should exercise caution when using surrogates as proxies for stakeholders. It also demonstrates a quantitative way to evaluate EDSS that could assist others developing similar tools.

[1]  Lynn E. Johnson.  Water Resource Management Decision Support Systems , 1986 .

[2]  Mordechai Haklay,et al.  Usability evaluation and PPGIS: towards a user-centred design approach , 2003, Int. J. Geogr. Inf. Sci..

[3]  T. Nyerges,et al.  Collaborative Water Resource Decision Support: Results of a Field Experiment , 2006 .

[4]  H. G. Wind,et al.  Application of fuzzy sets and cognitive maps to incorporate social science scenarios in integrated assessment modelsA case study of urbanization in Ujung Pandang, Indonesia , 2000 .

[5]  Geraldine Abrami,et al.  Variable time scales, agent-based models, and role-playing games: The PIEPLUE river basin management game , 2007 .

[6]  Daniel C. Cliburn,et al.  Evaluating the Usability of a Tool for Visualizing the Uncertainty of the Future Global Water Balance , 2003 .

[7]  Anthony J. Jakeman,et al.  Environmental decision support systems (EDSS) development - Challenges and best practices , 2011, Environ. Model. Softw..

[8]  S. Dessai,et al.  Actionable Knowledge for Environmental Decision Making: Broadening the Usability of Climate Science , 2013 .

[9]  Wendy Merritt,et al.  The Role of Decision Support Systems (DSS) in Planning for Improved Water Quality in Coastal Lakes , 2010 .

[10]  Meghna Babbar-Sebens,et al.  Decentralized and partially decentralized reinforcement learning for designing a distributed wetland system in watersheds , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[11]  Mohamad Y. Jaber,et al.  Learning Curves : Theory, Models, and Applications , 2011 .

[12]  Meghna Babbar-Sebens,et al.  Optimizing conservation practices in watersheds: Do community preferences matter? , 2013 .

[13]  Piotr Jankowski,et al.  Design Considerations and Evaluation of a Collaborative, Spatio-Temporal Decision Support System , 2006, Trans. GIS.

[14]  John Bromley,et al.  Research, part of a Special Feature on Implementing Participatory Water Management: Recent Advances in Theory, Practice and Evaluation Evaluation of Bayesian Networks in Participatory Water Resources Management, Upper Guadiana Basin, Spain , 2010 .

[15]  Meghna Babbar-Sebens,et al.  Spatial identification and optimization of upland wetlands in agricultural watersheds , 2013 .

[16]  Meghna Babbar-Sebens,et al.  Interactive Genetic Algorithm with Mixed Initiative Interaction for multi-criteria ground water monitoring design , 2012, Appl. Soft Comput..

[17]  Hugh P. Possingham,et al.  Optimal restoration of altered habitats , 2010, Environ. Model. Softw..

[18]  John W. Nicklow,et al.  Generating alternative watershed-scale BMP designs with evolutionary algorithms , 2008 .

[19]  Boyan Brodaric,et al.  GroundWater Markup Language (GWML) - enabling groundwater data interoperability in spatial data infrastructures , 2012 .

[20]  G. Ice.  History of Innovative Best Management Practice Development and its Role in Addressing Water Quality Limited Waterbodies , 2004 .

[21]  Louis E. Yelle.  THE LEARNING CURVE: HISTORICAL REVIEW AND COMPREHENSIVE SURVEY , 1979 .

[22]  Jeffrey M. Sadler,et al.  A recipe for standards-based data sharing using open source software and low-cost electronics , 2016 .

[23]  Guillermo A. Mendoza,et al.  Multi-criteria decision analysis in natural resource management: A critical review of methods and new modelling paradigms , 2006 .

[24]  Manuel J. Rodriguez,et al.  ATES: a geo-informatics decision aid tool for the integration of groundwater into land planning , 2015 .

[25]  Jeffrey G. Arnold,et al.  Soil and Water Assessment Tool Theoretical Documentation Version 2009 , 2011 .

[26]  Sara S. Metcalf,et al.  Sharing the floodplain: Mediated modeling for environmental management , 2010, Environ. Model. Softw..

[27]  Raghavan Srinivasan,et al.  SWAT: Model Use, Calibration, and Validation , 2012 .

[28]  Robin Gregory,et al.  Using Stakeholder Values to Make Smarter Environmental Decisions , 2000 .

[29]  François Bousquet,et al.  Modelling with stakeholders , 2010, Environ. Model. Softw..

[30]  D. Helsel,et al.  Statistical methods in water resources , 2020, Techniques and Methods.

[31]  Anthony J. Jakeman.  Environmental modelling,software and decision support , 2012 .

[32]  P. Winne,et al.  Feedback and Self-Regulated Learning: A Theoretical Synthesis , 1995 .

[33]  Meghna Babbar-Sebens,et al.  A web-based software tool for participatory optimization of conservation practices in watersheds , 2015, Environ. Model. Softw..

[34]  William K. Estes,et al.  Classification and cognition , 1994 .

[35]  M. Arabi,et al.  Representation of agricultural conservation practices with SWAT , 2008 .

[36]  Ramesh Sharda,et al.  Model-driven decision support systems: Concepts and research directions , 2007, Decis. Support Syst..

[37]  Jeffrey G. Arnold,et al.  Estimation of soil cracking and the effect on surface runoff in a Texas Blackland Prairie watershed , 2005 .

[38]  M. Arabi,et al.  Cost‐effective allocation of watershed management practices using a genetic algorithm , 2006 .

[39]  John W. Nicklow,et al.  Optimizing Structural Best Management Practices Using SWAT and Genetic Algorithm to Improve Water Quality Goals , 2012, Water Resources Management.

[40]  Andrea Castelletti,et al.  Bayesian Networks and participatory modelling in water resource management , 2007, Environ. Model. Softw..