A formative and self-reflective approach to monitoring and evaluation of interdisciplinary team research: An integrated water resource modelling application in Australia

Abstract Recent literature on model evaluation has highlighted the need, particularly in an interdisciplinary problem and team setting, to go beyond the evaluation of results and outputs of a problem-solving pathway (process), and monitor and evaluate the pathway itself to improve ongoing activities undertaken by the research team that best achieve the desired impact. By reflecting on what decisions have been made, the reasons, influences and effects of each decision, such evaluation can improve the learning process of stakeholders and researchers. The authors have designed a formative, self-reflective monitoring and evaluation (M&E) process and applied it to an integrated assessment and modelling project in the Campaspe catchment, part of the Murray-Darling Basin in Victoria, Australia. The M&E process consists of six steps: 1) review of documents to understand research goals, priorities and records, 2) pathway assessment, 3) reflective meetings with each team member, 4) analysis of qualitative data from meetings to capture perceived strengths, weaknesses, successes and failures, and suggested actions, objections and goals, 5) survey to elicit feedback on identified goals, objectives and actions, and 6) a report of approved actions based on the conducted survey of team members. Four overall goals and constituent objectives were identified with 42 actions, and between 37 and 42 of these actions appear to have support for adoption. The evaluation indicated the benefits of the enhanced M&E approach to realise a more adaptive and effective problem-solving modelling and assessment project.

[1]  Dana H. Taplin and Hélene Clark,et al.  Theory of Change Basics. A primer on Theory of Change , 2012 .

[2]  Merritt Polk,et al.  Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact , 2018 .

[3]  Tuomas J. Lahtinen,et al.  Path dependence in Operational Research—How the modeling process can influence the results , 2016 .

[4]  D. Schoen The Reflective Practitioner , 1983 .

[5]  Carl D. Shapiro,et al.  From data to decisions: Processing information, biases, and beliefs for improved management of natural resources and environments , 2017 .

[6]  François Bousquet,et al.  Modelling with stakeholders , 2010, Environ. Model. Softw..

[7]  Tuomas J. Lahtinen,et al.  Why pay attention to paths in the practice of environmental modelling? , 2017, Environ. Model. Softw..

[8]  Alan R. Hevner,et al.  The Three Cycle View of Design Science , 2007, Scand. J. Inf. Syst..

[9]  Murray C. Peel,et al.  Equifinality and Flux Mapping: A New Approach to Model Evaluation and Process Representation Under Uncertainty , 2019, Water Resources Research.

[10]  Cees Leeuwis,et al.  Enhancing the Reflexivity of System Innovation Projects With System Analyses , 2010 .

[11]  A. Curtis,et al.  Assessing Conjunctive Use Opportunities with Stakeholders in the Murray-Darling Basin, Australia , 2019, Journal of Water Resources Planning and Management.

[12]  Joseph H. A. Guillaume,et al.  A framework for characterising and evaluating the effectiveness of environmental modelling , 2019, Environ. Model. Softw..

[13]  Takuya Iwanaga,et al.  A socio-environmental model for exploring sustainable water management futures: Participatory and collaborative modelling in the Lower Campaspe catchment , 2020 .

[14]  Fateme Zare,et al.  Structuring and evaluating decision support processes to enhance the robustness of complex human-natural systems , 2020, Environ. Model. Softw..

[15]  K. S. Murali,et al.  A reflection on collaborative adaptation research in Africa and Asia , 2017, Regional Environmental Change.

[16]  E. A. Moallemi,et al.  Coping with uncertainties of sustainability transitions using exploratory modelling: The case of the MATISSE model and the UK's mobility sector , 2019, Environmental Innovation and Societal Transitions.

[17]  S. Atkins,et al.  Reflection: a review of the literature. , 1993, Journal of advanced nursing.

[18]  Boru Douthwaite,et al.  Participatory Impact Pathways Analysis: a practical method for project planning and evaluation , 2010 .

[19]  P. Stålnacke,et al.  Integrated Water Resources Management , 2010 .

[20]  Michael Reynolds,et al.  Reflective practice: origins and interpretations , 2011 .

[21]  Derek Armitage,et al.  A resilience-based framework for evaluating adaptive co-management: Linking ecology, economics and society in a complex world , 2007 .

[22]  Sina Khatami,et al.  Nash-reinforcement learning (N-RL) for developing coordination strategies in non-transferable utility games , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[23]  J. Butler,et al.  Priming adaptation pathways through adaptive co-management: Design and evaluation for developing countries , 2016 .

[24]  E. A. Moallemi,et al.  An agent-monitored framework for the output-oriented design of experiments in exploratory modelling , 2018, Simul. Model. Pract. Theory.

[25]  Jeroen P. van der Sluijs,et al.  A way out of the credibility crisis of models used in integrated environmental assessment , 2002 .

[26]  Anthony J. Jakeman,et al.  Habitat suitability modelling of rare species using Bayesian networks: Model evaluation under limited data , 2015 .

[27]  Joseph H. A. Guillaume,et al.  Realizing modelling outcomes: A synthesis of success factors and their use in a retrospective analysis of 15 Australian water resource projects , 2017, Environ. Model. Softw..

[28]  Anthony J. Jakeman,et al.  Integrated assessment and modelling: features, principles and examples for catchment management , 2003, Environ. Model. Softw..

[29]  F. Wickson,et al.  Transdisciplinary research: characteristics, quandaries and quality , 2006 .

[30]  D. Kahneman Thinking, Fast and Slow , 2011 .

[31]  Mary C. Hill,et al.  Integrated environmental modeling: A vision and roadmap for the future , 2013, Environ. Model. Softw..

[32]  T. Dedeurwaerdere,et al.  A Pragmatist Approach to Transdisciplinarity in Sustainability Research: From Complex Systems Theory to Reflexive Science , 2015 .

[33]  P. Blackett,et al.  Reflexive monitoring in New Zealand: Evaluation lessons in supporting transformative change , 2015 .

[34]  Arthur C. Petersen,et al.  The reflective futures practitioner: Balancing salience, credibility and legitimacy in generating foresight knowledge with stakeholders , 2015 .

[35]  Anthony J. Jakeman,et al.  Integrated water assessment and modelling: A bibliometric analysis of trends in the water resource sector , 2017 .

[36]  J. Coutts,et al.  Evaluating for learning and accountability in system innovation: Incorporating reflexivity in a logical framework , 2017 .

[37]  Barbara J. Regeer,et al.  Six Guiding Principles for Evaluating Mode-2 Strategies for Sustainable Development , 2009 .

[38]  R. Gregory,et al.  Deconstructing adaptive management: criteria for applications to environmental management. , 2006, Ecological applications : a publication of the Ecological Society of America.

[39]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[40]  Raimo P. Hämäläinen,et al.  On the importance of behavioral operational research: The case of understanding and communicating about dynamic systems , 2013, Eur. J. Oper. Res..

[41]  Cees Leeuwis,et al.  Reflexive Monitoring in Action. A guide for monitoring system innovation projects , 2010 .

[42]  Joseph H. A. Guillaume,et al.  Reflective communication to improve problem-solving pathways: Key issues illustrated for an integrated environmental modelling case study , 2020, Environ. Model. Softw..

[43]  S. Fielke,et al.  Hitting the bullseye: Learning to become a reflexive monitor in New Zealand , 2017 .

[44]  François Molle,et al.  Nirvana concepts, narratives and policy models : insights from the water sector , 2008 .

[45]  Fateme Zare Designing and evaluating practices and tools to improve integrated assessment and modelling processes in the water sector , 2020 .

[46]  L. Finlay,et al.  Reflecting on ‘Reflective practice’ , 2008 .

[47]  Anthony J. Jakeman,et al.  Development of an integrated model for the Campaspe catchment: a tool to help improve understanding of the interaction between society, policy, farming decision, ecology, hydrology and climate , 2018, Proceedings of the International Association of Hydrological Sciences.

[48]  D. Roux,et al.  Framework for participative reflection on the accomplishment of transdisciplinary research programs , 2010 .

[49]  Robert MacIntosh,et al.  Reflexivity: recursion and relationality in organizational research processes , 2010 .

[50]  Cees Leeuwis,et al.  The need for reflexive evaluation approaches in development cooperation , 2015 .

[51]  Andrea Gregg,et al.  Learning From Doing: Lessons Learned From Designing and Developing an Educational Software Within a Heterogeneous Group , 2021, Int. J. Web Based Learn. Teach. Technol..

[52]  Wasana Bandara,et al.  Process modelling success factors and measures , 2007 .

[53]  N. Mauthner,et al.  Reflexive Accounts and Accounts of Reflexivity in Qualitative Data Analysis , 2003 .

[54]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[55]  Jennifer Badham,et al.  Effective modeling for Integrated Water Resource Management: A guide to contextual practices by phases and steps and future opportunities , 2019, Environ. Model. Softw..

[56]  Anthony J. Jakeman,et al.  Selecting among five common modelling approaches for integrated environmental assessment and management , 2013, Environ. Model. Softw..

[57]  Jan Rotmans,et al.  Integrated assessment: A growing child on its way to maturity , 1996 .

[58]  Sina Khatami,et al.  Water for Energy: Inconsistent Assessment Standards and Inability to Judge Properly , 2015 .

[59]  P. Glynn Integrated Environmental Modelling: human decisions, human challenges , 2015, Special Publications.

[60]  Janos J. Bogardi,et al.  Integrated Water Resources Management: Concept, Research and Implementation , 2016 .

[61]  H. Simon,et al.  Rational choice and the structure of the environment. , 1956, Psychological review.

[62]  Daniel E. Orenstein,et al.  A methodology for evaluating transdisciplinary research on coupled socio-ecological systems , 2018 .

[63]  Olivier Barreteau,et al.  The MEPPP Framework: A Framework for Monitoring and Evaluating Participatory Planning Processes , 2015, Environmental Management.

[64]  Roman Seidl,et al.  A functional-dynamic reflection on participatory processes in modeling projects , 2015, Ambio.

[65]  Anke Bergmann,et al.  Atenção fisioterapêutica no controle do linfedema secundário ao tratamento do câncer de mama: rotina do Hospital do Câncer III/Instituto Nacional de Câncer , 2016 .

[66]  S. Ball Policy Sociology and Critical Social Research: A Personal Review of Recent Education Policy and Policy Research. , 1997 .

[67]  Dawn Freshwater,et al.  Critical Reflection for Nursing and the Helping Professions : A User's Guide , 2001 .

[68]  Utilising reflective practice interviews in professional development , 2003 .