Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method.

Most Transdisciplinary Research (TdR) projects combine scientific research with the building of decision making capacity for the involved stakeholders. These projects usually deal with complex, societally relevant, real-world problems. This paper focuses on TdR projects, which integrate the knowledge of researchers and stakeholders in a collaborative transdisciplinary process through structured methods of mutual learning. Previous research on the evaluation of TdR has insufficiently explored the intended effects of transdisciplinary processes on the real world (societal effects). We developed an evaluation framework for assessing the societal effects of transdisciplinary processes. Outputs (measured as procedural and product-related involvement of the stakeholders), impacts (intermediate effects connecting outputs and outcomes) and outcomes (enhanced decision making capacity) are distinguished as three types of societal effects. Our model links outputs and outcomes of transdisciplinary processes via the impacts using a mediating variables approach. We applied this model in an ex post evaluation of a transdisciplinary process. 84 out of 188 agents participated in a survey. The results show significant mediation effects of the two impacts "network building" and "transformation knowledge". These results indicate an influence of a transdisciplinary process on the decision making capacity of stakeholders, especially through social network building and the generation of knowledge relevant for action.

[1]  Bruce Tonn,et al.  A Framework for Understanding and Improving Environmental Decision Making , 2000 .

[2]  Luke Georghiou,et al.  Issues in the Evaluation of Innovation and Technology Policy , 1998 .

[3]  J. Hillier.  Fighting over the Forests: Environmental Conflict and Decision-making Capacity in Forest Planning Processes , 2003 .

[4]  Martin Schweitzer,et al.  Measuring the success of public participation on environmental restoration and waste management activities in the U.S. Department of Energy , 1998 .

[5]  Víctor-Andrés Cloquell-Ballester,et al.  Indicators validation for the improvement of environmental and social impact quantitative assessment , 2006 .

[6]  David M. Konisky,et al.  Innovations in Public Participation and Environmental Decision Making: Examples from the Great Lakes Region , 2001 .

[7]  H. Nowotny,et al.  The place of people in our knowledge , 1999, European Review.

[8]  Roland W. Scholz,et al.  Appenzell Ausserrhoden: Umwelt - Wirtschaft - Region , 2003 .

[9]  C. Pohl,et al.  Implications of transdisciplinarity for sustainability research , 2006 .

[10]  Roland W. Scholz,et al.  Critical success conditions of collaborative methods: a comparative evaluation of transport planning projects , 2007 .

[11]  T. Gilovich,et al.  The experience of regret: what, when, and why. , 1995, Psychological review.

[12]  S. Funtowicz,et al.  Science for the Post-Normal Age , 1993, Commonplace.

[13]  Frank Vanclay,et al.  The International Handbook of Social Impact Assessment: Conceptual and Methodological Advances , 2006 .

[14]  D. Aylward,et al.  Some recent developments in the evaluation of university research outcomes in the United Kingdom , 2000 .

[15]  Norbert Schwarz,et al.  Asking Questions About Behavior: Cognition, Communication, and Questionnaire Construction , 2001 .

[16]  M. Law,et al.  A model of impacts of research partnerships in health and social services , 2005 .

[17]  Gary Fry,et al.  Integrative studies on rural landscapes: policy expectations and research practice , 2005 .

[18]  S. West,et al.  A comparison of methods to test mediation and other intervening variable effects. , 2002, Psychological methods.

[19]  Roderic A. Gill,et al.  Planning for sustainability as a learning concept , 1998 .

[20]  P. Burger.  Embedded Case Study Methods: Integrating Quantitative and Qualitative Knowledge , 2001 .

[21]  James Tansey,et al.  Co-production, emergent properties and strong interactive social research: the Georgia Basin Futures Project , 2006 .

[22]  John Randolph,et al.  IMPROVING ENVIRONMENTAL DECISION-MAKING THROUGH COLLABORATIVE METHODS , 2005 .

[23]  C. Pohl,et al.  Handbook of transdisciplinary research , 2008 .

[24]  A. Grunwald.  Strategic knowledge for sustainable development: the need for reflexivity and learning at the interface between science and society , 2004 .

[25]  Stewart I. Donaldson,et al.  Mediator and Moderator Analysis in Program Development , 2001 .

[26]  M. Gibbons,et al.  Re-Thinking Science: Knowledge and the Public in an Age of Uncertainty , 2003 .

[27]  Terttu Luukkonen,et al.  Research evaluation in Europe: state of the art , 2002 .

[28]  G. Rowe,et al.  Evaluating Public-Participation Exercises: A Research Agenda , 2004 .

[29]  Harald A. Mieg,et al.  System experts and decision making experts in transdisciplinary projects , 2006 .

[30]  Kristopher J Preacher,et al.  SPSS and SAS procedures for estimating indirect effects in simple mediation models , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[31]  Sally Davenport,et al.  Collaboration and organisational learning: a study of a New Zealand collaborative research program , 1999 .

[32]  Kristopher J Preacher,et al.  Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models , 2008, Behavior research methods.

[33]  Thierry Ramadier,et al.  Transdisciplinarity and its challenges: the case of urban studies , 2004 .

[34]  A. Petrosino.  Mediators and Moderators in the Evaluation of Programs for Children , 2000, Evaluation review.

[35]  Thomas C. Beierle.  USING SOCIAL GOALS TO EVALUATE PUBLIC PARTICIPATION IN ENVIRONMENTAL DECISIONS , 2005 .

[36]  Christian Pohl,et al.  Transdisciplinary collaboration in environmental research , 2005 .

[37]  S. Schwartzman,et al.  The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies , 1994 .

[38]  Arnim Wiek,et al.  Challenges of transdisciplinary research as interactive knowledge generation experiences from transdisciplinary case study research , 2007 .

[39]  Arnim Wiek,et al.  Learning to research environmental problems from a functional socio‐cultural constructivism perspective: The transdisciplinary case study approach , 2006 .

[40]  Dealing with diversity in scientific outputs: implications for international research evaluation , 2004 .

[41]  Preben H. Lindøe,et al.  Trailing research based evaluation; phases and roles , 2004 .

[42]  W. Clark,et al.  Sustainability science: The emerging research program , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[43]  Roland W. Scholz,et al.  Managing Transition in Clusters: Area Development Negotiations as a Tool for Sustaining Traditional Industries in a Swiss Prealpine Region , 2007 .

[44]  Martin O'Connor,et al.  Dialogue and debate in a post-normal practice of science: a reflexion , 1999 .

[45]  D. Stokols,et al.  In vivo studies of transdisciplinary scientific collaboration Lessons learned and implications for active living research. , 2005, American journal of preventive medicine.

[46]  Christian Bockstaller,et al.  How to validate environmental indicators , 2003 .

[47]  Roger L. Brown.  Assessing specific mediational effects in complex theoretical models , 1997 .

[48]  R. Scholz,et al.  Transdisciplinarity : joint problem solving among science, technology, and society : an effective way for managing complexity , 2001 .

[49]  R. Margerum.  Collaborative Planning , 2002 .

[50]  J. Rugh,et al.  Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints , 2004 .

[51]  G. Tress,et al.  Clarifying Integrative Research Concepts in Landscape Ecology , 2005, Landscape Ecology.

[52]  Peter Loukopoulos,et al.  Sustainable Future Urban Mobility: Using ‘Area Development Negotiations’ for Scenario Assessment and Participatory Strategic Planning , 2004 .

[53]  Martin O'Connor,et al.  Science for the Twenty‐First Century: From Social Contract to the Scientific Core , 2001 .

[54]  Susan E. Cozzens,et al.  The knowledge pool: Measurement challenges in evaluating fundamental research programs , 1997 .

[55]  Gabriele Bammer,et al.  Integration and Implementation Sciences: Building a New Specialisation , 2005 .

[56]  N. Schwarz,et al.  Debiasing the hindsight bias: The role of accessibility experiences and (mis)attributions , 2003 .

[57]  Joe Ravetz,et al.  Integrated assessment for sustainability appraisal in cities and regions , 2000 .

[58]  Arnim Wiek,et al.  Transdisciplinary case studies as a means of sustainability learning: Historical framework and theory , 2006 .

[59]  Daniel Müller,et al.  A transdisciplinary learning approach to foster convergence of design, science and deliberation in urban and regional planning , 2005 .

[60]  Frank Vanclay,et al.  Conceptualizing Social Change Processes and Social Impacts , 2003 .

[61]  Roland W. Scholz,et al.  Transdisciplinarity in Groundwater Management — Towards Mutual Learning of Science and Society , 2000 .

[62]  David H. Marks,et al.  Learning about transdisciplinarity: Where are we? Where have we been? Where should we go? , 2001 .

[63]  Mark W. Lipsey,et al.  Evaluation: A Systematic Approach , 1979 .

[64]  Steven Yale Sussman,et al.  Handbook of Program Development for Health Behavior Research and Practice , 2001 .

[65]  J. Friedmann.  Toward a Non-Euclidian Mode of Planning , 1993 .

[66]  Eliot R. Smith,et al.  Research methods in social relations , 1962 .

[67]  Steven C. Currall,et al.  Measuring trust between organizational boundary role persons. , 1995 .

[68]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[69]  Frank Vanclay,et al.  Conceptualising social impacts , 2002 .

[70]  M. Siegrist,et al.  Morality Information, Performance Information, and the Distinction Between Trust and Confidence1 , 2006 .