Toward rigorous use of expert knowledge in ecological research

Practicing ecologists who excel at their work (“experts”) hold a wealth of knowledge. This knowledge offers a wide range of opportunities for application in ecological research and natural resource decision-making. While experts are often consulted ad-hoc, their contributions are not widely acknowledged. These informal applications of expert knowledge lead to concerns about a lack of transparency and repeatability, causing distrust of this knowledge source in the scientific community. Here, we address these concerns with an exploration of the diversity of expert knowledge and of rigorous methods in its use. The effective use of expert knowledge hinges on an awareness of the spectrum of experts and their expertise, which varies by breadth of perspective and critical assessment. Also, experts express their knowledge in different forms depending on the degree of contextualization with other information. Careful matching of experts to application is therefore essential and has to go beyond a simple fitting of the expert to the knowledge domain. The standards for the collection and use of expert knowledge should be as rigorous as for empirical data. This involves knowing when it is appropriate to use expert knowledge and how to identify and select suitable experts. Further, it requires a careful plan for the collection, analysis and validation of the knowledge. The knowledge held by expert practitioners is too valuable to be ignored. But only when thorough methods are applied, can the application of expert knowledge be as valid as the use of empirical data. The responsibility for the effective and rigorous use of expert knowledge lies with the researchers.

[1]  Chris J. Johnson,et al.  Historical-ecological evaluation of the long-term distribution of woodland caribou and moose in central British Columbia , 2012 .

[2]  Brean W. Duncan,et al.  SETTING RELIABILITY BOUNDS ON HABITAT SUITABILITY INDICES , 2001 .

[3]  A. Clevenger,et al.  GIS‐Generated, Expert‐Based Models for Identifying Wildlife Habitat Linkages and Planning Mitigation Passages , 2002 .

[4]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[5]  Francisco Herrera,et al.  Combining Numerical and Linguistic Information in Group Decision Making , 1998, Inf. Sci..

[6]  H. V. D. Klis,et al.  Uncertainty analysis of a spatial habitat suitability model and implications for ecological management of water bodies , 2006, Landscape Ecology.

[7]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[8]  Erik Lebret,et al.  The use of expert elicitation in environmental health impact assessment: a seven step procedure , 2010, Environmental health : a global access science source.

[9]  Hubert L. Dreyfus,et al.  Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer , 1987, IEEE Expert.

[10]  Matthew J. Salganik,et al.  5. Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling , 2004 .

[11]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[12]  Kerrie L. Mengersen,et al.  Elicitator: An expert elicitation tool for regression in ecology , 2010, Environ. Model. Softw..

[13]  R. W. Carter,et al.  Conservation in the dark? The information used to support management decisions , 2010 .

[14]  Ioan Fazey,et al.  Learning more effectively from experience , 2005 .

[15]  D. Lindenmayer,et al.  The nature and role of experiential knowledge for environmental conservation , 2006, Environmental Conservation.

[16]  Ajith H. Perera,et al.  Uncertainty in expert knowledge of forest succession: A case study from boreal Ontario , 2008 .

[17]  K. Mengersen,et al.  Eliciting Expert Knowledge in Conservation Science , 2012, Conservation biology : the journal of the Society for Conservation Biology.

[18]  Andre Zerger,et al.  Eliciting and integrating expert knowledge for wildlife habitat modelling , 2003 .

[19]  R. Gregory,et al.  Deconstructing adaptive management: criteria for applications to environmental management. , 2006, Ecological applications : a publication of the Ecological Society of America.

[20]  Fridolin Zimmermann,et al.  Testing expert groups for a habitat suitability model for the lynx Lynx lynx in the Swiss Alps , 2007 .

[21]  Roger M. Cooke,et al.  Eliciting expert knowledge of ecosystem vulnerability to human stressors to support comprehensive ocean management , 2012 .

[22]  Anthony Davis,et al.  Constructing confidence: rational skepticism and systematic enquiry in local ecological knowledge research. , 2010, Ecological applications : a publication of the Ecological Society of America.

[23]  J. Tirole,et al.  Incentives and Prosocial Behavior , 2004 .

[24]  R. Gregory,et al.  When experts disagree (and better science won't help much): using structured deliberations to support endangered species recovery planning. , 2012, Journal of environmental management.

[25]  Hugh P. Possingham,et al.  How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rock-wallabies Petrogale penicillata , 2009 .

[26]  B. Marcot,et al.  A Bayesian approach to evaluating habitat for woodland caribou in north-central British Columbia. , 2006 .

[27]  Edward J. Rykiel,et al.  Artificial intelligence and expert systems in ecology and natural resource management , 1989 .

[28]  A. R. Halford,et al.  Crucial knowledge gaps in current understanding of climate change impacts on coral reef fishes , 2010, Journal of Experimental Biology.

[29]  George Wright,et al.  Improving the Delphi process: Lessons from social psychological research , 2011 .

[30]  Tara G Martin,et al.  A guide to eliciting and using expert knowledge in Bayesian ecological models. , 2010, Ecology letters.

[31]  Chris J. Johnson,et al.  Maintaining or restoring connectivity of modified landscapes: evaluating the least-cost path model with multiple sources of ecological information , 2010, Landscape Ecology.

[32]  Claudia Pahl-Wostl,et al.  Toward a Relational Concept of Uncertainty: about Knowing Too Little, Knowing Too Differently, and Accepting Not to Know , 2008 .

[33]  M. Burgman.  Risks and Decisions for Conservation and Environmental Management: Experts, stakeholders and elicitation , 2005 .

[34]  Helen M. Regan,et al.  Mapping epistemic uncertainties and vague concepts in predictions of species distribution , 2002 .

[35]  Ajith H. Perera,et al.  Comparing two sets of forest cover change knowledge used in forest landscape management planning , 2010 .

[36]  Mike Hulme,et al.  Using expert knowledge to assess uncertainties in future polar bear populations under climate change , 2008 .

[37]  Kerrie L. Mengersen,et al.  A proposed validation framework for expert elicited Bayesian Networks , 2013, Expert Syst. Appl..

[38]  Bruno S. Frey,et al.  Does pay for performance really motivate employees , 2002 .

[39]  J. Evetts.  The Sociological Analysis of Professionalism , 2003 .

[40]  Stephen C. Hora,et al.  Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management , 1996 .

[41]  Kerrie Mengersen,et al.  Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. , 2009, Ecology.

[42]  P. Tetlock.  Expert Political Judgment: How Good Is It? How Can We Know? , 2005 .

[43]  Eastman J. Ronald,et al.  RASTER PROCEDURES FOR MULTI-CRITERIA/MULTI-OBJECTIVE DECISIONS , 1995 .

[44]  Helen M. Regan,et al.  A TAXONOMY AND TREATMENT OF UNCERTAINTY FOR ECOLOGY AND CONSERVATION BIOLOGY , 2002 .

[45]  William J. Sutherland,et al.  Predicting the ecological consequences of environmental change: a review of the methods* , 2006 .

[46]  Kerrie Mengersen,et al.  Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush‐tailed rock‐wallaby Petrogale penicillata , 2009 .

[47]  Laura Johnson,et al.  How Many Interviews Are Enough? , 2006 .

[48]  Javier Seoane,et al.  Effect of Expert Opinion on the Predictive Ability of Environmental Models of Bird Distribution , 2005 .

[49]  M. Smithson,et al.  Conflict Aversion: Preference for Ambiguity vs Conflict in Sources and Evidence. , 1999, Organizational behavior and human decision processes.

[50]  Omar Al-Ubaydli,et al.  Can Tailored Communications Motivate Volunteers? A Field Experiment , 2011 .

[51]  Colin MacDougall,et al.  The Devil's Advocate: A Strategy to Avoid Groupthink and Stimulate Discussion in Focus Groups , 1997 .

[52]  H. Simon,et al.  Perception in chess , 1973 .

[53]  P. Bauer,et al.  Evaluation of experiments with adaptive interim analyses. , 1994, Biometrics.

[54]  Davide Geneletti,et al.  Towards a common set of criteria and indicators to identify forest restoration priorities: An expert panel-based approach , 2011 .

[55]  Paul J. Feltovich,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[56]  Virginia H. Dale,et al.  ECOLOGICAL SUPPORT FOR RURAL LAND‐USE PLANNING , 2005 .

[57]  M. Cooke,et al.  Researching the researchers: using a snowballing technique. , 2004, Nurse researcher.

[58]  Kerrie Mengersen,et al.  Elicitator: A User-Friendly, Interactive Tool to Support Scenario-Based Elicitation of Expert Knowledge , 2012, HiPC 2012.

[59]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[60]  David A. Cleaves,et al.  Assessing Uncertainty in Expert Judgments About Natural Resources , 1994 .

[61]  Roger M. Cooke,et al.  Future declines of the binational Laurentian Great Lakes fisheries : the importance of environmental and cultural change , 2010 .

[62]  A. McKerrow,et al.  Use of Occupancy Models to Evaluate Expert Knowledge-based Species-Habitat Relationships , 2012 .

[63]  C. Fabricius,et al.  Expert and Generalist Local Knowledge about Land-cover Change on South Africa's Wild Coast: Can Local Ecological Knowledge Add Value to Science? , 2007 .

[64]  A. Bryman,et al.  The Sage Encyclopedia of Social Science Research Methods (three volumes) , 2003 .

[65]  Mark A. Burgman,et al.  Redefining expertise and improving ecological judgment , 2011 .

[66]  Ajith H. Perera,et al.  Expert Knowledge as a Basis for Landscape Ecological Predictive Models , 2011 .

[67]  W. Aspinall.  A route to more tractable expert advice , 2010, Nature.

[68]  A. Yancey,et al.  Effective recruitment and retention of minority research participants. , 2006, Annual review of public health.

[69]  Graham T. F. Horn,et al.  Using Expert Judgment and Stakeholder Values to Evaluate Adaptive Management Options , 2004 .

[70]  Christina A. Czembor,et al.  Incorporating between-expert uncertainty into state-and-transition simulation models for forest restoration. , 2009 .

[71]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[72]  J. Kadane,et al.  Experiences in elicitation , 1998 .

[73]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[74]  Lindsay A. Turnbull,et al.  Identification of 100 fundamental ecological questions , 2013 .

[75]  Petra M. Kuhnert,et al.  Four case studies in using expert opinion to inform priors , 2011 .

[76]  Ricardo M. Rodriguez Iglesias,et al.  Evaluating expert knowledge: plant species responses to cattle grazing and fire. , 1998 .

[77]  C. Folke,et al.  Local Ecological Knowledge and Institutional Dynamics for Ecosystem Management: A Study of Lake Racken Watershed, Sweden , 2001, Ecosystems.

[78]  Gareth W. Parry,et al.  The characterization of uncertainty in probabilistic risk assessments of complex systems , 1996 .

[79]  H. Marsh,et al.  Rapid Assessment of Risks to a Mobile Marine Mammal in an Ecosystem‐Scale Marine Protected Area , 2008, Conservation biology : the journal of the Society for Conservation Biology.

[80]  Bonnie H. Erickson,et al.  Some Problems of Inference from Chain Data , 1979 .

[81]  E. Shevliakova,et al.  Elicitation of Expert Judgments of Climate Change Impacts on Forest Ecosystems , 2001 .

[82]  Ajith H. Perera,et al.  A network approach for evaluating and communicating forest change models , 2010 .

[83]  Clyde W. Holsapple,et al.  Handbook Of Knowledge Management: Knowledge Matters , 2004 .

[84]  Shankar Aswani,et al.  Indigenous Knowledge and Long-term Ecological Change: Detection, Interpretation, and Responses to Changing Ecological Conditions in Pacific Island Communities , 2010, Environmental management.

[85]  Andrew P. Sage,et al.  On the management of information imperfection in knowledge based systems , 1986, IPMU.

[86]  Bilal M. Ayyub,et al.  Elicitation of expert opinions for uncertainty and risks: Answer to the Book Review by Roger M. Cooke , 2003, Fuzzy Sets Syst..

[87]  L. Firbank,et al.  Knowledge‐based models for predicting species occurrence in arable conditions , 2003 .

[88]  Roland W Scholz,et al.  Combining Experts' Risk Judgments on Technology Performance of Phytoremediation: Self‐Confidence Ratings, Averaging Procedures, and Formative Consensus Building , 2007, Risk analysis : an official publication of the Society for Risk Analysis.

[89]  Kimberly M. Christopherson.  The positive and negative implications of anonymity in Internet social interactions: "On the Internet, Nobody Knows You're a Dog" , 2007, Comput. Hum. Behav..

[90]  C. J. Moore,et al.  Knowledge elicitation using more than one expert to cover the same domain , 1991, Artificial Intelligence Review.

[91]  Douglas D. Heckathorn,et al.  Respondent-driven sampling : A new approach to the study of hidden populations , 1997 .

[92]  K. Ribisl,et al.  Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: Review and recommendations , 1996 .

[93]  Sakari Kuikka,et al.  Estimation of Atlantic salmon smolt carrying capacity of rivers using expert knowledge , 2005 .

[94]  David H. Gustafson,et al.  Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes , 1976 .

[95]  Brian Dennis,et al.  Discussion: Should Ecologists Become Bayesians? , 1996 .

[96]  Grant Gilchrist,et al.  Can Local Ecological Knowledge Contribute to Wildlife Management? Case Studies of Migratory Birds , 2005 .

[97]  T. L. Saaty.  A Scaling Method for Priorities in Hierarchical Structures , 1977 .

[98]  Henry P. Huntington,et al.  USING TRADITIONAL ECOLOGICAL KNOWLEDGE IN SCIENCE: METHODS AND APPLICATIONS , 2000 .

[99]  L. J. Valverde,et al.  Expert Judgment Resolution in Technically-Intensive Policy Disputes , 2001 .

[100]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[101]  R N Coulson,et al.  Artificial intelligence and natural resource management. , 1987, Science.

[102]  Jane M. Booker,et al.  Eliciting and analyzing expert judgement - a practical guide , 2001, ASA-SIAM series on statistics and applied probability.

[103]  Robert L. Pressey,et al.  The expert or the algorithm?—comparison of priority conservation areas in the Cape Floristic Region identified by park managers and reserve selection software , 2003 .

[104]  Elizabeth C. Hirschman,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[105]  David Bart,et al.  Integrating local ecological knowledge and manipulative experiments to find the causes of environmental change , 2006 .

[106]  K. Mengersen,et al.  The power of expert opinion in ecological models using Bayesian methods: impact of grazing on birds , 2005 .

[107]  Chris J. Johnson,et al.  Utility of Expert-Based Knowledge for Predicting Wildlife–Vehicle Collisions , 2009 .

[108]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[109]  S. Page.  Prologue to The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies , 2007 .

[110]  D. Wilkinson,et al.  Modelling disease spread in a novel host: rabies in the European badger Meles meles , 2002 .

[111]  O. Boiral.  Tacit Knowledge and Environmental Management , 2002 .

[112]  Chris J. Johnson,et al.  Mapping uncertainty: sensitivity of wildlife habitat ratings to expert opinion , 2004 .

[113]  A. Perera,et al.  Expert Knowledge and Its Application in Landscape Ecology , 2012, Springer New York.

[114]  R. Cowling,et al.  One Hundred Questions of Importance to the Conservation of Global Biological Diversity , 2009, Conservation biology : the journal of the Society for Conservation Biology.

[115]  Rhys Bullman,et al.  Expert opinion as a tool for quantifying bird tolerance to human disturbance , 2008 .

[116]  Michael J. Samways,et al.  Planning for biodiversity conservation based on the knowledge of biologists , 2000, Biodiversity & Conservation.

[117]  Ruth Beilin,et al.  Biosecurity risk and peri‐urban landholders – using a stakeholder consultative approach to build a risk communication strategy , 2011 .

[118]  Janet M. Carey,et al.  Linguistic Uncertainty in Qualitative Risk Analysis and How to Minimize It , 2008, Annals of the New York Academy of Sciences.

[119]  Mark A. Burgman,et al.  Expert Status and Performance , 2011, PloS one.

[120]  Ajith H. Perera,et al.  Experts, Expert Knowledge, and Their Roles in Landscape Ecological Applications , 2012 .

[121]  Jyrki Kangas,et al.  Modelling ecological expertise for forest planning calculations-rationale, examples, and pitfalls. , 2005, Journal of environmental management.

[122]  Chris M. Theobald,et al.  Assessment of Expert Opinion: Seasonal Sheep Preference and Plant Response to Grazing , 2007 .

[123]  J. Bentley.  What farmers don't know can't help them: The strengths and weaknesses of indigenous technical knowledge in Honduras , 1989 .

[124]  Mark A. Burgman,et al.  Evaluating extreme risks in invasion ecology: learning from banking compliance , 2008 .

[125]  M. Reed.  Stakeholder participation for environmental management: A literature review , 2008 .

[126]  John M. Kovacs,et al.  Perceptions of environmental change in a tropical coastal wetland , 2000 .

[127]  Grant Ballard,et al.  EFFECTS OF FLIPPER BANDS ON FORAGING BEHAVIOR AND SURVIVAL OF ADÉLIE PENGUINS (PYGOSCELIS ADELIAE) , 2006 .

[128]  Jane M. Booker,et al.  Solving black box computation problems using expert knowledge theory and methods , 2004, Reliab. Eng. Syst. Saf..

[129]  Ilan Fischer,et al.  When do those who know more also know more about how much they know? The development of confidence and performance in categorical decision tasks , 2005 .

[130]  Louis Liebenberg,et al.  The Value, Limitations, and Challenges of Employing Local Experts in Conservation Research , 2011, Conservation biology : the journal of the Society for Conservation Biology.

[131]  David Bart,et al.  Using Weed Control Knowledge from Declining Agricultural Communities in Invasive-Species Management , 2010 .

[132]  Kerrie L. Mengersen,et al.  A software tool for elicitation of expert knowledge about species richness or similar counts , 2012, Environ. Model. Softw..

[133]  Leonard Michael Jessup,et al.  Decision Making in an Automated Environment: The Effects of Anonymity and Proximity with a Group Decision Support System* , 1991 .

[134]  Judith Rousseau,et al.  Combining expert opinions in prior elicitation , 2010 .

[135]  Lotfi A. Zadeh,et al.  Fuzzy Sets , 1996, Inf. Control..

[136]  N. Cowan,et al.  Separating cognitive capacity from knowledge: a new hypothesis , 2007, Trends in Cognitive Sciences.

[137]  Mark L. Mitchell,et al.  Research Design Explained , 1987 .

[138]  Mark A. Burgman,et al.  Evaluating the accuracy and calibration of expert predictions under uncertainty: predicting the outcomes of ecological research , 2012 .

[139]  I. Pérez,et al.  Evaluation of Local Ecological Knowledge as a Method for Collecting Extensive Data on Animal Abundance , 2009, Conservation biology : the journal of the Society for Conservation Biology.

[140]  Bonnie C. Wintle,et al.  Interpreting risk in international trade , 2012 .

[141]  Peter J Pronovost,et al.  Systematic review identifies number of strategies important for retaining study participants. , 2007, Journal of clinical epidemiology.

[142]  Markus Neuhäuser,et al.  An adaptive interim analysis a useful tool for ecological studies , 2001 .

[143]  Bronwyn Price,et al.  Using a Bayesian belief network to predict suitable habitat of an endangered mammal – The Julia Creek dunnart (Sminthopsis douglasi) , 2007 .

[144]  Torsten Grothmann,et al.  Guiding principles for adaptation to climate change in Europe. , 2010 .