Reliability and Validity in Nonmarket Valuation

A central goal of nonmarket valuation studies is to provide accurate value estimates. We suggest a systematic, comprehensive approach to accuracy assessment using the twin concepts of reliability and validity. Reliability has to do with variance; validity has to do with bias. If procedures applied in a valuation study are produce erratic results, accuracy suffers even if those procedures are unbiased. And, even if the procedures in question are reliable, they will be less useful if the procedures applied produce large biases in the estimates. We adapt the general concepts of reliability and validity to apply to nonmarket valuation studies. As in many other disciplines, the concept to be measured, the “true values,” is unobservable. Hence, criteria must be developed to serve as indicators of accuracy. Reliability is typically observed using the estimated standard error of the mean from repeated estimates of value. Validity is assessed using the concepts of content validity, construct validity, and criterion validity; what we refer to as the “three Cs” of validity. After fleshing out these concept, we illustrate how they can be applied using two case studies. Contingent valuation serves as an example of stated preference methods. The travel cost method provides a case study of revealed preference methods.

[1]  Jerry A. Hausman,et al.  Contingent valuation : a critical assessment , 1993 .

[2]  Frank Lupi,et al.  Using an Economic Model of Recreational Fishing to Evaluate the Benefits of Sea Lamprey (Petromyzon marinus) Control on the St. Marys River , 2003 .

[3]  Frank J. Cesario,et al.  Time Bias in Recreation Benefit Estimates , 1970 .

[4]  Thomas C. Brown,et al.  A primer on nonmarket valuation , 2003 .

[5]  Christian A. Vossler,et al.  Externally validating contingent valuation: an open-space survey and referendum in Corvallis, Oregon , 2003 .

[6]  P. Boxall,et al.  Understanding Heterogeneous Preferences in Random Utility Models: A Latent Class Approach , 2002 .

[7]  Klaus Moeltner,et al.  Choice Behavior Under Time-Variant Quality , 2004 .

[8]  Alan Randall,et al.  A Difficulty with the Travel Cost Method , 1994 .

[9]  R. Kopp,et al.  Contingent Valuation and Lost Passive Use: Damages from the Exxon Valdez Oil Spill , 2003 .

[10]  Christian A. Vossler,et al.  Contemporary Guidance for Stated Preference Studies , 2017, Journal of the Association of Environmental and Resource Economists.

[11]  R. Berrens.  Reluctant respondents and contingent valuation surveys , 2000 .

[12]  J. Loomis,et al.  Meta-Analytic Benefit Transfer of Outdoor Recreation Economic Values: Testing Out-of-Sample Convergent Validity , 2003 .

[13]  Gregory L. Poe,et al.  Consequentiality and contingent values: an emerging paradigm , 2009 .

[14]  Jason C. Kinnell,et al.  Estimating Site Choice Decisions for Urban Recreators , 2006, Land Economics.

[15]  J. Hausman,et al.  ON CONTINGENT VALUATION MEASUREMENT OF NONUSE VALUES , 1993 .

[16]  J. List,et al.  Consequentiality: A Theoretical and Experimental Exploration of a Single Binary Choice , 2014, Journal of the Association of Environmental and Resource Economists.

[17]  J. Bennett,et al.  Minimising Payment Vehicle Bias in Contingent Valuation Studies , 2000 .

[18]  J. List,et al.  Using Ex Ante Approaches to Obtain Credible Signals for Value in Contingent Markets: Evidence from the Field , 2007 .

[19]  Timothy C. Haab,et al.  Valuing Environmental and Natural Resources: The Econometrics of Non-Market Valuation , 2002 .

[20]  Richard T. Carson,et al.  Contingent Valuation: A Comprehensive Bibliography and History , 2012 .

[21]  J. Hausman.  Contingent Valuation: From Dubious to Hopeless , 2012 .

[22]  David J. Bjornstad,et al.  A Learning Design for Reducing Hypothetical Bias in the Contingent Valuation Method , 1997 .

[23]  Robert P. Berrens,et al.  Contingent values for New Mexico instream flows : With tests of scope, group-size reminder and temporal reliability , 2000 .

[24]  E. Loehman,et al.  Application of Stochastic Choice Modeling to Policy Analysis of Public Goods: A Case Study of Air Quality Improvements , 1982 .

[25]  Bryan J. Hubbell,et al.  An Empirical Bayes Approach to Combining and Comparing Estimates of the Value of a Statistical Life for Environmental Policy Analysis , 2006 .

[26]  Ian J. Bateman,et al.  Temporal stability and transferability of models of willingness to pay for flood control and wetland conservation , 2005 .

[27]  W. Michael Hanemann,et al.  Estimating the Value of Water Quality Improvements in a Recreational Demand Framework , 1987 .

[28]  K. S. Carson,et al.  Resolving questions about bias in real and hypothetical referenda , 2005 .

[29]  K. McConnell,et al.  Testing Temporal Reliability and Carry-over Effect: The Role of Correlated Responses in Test-retest Reliability Studies , 1998 .

[30]  Nick Hanley,et al.  Valuing rural recreation benefits: an empirical comparison of two approaches. , 1989 .

[31]  Richard T. Carson,et al.  Contingent Valuation and Revealed Preference Methodologies: Comparing the Estimates for Quasi-Public Goods , 1995 .

[32]  Peter B. Bayley,et al.  Effects of recall bias and nonresponse bias on self-report estimates of angling participation. , 1993 .

[33]  R. G. Cummings,et al.  Are Hypothetical Referenda Incentive Compatible? , 1997, Journal of Political Economy.

[34]  L. Stinson,et al.  Income Measurement Error in Surveys: A Review , 2000 .

[35]  S. Newbold,et al.  Valuing water quality changes using a bioeconomic model of a coastal recreational fishery , 2006 .

[36]  John B. Loomis,et al.  Testing a meta-analysis model for benefit transfer in international outdoor recreation , 2001 .

[37]  Y. Kaoru.  Measuring marine recreation benefits of water quality improvements by the nested random utility model , 1995 .

[38]  G. Parsons,et al.  Gauging the Value of Short-Term Site Closures in a Travel-Cost RUM Model of Recreation Demand With a Little Help from Stated Preference Data , 2011 .

[39]  Luke Brander,et al.  The recreational value of coral reefs: A meta-analysis , 2007 .

[40]  金思憲.  Valuing Nature with Travel Cost Models: A Manual, Frank A..Ward and Diana Beal 共著, Cheltenham, U.K: Edgar Elgar Publishing, Inc. 255쪽, 2000년 UKL 49.95 , 2000 .

[41]  Kevin J. Boyle,et al.  Measuring Nonuse Damages Using Contingent Valuation: An Experimental Evaluation of Accuracy , 2010 .

[42]  Frank A. Ward,et al.  Valuing Nature with Travel Cost Models , 2000 .

[43]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[44]  Kenneth Train,et al.  Utility in Willingness to Pay Space: A Tool to Address Confounding Random Scale Effects in Destination Choice to the Alps , 2008 .

[45]  P. Boxall,et al.  HETEROGENEOUS PREFERENCES FOR CONGESTION DURING A WILDERNESS EXPERIENCE , 2003 .

[46]  K. Boyle,et al.  Temporal Reliability of Contingent Values , 1990 .

[47]  Thomas C. Brown,et al.  Further tests of entreaties to avoid hypothetical bias in referendum contingent valuation , 2003 .

[48]  R. Bishop,et al.  Nonpoint Source Pollution and Present Values: A Contingent Valuation Study of Lake Mendota , 2001 .

[49]  R. Berrens,et al.  Valuing the Loss of Rock Climbing Access in Wilderness Areas: A National-Level, Random-Utility Model , 2002, Land Economics.

[50]  Christian A. Vossler,et al.  Truth in Consequentiality: Theory and Field Evidence on Discrete Choice Experiments , 2010 .

[51]  C. Kling,et al.  Valuing Environmental Quality: Weak Complementarity with Sets of Goods , 1988 .

[52]  M. Teisl,et al.  Test-Retest Reliability of Contingent Valuation with Independent Sample Pretest and Posttest Control Groups , 1995 .

[53]  Henrik Lindhjem,et al.  Asking for Individual or Household Willingness to Pay for Environmental Goods? , 2008 .

[54]  Daniel McFadden,et al.  Contingent Valuation and Social Choice , 1994 .

[55]  J. Herriges,et al.  Kuhn-Tucker Estimation of Recreation Demand – A Study of Temporal Stability , 2009 .

[56]  Incentive Compatible Referenda and the Valuation of Environmental Goods , 1998, Agricultural and Resource Economics Review.

[57]  Mary Jo Kealy,et al.  Accuracy in Valuation Is a Matter of Degree , 1988 .

[58]  John B. Loomis,et al.  Using meta‐analysis for benefit transfer: In‐sample convergent validity tests of an outdoor recreation database , 2000 .

[59]  Richard C. Bishop,et al.  Rethinking the scope test as a criterion for validity in contingent valuation , 2005 .

[60]  David R. Just,et al.  Internal and External Validity in Economics Research: Tradeoffs between Experiments, Field Experiments, Natural Experiments, and Field Data , 2009 .

[61]  J. Whitehead,et al.  From Hopeless to Curious? Thoughts on Hausman's “Dubious to Hopeless” Critique of Contingent Valuation , 2013 .

[62]  J. Downing,et al.  The Role of Water Quality Perceptions in Modelling Lake Recreation Demand , 2005 .

[63]  K. McConnell,et al.  Valuing Environmental and Natural Resources , 2002 .

[64]  Catherine L. Kling,et al.  Understanding Behavioral Explanations of the WTP-WTA Divergence Through a Neoclassical Lens: Implications for Environmental Policy , 2015 .

[65]  Mary Jo Kealy,et al.  Reliability and predictive validity of contingent values: Does the nature of the good matter? , 1990 .

[66]  M. Hammerton,et al.  THE VALUE OF SAFETY: RESULTS OF A NATIONAL SAMPLE SURVEY. IN: URBAN TRANSPORT , 1985 .

[67]  Nicholas E. Flores,et al.  Contingent Valuation: Controversies and Evidence , 2000 .

[68]  Nick Hanley,et al.  Valuing non-market goods using contingent valuation , 1989 .

[69]  K. McConnell.  CONGESTION AND WILLINGNESS TO PAY: A STUDY OF BEACH USE , 1977 .

[70]  J. Payne,et al.  How People Respond to Contingent Valuation Questions: A Verbal Protocol Analysis of Willingness to Pay for an Environmental Regulation , 1994 .

[71]  R. Berrens,et al.  Testing the Validity of Contingent Behavior Trip Responses , 2002 .

[72]  Edward G. Carmines,et al.  Measurement in the Social Sciences: The Link Between Theory and Data , 1980 .

[73]  Nick Hanley,et al.  Are There Income Effects on Global Willingness to Pay for Biodiversity Conservation? , 2009 .

[74]  V. Kerry Smith,et al.  What have we learned since hotelling's letter?: A meta-analysis , 1990 .

[75]  R. G. Cummings,et al.  Does Realism Matter in Contingent Valuation Surveys , 1998 .

[76]  John B. Loomis,et al.  Test-Retest Reliability of the Contingent Valuation Method: A Comparison of General Population and Visitor Responses , 1989 .

[77]  K. Boyle,et al.  CONTINGENT VALUATION OF A PUBLIC PROGRAM TO CONTROL BLACK FLIES , 1989 .

[78]  K. Train,et al.  Adequate responsiveness to scope in contingent valuation , 2012 .

[79]  John B. Loomis,et al.  Comparative reliability of the dichotomous choice and open-ended contingent valuation techniques , 1990 .

[80]  V. Kerry Smith,et al.  Signals or Noise? Explaining the Variation in Recreation Benefit Estimates , 1990 .

[81]  Craig E. Landry,et al.  A semi-parametric estimator for revealed and stated preference data--An application to recreational beach visitation , 2009 .

[82]  John A. List,et al.  What Experimental Protocol Influence Disparities Between Actual and Hypothetical Stated Values? , 2001 .

[83]  Felix Schläpfer.  Survey protocol and income effects in the contingent valuation of public goods: A meta-analysis , 2006 .

[84]  J. Downing,et al.  Valuing Water Quality as a Function of Water Quality Measures , 2008 .

[85]  M. Loureiro,et al.  How Sensitive Are Environmental Valuations To Economic Downturns , 2017 .

[86]  Daniel Kahneman,et al.  Valuing public goods: The purchase of moral satisfaction , 1992 .

[87]  Nick Hanley,et al.  Aggregating the benefits of environmental improvements: distance-decay functions for use and non-use values. , 2003, Journal of environmental management.

[88]  N. Hanley,et al.  Up the Proverbial Creek without a Paddle: Accounting for Variable Participant Skill Levels in Recreational Demand Modelling , 2007 .

[89]  Daniel J. Phaneuf,et al.  A random utility model for total maximum daily loads: Estimating the benefits of watershed‐based ambient water quality improvements , 2002 .

[90]  Christian A. Vossler,et al.  A criterion validity test of the contingent valuation method: comparing hypothetical and actual voting behavior for a public referendum , 2003 .

[91]  D. Marsh,et al.  Testing the stability of welfare estimates in travel cost random utility models of recreation: An application to the Rotorua Lakes, New Zealand , 2015 .

[92]  Robert P. Berrens,et al.  Explaining Disparities between Actual and Hypothetical Stated Values: Further Investigation Using Meta-Analysis , 2004 .

[93]  Robert J. Johnston,et al.  Is hypothetical bias universal? Validating contingent valuation responses using a binding public referendum , 2006 .

[94]  Andrew H. Trice,et al.  Measurement of Recreation Benefits , 1958 .

[95]  Peter A. Diamond,et al.  Testing the Internal Consistency of Contingent Valuation Surveys , 1996 .

[96]  W. G. Brown,et al.  Impact of Aggregation on the Estimation of Outdoor Recreation Demand Functions , 1973 .

[97]  J. Hausman,et al.  Contingent Valuation: Is Some Number Better than No Number? , 1994 .

[98]  M. Teisl,et al.  Recall bias and reliability of survey data: moose hunting in Maine. , 1996 .

[99]  Christopher Timmins,et al.  A revealed preference approach to the measurement of congestion in travel cost models , 2007 .

[100]  Criterion and predictive validity of revealed and stated preference data: the case of “Mountain Home Music” concert demand , 2014 .

[101]  Craig E. Landry,et al.  Valuing Beach Quality with Hedonic Property Models , 2011, Land Economics.

[102]  Wiktor L. Adamowicz,et al.  Aggregation Bias in Recreation Site Choice Models: Resolving the Resolution Problem , 2004, Land Economics.

[103]  Brad Gentner,et al.  Addressing Onsite Sampling in Recreation Site Choice Models , 2011 .

[104]  Robert Cameron Mitchell,et al.  Using Surveys to Value Public Goods: The Contingent Valuation Method , 1989 .

[105]  R. G. Cummings,et al.  Induced-value tests of the referendum voting mechanism , 2001 .

[106]  T. Stevens,et al.  Sensitivity of Contingent Valuation to Alternative Payment Schedules , 1997 .

[107]  Richard T. Carson,et al.  Incentive and informational properties of preference questions , 2007 .

[108]  James J. Murphy,et al.  A Meta-analysis of Hypothetical Bias in Stated Preference Valuation , 2003 .

[109]  Jennifer Murdock,et al.  Handling unobserved site characteristics in random utility models of recreation demand , 2006 .

[110]  Richard T. Carson,et al.  Temporal Reliability of Estimates from Contingent Valuation , 1997 .

[111]  Sheldon Gen.  Meta-Analysis of Environmental Valuation Studies , 2004 .