Rating reliability and representation validity in scenic landscape assessments

The US Supreme Court recently determined that experts from all fields of knowledge must demonstrate the reliability and validity of their testimony. While the broader implications of their finding have yet to manifest itself, it clearly has the potential to challenge all manner of professional practices. This paper explores the reliability of visual quality ratings of landscapes, and the validity of photographic representations used when making these ratings. A review of the literature finds that relatively few studies report reliability or validity coefficients. Those including such reports give reason for some concern. Data from several studies are re-analyzed to demonstrate how professionals should evaluate the reliability and validity of their visual landscape assessments.

[1]  Paul H. Gobster,et al.  The dimensions of aesthetic preference: a quantitative analysis. , 1989 .

[2]  Nickolaus R. Feimer,et al.  The prediction of scenic beauty from landscape content and composition , 1984 .

[3]  W. S. Robinson.  Ecological correlations and the behavior of individuals. , 1950, International journal of epidemiology.

[4]  Thomas R. Herzog,et al.  Tranquility and preference as affective qualities of natural environments , 1992 .

[5]  Seppo Kellomäki,et al.  The scenic value of the forest landscape as assessed in the field and the laboratory , 1984 .

[6]  P. Kane.  Assessing landscape attractiveness: a comparative test of two new methods , 1981 .

[7]  Forest inventory and management-based visual preference models of southern pine stands , 1988 .

[8]  Allan P. Jones,et al.  Apples and Oranges: An Empirical Comparison of Commonly Used Indices of Interrater Agreement , 1983 .

[9]  Peter Clamp.  A study in the evaluation of landscape and the impact of roads , 1975 .

[10]  D. Levine.  True Scores, Error, Reliability, and Unit of Analysis in Environment and Behavior Research , 1994 .

[11]  R. Gimblett,et al.  Comparing live experience with pictures in articulating landscape preference , 1992 .

[12]  Mary W. Downton,et al.  Judgments of photographs vs. field observations in studies of perception and judgment of the visual environment , 1984 .

[13]  R. B. Hull,et al.  Individual and group reliability of landscape assessments , 1984 .

[14]  Southeastern Forest Experiment Station.  General technical report , 1985 .

[15]  R. B. Hull,et al.  Validity of photo-based scenic beauty judgments , 1992 .

[16]  T. R. Herzog,et al.  A cognitive analysis of preference for urban nature , 1989 .

[17]  T. Daniel,et al.  Assessing visibility impairment in class I parks and wilderness areas: A comparison of policy‐relevant methods , 1988 .

[18]  Stephen J Sheppard,et al.  Visual Simulation: A Users Guide for Architects, Engineers, and Planners , 1989 .

[19]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .

[20]  K. Matsushita,et al.  National forest management. , 2002 .

[21]  R. O. Brush.  The attractiveness of woodlands: perceptions of forest landowners in Massachusetts. , 1979 .

[22]  D. Gottfredson,et al.  Units of Analysis and the Psychometrics of Environmental Assessment Scales , 1991 .

[23]  J. Palmer.  Stability of landscape perceptions in the face of landscape change , 1997 .

[24]  Stephen R.J. Sheppard,et al.  Predictive Landscape Portrayals: A Selective Research Review , 1982, Landscape Journal.

[25]  Thomas C. Brown,et al.  Recreation participation and the validity of photo-based preference judgments. , 1989 .

[26]  D. Levine.  Why Choose One Level of Analysis? and Other Issues in Multilevel Research , 1996 .

[28]  T. R. Herzog,et al.  A cognitive analysis of preference for waterscapes , 1985 .

[29]  T. Daniel,et al.  Measuring landscape esthetics: the scenic beauty estimation method , 1976 .

[30]  David Pitt,et al.  Perceptions and Measurements of Scenic Resources in the Southern Connecticut River Valley , 1974 .

[31]  G. Revell,et al.  Issues in sampling landscapes for visual quality assessments , 1989 .

[32]  Thomas R. Herzog,et al.  A Cognitive Analysis of Preference for Natural Environments: Mountains, Canyons, and Deserts , 1987, Landscape Journal.

[33]  Robert E. Coughlin,et al.  The extent of agreement among observers on environmental attractiveness , 1970 .

[34]  Joel B. Greenhouse,et al.  Selection Models and the File Drawer Problem , 1988 .

[35]  Scenic beauty perceptions along the ROS , 1994 .

[36]  M. Deutsch,et al.  Intractable Conflicts and Their Transformation , 1989 .

[37]  L. M. Anderson,et al.  Mapping the scenic beauty of forest landscapes. , 1977 .

[38]  Elwood L. Shafer,et al.  A comparison of viewer reactions to outdoor scenes and photographs of those scenes , 1974 .

[39]  H. W. Schroeder,et al.  Estimating park tree densities to maximize landscape esthetics , 1986 .

[40]  R. L. Ebel,et al.  Estimation of the reliability of ratings , 1951 .

[41]  J. M. Richards.  Units of Analysis and the Individual Differences Fallacy in Environmental Assessment , 1990 .

[42]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[43]  E Marshall.  Supreme Court to weigh science , 1993, Science.

[44]  Robert L. Ebel,et al.  Must all tests be valid , 1961 .

[45]  T. Brown,et al.  Context effects in perceived environmental quality assessment: Scene selection and landscape quality ratings , 1987 .

[46]  R. Rosenthal.  Meta-analytic procedures for social research , 1984 .

[47]  J. Richards.  Units of Analysis, Measurement Theory, and Environmental Assessment , 1996 .

[48]  Bo Shelby,et al.  Comparing methods for determining visitor evaluations of ecological impacts: site visits, photographs, and written descriptions , 1985 .

[49]  James F. Palmer,et al.  Reliability of Rating Visible Landscape Qualities , 2000, Landscape Journal.

[50]  Perceived scenic beauty and contingent valuation of forest campgrounds. , 1989 .

[51]  Nickolaus R. Feimer,et al.  Appraising the reliability of visual impact assessment methods , 1979 .

[52]  Thomas C. Brown,et al.  Is motion more important than it sounds?: The medium of presentation in environment perception research , 1993 .

[53]  Scott Danford,et al.  Subjective Responses To Architectural Displays , 1975 .

[54]  A. Stamps.  Use of Photographs to Simulate Environments: A Meta-Analysis , 1990 .