Assessing Effects of Unreplicated Perturbations: No Simple Solutions

We address the task of determining the effects, on mean population density or other parameters, of an unreplicated perturbation, such as arises in environmental assessments and some ecosystem-level experiments. Our context is the Before-After-Con- trol-Impact-Pairs design (BACIP): on several dates Before and After the perturbation, samples are collected simultaneously at both the Impact site and a nearby "Control." One approach is to test whether the mean of the Impact-Control difference has changed from Before to After the perturbation. If a conventional test is used, checks of its as- sumptions are an important and messy part of the analysis, since BACIP data do not necessarily satisfy them. It has been suggested that these checks are not needed for ran- domization tests, because they are insensitive to some of these assumptions and can be adjusted to allow for others. A major aim of this paper is to refute this suggestion: there is no panacea for the difficult and messy technical problems in the analysis of data from assessments or unreplicated experiments. We compare the randomization t test with the standard t test and the modified (Welch- Satterthwaite-Aspin) t test, which allows for unequal variances. We conclude that the randomization t test is less likely to yield valid inferences than is the Welch t test, because it requires identical distributions for small sample sizes and either equal variances or equal sample sizes for larger ones. The formal requirement of Normality is not crucial to the Welch t test. Both parametric and randomization tests require that time and location effects be additive and that Impact-Control differences on different dates be independent. These assumptions should be tested; if they are seriously wrong, alternative analyses are needed. This will often require a long time series of data. Finally, for assessing the importance of a perturbation, the P value of a hypothesis test is rarely as useful as an estimate of the size of the effect. Especially if effect size varies with time and conditions, flexible estimation methods with approximate answers are preferable to formally exact P values.

[1]  J. Durbin,et al.  Testing for serial correlation in least squares regression. I. , 1950, Biometrika.

[2]  G. Mittelbach.  Competition Among Refuging Sunfishes and Effects of Fish Density on Littoral Zone Invertebrates , 1988 .

[3]  M. Pagano,et al.  On Obtaining Permutation Distributions in Polynomial Time , 1983 .

[4]  Dilip Mathur,et al.  Assessment of Thermal Discharges on Zooplankton in Conowingo Pond, Pennsylvania , 1980 .

[5]  D. J. Gans.  Use of a preliminary test in comparing two sample means , 1981 .

[6]  Stephen R. Carpenter,et al.  Randomized Intervention Analysis and the Interpretation of Whole‐Ecosystem Experiments , 1989 .

[7]  Craig W. Osenberg,et al.  Effects of Body Size on the Predator‐Prey Interaction Between Pumpkinseed Sunfish and Gastropods , 1989 .

[8]  D. J. Hall,et al.  Growth Patterns in Bluegill (Lepomis macrochirus) and Pumpkinseed (L. gibbosus) Sunfish: Environmental Variation and the Importance of Ontogenetic Niche Shifts , 1988 .

[9]  A. Jassby,et al.  Detecting Changes in Ecological Time Series , 1990 .

[10]  Noel A Cressie,et al.  Playing safe with misweighted means , 1982 .

[11]  Moti L. Tiku,et al.  Robustness of MML estimators based on censored samples and robust test statistics , 1980 .

[12]  D. F. Andrews,et al.  Robust Estimates of Location , 1972 .

[13]  Stephen R. Carpenter,et al.  Cascading Trophic Interactions and Lake Productivity , 1985 .

[14]  Harry O. Posten,et al.  The robustness of the two—sample t—test over the Pearson system , 1978 .

[15]  D. Hinkley,et al.  The Analysis of Transformed Data , 1984 .

[16]  A. M. Gross.  Confidence Interval Robustness with Long-Tailed Symmetric Distributions , 1976 .

[17]  H. J. Whitford,et al.  How to Use the Two Sample t‐Test , 1986 .

[18]  Kenneth H. Reckhow,et al.  Bayesian inference in non-replicated ecological studies , 1990 .

[19]  G. Box,et al.  On a measure of lack of fit in time series models , 1978 .

[20]  K. Kafadar.  Using biweight m-estimates in the two-sample problem part 1: symmetric populations , 1982 .

[21]  David Ruppert,et al.  On prediction and the power transformation family , 1981 .

[22]  D. V. Lindley,et al.  An Introduction to Probability Theory and Its Applications. Volume II , 1967, The Mathematical Gazette.

[23]  G. Box.  Some Theorems on Quadratic Forms Applied in the Study of Analysis of Variance Problems, II. Effects of Inequality of Variance and of Correlation Between Errors in the Two-Way Classification , 1954 .

[24]  K. Yuen,et al.  The two-sample trimmed t for unequal population variances , 1974 .

[25]  M. Tiku,et al.  Robust test for means when population variances are unequal , 1981 .

[26]  D. McDowall,et al.  Interrupted Time Series Analysis , 1980 .

[27]  S. Stigler.  The Effect of Sample Heterogeneity on Linear Functions of Order Statistics, with Applications to Robust Estimation , 1976 .

[28]  Richard H. Jones,et al.  Maximum Likelihood Fitting of ARMA Models to Time Series With Missing Observations , 1980 .

[29]  George E. P. Box,et al.  Intervention Analysis with Applications to Economic and Environmental Problems , 1975 .

[30]  Gene V. Glass,et al.  Consequences of Failure to Meet Assumptions underlying the Analysis of Variance and Covariance. , 1972 .

[31]  Stephen R. Carpenter,et al.  Replication and Treatment Strength in Whole‐Lake Experiments , 1989 .

[32]  Allan Stewart-Oaten,et al.  ENVIRONMENTAL IMPACT ASSESSMENT: "PSEUDOREPLICATION" IN TIME?' , 1986 .

[33]  B. Efron.  Student's t-Test under Symmetry Conditions , 1969 .

[34]  D. F. Andrews,et al.  A note on the selection of data transformations , 1971 .

[35]  David Tritchler.  On Inverting Permutation Tests , 1984 .

[36]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[37]  G. Mittelbach,et al.  Trophic Relations and Ontogenetic Niche Shifts in Aquatic Ecosystems , 1988 .

[38]  Stephen R. Carpenter,et al.  Large‐Scale Perturbations: Opportunities for Innovation , 1990 .

[39]  Joseph P. Romano.  On the behaviour of randomization tests without the group invariance assumption , 1990 .

[40]  G. C. Tiao,et al.  Analysis of Los Angeles Photochemical Smog Data: A Statistical Overview , 1975 .

[41]  W. Hoeffding.  The Large-Sample Power of Tests Based on Permutations of Observations , 1952 .

[42]  S. Hurlbert.  Pseudoreplication and the Design of Ecological Field Experiments , 1984 .

[43]  W. J. Dixon,et al.  The approximate behaviour and performance of the two-sample trimmed t , 1973 .

[44]  G. C. Tiao,et al.  A CHANGE IN LEVEL OF A NON-STATIONARY TIME SERIES. , 1965 .

[45]  John W. Pratt,et al.  Bayesian Interpretation of Standard Inference Statements , 1965 .

[46]  B. Murphy.  Comparison of some two sample means tests by simulation , 1976 .

[47]  J. Berger,et al.  Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence , 1987 .