Following a suggestion from Greenwald, Gonzalez, Harris, and Guthrie (1996), Posavac (2002) developed an approach to estimate the probability that replications of completed studies would be statistically significant. We argue that this approach is faulty because it is based on the unjustifiably restrictive assumption that the effect size observed in the initial study is the population effect size. Using the noncentral t distribution and methods presented by Cumming and Finch (2001), we construct a confidence interval for the observed effect size. Boundaries of that confidence interval are used to compute the limits of plausible replicability (post hoc power) values. It is demonstrated that the possible values span an impractically large range, rendering the post hoc analysis of replicability without merit. These issues are discussed in the broader context of post hoc power analyses.
[1]
A. Greenwald,et al.
Effect sizes and p values: what should be reported and what should be replicated?
,
1996,
Psychophysiology.
[2]
G. Cumming,et al.
A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions
,
2001
.
[3]
D. Heisey,et al.
The Abuse of Power
,
2001
.
[4]
Response to Macdonald
,
2003
.
[5]
Emil J. Posavac,et al.
Using p values to estimate the probability of a statistically significant replication.
,
2002
.
[6]
Ranald R. Macdonald.
On Determining Replication Probabilities: Comments on Posavac (2002)
,
2003
.