Empirical and statistical p values and Type 1 error rates: Putting it all together

The original concepts of p values and Type 1 errors (α) were first developed in the statistical literature nearly a hundred years back. Although still widely used, their application has broadened beyond the original statistical foundation. For example, most classical statistics is based around distributions, often the normal distribution. The concepts of training and test sets were introduced later, primarily in the area of machine learning, and the early statistical literature did not discuss these concepts explicitly. Datasets were often relatively small compared to what is possible nowadays. Many of the more elaborate approaches were introduced in a very mathematical manner, with little or no experimental data available to test out or use the theory. The widespread availability of large empirical datasets with quite complex structures is a relatively recent phenomenon. So it is useful to re-evaluate these historic statistical ideas in a more contemporary framework.