Bias-Variance Trade-offs: Novel Applications

Consider a given random variable F and a random variable that we can modify, F̂ . We wish to use a sample of F̂ as an estimate of a sample of F . The mean squared error between such a pair of samples is a sum of four terms. The first term reflects the statistical coupling between F and F̂ and is conventionally ignored in bias-variance analysis. The second term reflects the inherent noise in F and is independent of the estimator F̂ . Accordingly, we cannot affect this term. In contrast, the third and fourth terms depend on F̂ . The third term, called the bias, is independent of the precise samples of both F and F̂ , and reflects the difference between the means of F and F̂ . The fourth term, called the variance, is independent of the precise sample of F , and reflects the inherent noise in the estimator as one samples it. These last two terms can be modified by changing the choice of the estimator. In particular, on small sample sets, we can often decrease our mean squared error by, for instance, introducing a small bias that causes a large reduction the variance. While most commonly used in machine learning, this article shows that such bias-variance trade-offs are applicable in a much broader context and in a variety of situations. We also show, using experiments, how existing bias-variance trade-offs can be applied in novel circumstances to improve the performance of a class of optimization algorithms.