We show that all algorithms that search for an extremum of a cost function perform exactly the same, according to any performance measure , when averaged over all possible cost functions. In particular, if algorithm A outperforms algorithm B on some cost functions, then loosely speaking there must exist exactly as many other functions where B out-performs A. Starting from this we analyze a number of the other a priori characteristics of the search problem, like its geometry and its information-theoretic aspects. This analysis allows us to derive mathematical benchmarks for assessing a particular search algorithm's performance. We also investigate minimax aspects of the search problem, the validity of using characteristics of a partial search over a cost function to predict future behavior of the search algorithm on that cost function, and time-varying cost functions. We conclude with some discussion of the justiiability of biologically-inspired search methods.
[1]
D. R. Wolf,et al.
Alpha, Evidence, and the Entropic Prior
,
1993
.
[2]
L. Goddard,et al.
Operations Research (OR)
,
2007
.
[3]
D. Wolpert.
OFF-TRAINING SET ERROR AND A PRIORI DISTINCTIONS BETWEEN LEARNING ALGORITHMS
,
1994
.
[4]
G. Reinelt.
The traveling salesman: computational solutions for TSP applications
,
1994
.
[5]
Judea Pearl,et al.
Heuristics : intelligent search strategies for computer problem solving
,
1984
.
[6]
J. Berger.
Statistical Decision Theory and Bayesian Analysis
,
1988
.
[7]
Cullen Schaffer,et al.
Conservation of Generalization: A Case Study
,
1995
.
[8]
Thomas M. Cover,et al.
Elements of Information Theory
,
2005
.
[9]
John H. Holland,et al.
Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence
,
1992
.