Surveyor's Forum: Experimental Analysis Discretely Done: Author's Reply

Bruce Weide's survey of contemporary research in the analysis of discrete algorithms illustrates numerous useful analytic techniques. [See Comput. Surv. 9, 4 (Dec. 1977), pp. 291-313]. However, the paper does not develop the theme that experimental techniques are potential useful complements to analysis. I think it is useful to cite several examples. There seem to be two broad classes of experimental techniques for analyzing combinatorial algorithms: the "brute force" techniques, which use exhaustive enumeration of cases, and the statistical techniques, which sample the algorithm's behavior under various conditions. Perhaps the most notable example of case enumeration is the use of a program to prove the Four Color Map Theorem [1] by enumerating all the maps that previous analysis had revealed to be in need of testing. A second example of case enumeration arose in the analysis of the so-caUed Alpha-Beta algorithm, which is often used in game-playing programs. The objective was a formula for the average number of nodes expanded by the algorithm in searching uniform trees of depth D and branching factor AT. Fuller, Gaschnig, and Gillogly wrote a paper analyzing this algorithm [2]. Knuth and Moore sent Fuller et al. a proposed formula; Fuller et al. wrote a program to enumerate 8! cases and discovered an error in the proposed formula in one of the cases. Knuth and Moore subsequently derived a correct formula, which was reported with this story in [3]. There are two examples showing that experimental techniques helped find a proof that eluded available analytic techniques. Weide cites the same Alpha-Beta papers [2, 3] to illustrate that Monte Carlo simulations may lead to incorrect conclusions. Fuller, et al. conjectured on the basis of such experiments that the formula O(N °'72D) approximated the average number of nodes expanded [2]. Knuth and Moore subsequently proved (correctly) that an actual upper bound is O((N / log N)D). The latter bound grows more slowly than the former.