Choosing a Good Toolkit, II: Simulations and Conclusions

This paper continues our study of heuristics employed to choose dynamically tools to put in a toolkit, where the value of any tool can be discovered only by choosing it. This is a multi-armed bandit problem with “arms” that are not independent, hence it is a problem for which the optimal solution is virtually impossible to find. We study the relative performance of a number of heuristic decision rules for discount factors from 0.7 to 0.99 via simulation in the context of a small number of test problems, learning the strengths and weaknesses of the various heuristics.