Simulation in optimization and optimization in simulation: a markov chain perspective on adaptive Monte Carlo algorithms

This paper discusses some connections between adaptive Monte Carlo algorithms and general state space Markov chains. Adaptive algorithms are iterative methods in which previously generated samples are used to construct a more efficient sampling distribution at the current iteration. In this paper, we describe two such adaptive algorithms, one arising in a finite-horizon computation of expected reward and the other arising in the context of solving eigenvalue problems. We then discuss the connection between these adaptive algorithms and general state space Markov chain theory, and offer some insights into some of the technical difficulties that arise in trying to apply the known theory for general state space chains to such adaptive algorithms.