Adaptive control of Markov chains, I: Finite parameter set

Consider a controlled Markov chain whose transition probabilities depend upon an unknown parameter α taking values in finite set A . To each α is associated a prespecified stationary control law \phi(\alpha) . The adaptive control law selects at each time t the control action indicated by \phi(\alpha_{t}) where α t is the maximum likelihood estimate of α. It is shown that α t converges to a parameter α*such that the "closed-loop" transition probabilities corresponding to α*and \phi(\alpha^{\ast}) are the same as those corresponding to α0and \phi(\alpha) where α0is the true parameter. The situation when α0does not belong to the model set A is briefly discussed.