KERNEL TEMPORAL DIFFERENCES FOR REINFORCEMENT LEARNING WITH APPLICATIONS TO BRAIN MACHINE INTERFACES

of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy KERNEL TEMPORAL DIFFERENCES FOR REINFORCEMENT LEARNING WITH APPLICATIONS TO BRAIN MACHINE INTERFACES By Jihye Bae August 2013 Chair: Jose C. Principe Major: Electrical and Computer Engineering Reinforcement learning brain machine interfaces (RLBMI) have been shown to be a promising avenue for practical implementations of BMIs. In the RLBMI, a computer agent and a user in the environment cooperate and learn co-adaptively. An essential component in the agent is the neural decoder which translates the neural states of the user into control actions for the external device in the environment. However, to realize the advantages of the RLBMI in practice, there are several challenges that need to be addressed. First, the neural decoder must be able to handle high dimensional neural states containing spatial-temporal information. Second, the mapping from neural states to actions must be flexible enough without making strong assumptions. Third, the computational complexity of the decoder should be reasonable such that real time implementations are feasible. Fourth, it should be robust in the presence of outliers or perturbations in the environment. We introduce algorithms that take into account these four issues. To efficiently handle the high dimensional state spaces, we adopt the temporal difference (TD) learning which allows the learning of the state value function using function approximation. For a flexible decoder, we propose the use of kernel base representations which provides nonlinear extensions of TD(λ) which we call kernel temporal difference (KTD)(λ). Two key advantages of KTD(λ) are its nonlinear functional approximation capabilities and convergence guarantees that gracefully emerge as