Evolution of Reinforcement Learning in Uncertain Environments

Acknowledgements I am grateful to Daphna Joel and Eytan Ruppin, my supervisors, for their invaluable guidance and support. To Daphy for boldly venturing into the field of neural network modelling and contributing her clear thoughts, sharp distinctions and original ideas to this work. To Eytan for his wisdom, his endless enthusiasm and his sincere appreciation of the significance of my work. Many thanks to Prof. Isaac Meilijson for the mathematical proof regarding the emergence of risk-aversion. I thank Dr. Tamar Keasar for introducing me to the BeeHave lab at HUJI, for providing me with the probability matching data, and for her invaluable comments and new ideas throughout my research. I have benefitted from discussions with the many people who have read drafts of this work or listened to my talks. All these have helped me immensly in making my ideas coherent and understandable. Special thanks to my family, and especially my father, Yehuda, who followed my research closely, nagged until I finally sat down to write this, and helped me tackle the difficult parts. Abstract Reinforcement learning is a fundamental process by which organisms learn to achieve a goal from interactions with the environment. Using Artificial Life techniques we evolve (near-)optimal neuronal learning rules in a simple neural network model of reinforcement learning in bumblebees foraging for nectar. The resulting neural networks exhibit efficient reinforcement learning, allowing the bees to respond rapidly to changes in reward contingencies. The evolved synaptic plasticity dynamics give rise to varying exploration/exploitation levels from which emerge the well-documented choice strategies of risk aversion and probability matching. These strategies are shown to be a direct result of reinforcement learning, providing a biologically founded, parsimonious and novel explanation for these behaviors. Our results are corroborated by a rigorous mathematical analysis and their robustness in real-world situations is supported by experiments in a mobile robot.