TD Learning of Game Evaluation Functions with Hierarchies Neural Architectures

This Master s thesis describes the e ciency of temporal di erence TD learning and the advantages of using modular neural network architectures for learning game evaluation functions These modular architectures use a hierarchy of gating networks to divide the input space in subspaces for which expert networks are trained This divide and conquer principle might be advantageous when learning game evaluation functions which contain discontinuities and can also lead to more understandable solutions in which strategies can be identi ed and explored We compare the following three modular architectures the hierarchical mixtures of experts the Meta Pi network and the use of xed symbolic rules In order to generate learning samples we combine reinforcement learning with the temporal di erence method When training neural networks with these examples it is possible to learn to play any desired game An extension of normal back propagation has been used in which the sensitivities of neurons are adapted by a learning rule We discuss how these neuron sensitivities can be used to learn discontinuous and smooth game evaluation functions Experiments with the games of tic tac toe and the endgame of backgammon have been performed to compare the hierarchical architectures with a single network and to validate the e ciency of TD learning The results with tic tac toe show that modular architec tures learn faster because independent expert networks can be invoked for evaluating a particular position without the need of invoking one large neural network every time Fur thermore the use of high neuron sensitivities has been proven to be useful when learning discontinuous functions The results with the endgame of backgammon show that TD learning is a viable alternative for supervised learning when only a small learning set is available For both games the performance of the architectures is improved when more games are being played High performance levels can be obtained when a large amount of games are played and the input of the networks contains su cient features keywords Game Playing Modular Neural Networks Extended Back propagation Temporal Di erence Learning Expert Systems Multi strategy Learning Mixtures of Experts Meta Pi network Tic tac toe Backgammon