Action Selection and Action Control for Playing Table Soccer Using Markov Decision Processes Master Thesis

StarKick is a commercially available table soccer robot which challenges even advanced human players. However, the available set of actions for StarKick is limited and the way of selecting the actions is not flexible enough for incorporating more elaborate actions. In the context of this thesis, new actions for stopping and dribbling the ball are developed. Stopping is achieved by locking the ball between the playing surface and a playing figure. Dribbling makes the ball rolling at a controllable speed within the reachable area of the playing figures of one rod. By these new actions, the ball can be deliberately passed and stopped. To decide, which action should be taken in a given situation, an action selection scheme using Markov Decision Processes (MDPs) and reinforcement learning is developed. In order to reduce the state space, the basic actions are combined to more complex actions and the MDP is structured into four modules. Each module contains a set of states, and the actions that are applicable in these states. A simple reinforcement learning algorithm is implemented in the MDP framework. The transition probabilities are updated by counting. These updates are spread by policy iteration algorithm during a game. A series of experiments are carried out in real table soccer games. These experiments show that the newly developed actions are robust, the MDP model works fine, and the reinforcement learning improves the performance of StarKick in a simplified game. The attempt of making the reinforcement learning work in the whole game seems too tedious to be finished in real games.