Learning Robot Control - Using Control Policies as Abstract Actions

Autonomous robot systems operating in an uncertain environment have to be able to cope with new situations and task requirements. Important properties of the control architecture of such systems are thus that it is reactive, allows for exible responses to novel situations, and that it adapts to longer lasting changes in the environment or the task requirements. In the extreme case, this learning has to occur without the direct innuence of an outside teacher, making the reinforcement learning paradigm an attractive option since it allows to learn sequences of behavior from simple reinforcement signals 1, 17]. However, while these techniques have been applied to simple robot systems and in simulationn2, 5, 7, 10, 11, 12, 6], the complexity of the primitive action and state spaces of most robots leads to a need for large amounts of experiences to learn a given task, thus rendering these methods impracticable for on-line learning on such systems. Furthermore, most such learning systems do not provide a means for introducing a priori knowledge, thus permitting the occurrence of catastrophic failures which is often not permissible in real world systems which have to learn new tasks in a single trial. To address these issues, the control architecture presented here uses more abstract actions which allow to deene the system as a Discrete Event Dynamic System (DEDS) on an abstract, discrete state space, within which a policy for the given task is learned. To illustrate this, the architecture has been applied to walking tasks on a four-legged walking robot. The use of abstract actions within the reinforcement learning frameworkk14] promises to make it possible to address more complex tasks and platforms. Much of this promise stems from the possibility to treat the resulting system as an event driven system rather than a clock driven one, reducing the set of points at which the learning agent has to consider a new action to the times when certain control or sensor events happen. While this allows for optimal decision points to be missed if the corresponding sensor signals lie outside the scope of the current set of control and sensor alternatives, it also leads to a focus of attention and can dramatically