Adaptive confidence and adaptive curiosity

Much of the recent research on adaptive neuro-control and reinforcement learning focusses on systems with adaptivèworld models'. Previous approaches, however, do not address the problem of modelling the reliability of the world model's predictions in uncertain environments. Furthermore, with previous approaches usually some ad-hoc method (like random search) is used to train the world model to predict future environmental inputs from previous inputs and control outputs of the system. This paper introduces ways for modelling the reliability of the outputs of adaptive predictors, and it describes more sophisticated and sometimes more ecient methods for their adaptive construction by on-line state space exploration: For instance, a 4-network reinforcement learning system is described which tries to maximize the expectation of the temporal derivative of the adaptive assumed reliability of future predictions. The system is`curious' in the sense that it actively tries to provoke situations for which it learned to expect to learn something about the environment. An experiment with an articial non-deterministic environment demonstrates that the method can be faster than the conventional model-building strategy. Much of the recent research on adaptive neuro-control and reinforcement learning focusses on systems with sub-modules that learn to predict inputs from the environment. These sub-modules often are calledàdaptive world models'; they are useful for a whole variety of control tasks. For instance, Werbos' and Jordan's architectures for neuro-control [16][3] contain an adaptive world model in form of a back-propagation module (the model network) which is trained to predict the next input, given the current input and the current output of an adaptive control network. The model network allows to compute error gradients for the controller outputs. This is essential, since with typical adaptive neuro-control tasks there is no teacher who provides desired controller outputs. There is only a desired environmental input. Extensions of this approach (e.g. [11]) rely on the same basic principles. Sutton's `DYNA-systems' [13] use adaptive world models for limiting the number of`real-world experiences' necessary to solve certain reinforcement learning tasks. There are at least two important problems with all of these approaches that have not been addressed so far: 1. Previous model-building control systems are not well-suited for uncertain non-deterministic environments. In particular, they do not model the reliability of the predictions of the adaptive world models. Therefore, if credit assignment for the controller is based on the assumption of a correct world model, unexpected results may be obtained. 2. Previous model-building control …