Explaining Temporal Differences to Create Useful Concepts for Evaluating States

We describe a technique for improving problem-solving performance by creating concepts that allow problem states to be evaluated through an efficient recognition process. A temporal-difference (TD) method is used to bootstrap a collection of useful concepts by backing up evaluations from recognized states to their predecessors. This procedure is combined with explanation- based generalization (EBG) and goal regression to use knowledge of the problem domain to help generalize the new concept definitions. This maintains the efficiency of using the concepts and accelerates the learning process in comparison to knowledge-free approaches. Also, because the learned definitions may describe negative conditions, it becomes possible to use EBG to explain why some instance is not an example of a concept. The learning technique has been elaborated for minimax gameplaying and tested on a Tic-Tat-Toe system, T2. Given only concepts defining the end-game states and constrained to a two-ply search bound, experiments show that T2 learns concepts for achieving near-perfect play. T2's total searching time, including concept recognition, is within acceptable performance limits while perfect play without the concepts requires searches taking well over 100 times longer than T2's.