Reduced Training Time for Reinforcement Learning with Hidden State

Real robots with real sensors are not omniscient. When a robot's next course of action depends on information that is hidden from the sensors because of problems such as occlusion, restricted range, bounded eld of view and limited attention, we say the robot suuers from the hidden state problem. State identiication techniques use history information to uncover hidden state. Previous approaches to encoding history include: nite state machines Chrisman, 1992; McCallum, 1992a], recurrent neural networks Lin and Mitchell, 1992] and genetic programming with indexed memory Teller, 1994]. A chief disadvantage of all these techniques is their long training time. This paper presents instance-based state identiication, a new approach to hidden state reinforcement learning that learns with much fewer training steps. Noting that gathering world experience is often relatively more expensive than computation and storage, the approach applies instance-based (or \memory-based") learning to history sequences. The rst implementation of this approach, called Nearest Sequence Memory learns with an order of magnitude fewer steps than several previous approaches. A reinforcement learning agent suuers from the hidden state problem if at any time the agent's state representation is missing information needed to determine the next correct action, that is, if the agent's state representation is non-Markovian with respect to actions and utility. The hidden state problem arises as a case of perceptual aliasing: the mapping between states of the world and sensations of the agent is not one-to-one White-head and Ballard, 1991]. If the agent's perceptual system produces the same outputs for two world states in which diierent actions are required, and if the agent's state representation consists only of its percepts, then the agent will fail to choose correct actions. There are many reasons that important features could be hidden from a robot's perception: sensors have noise, limited range and limited eld of view; occlu-sions hide areas from sensing; limited funds and space prevent equipping the robot with all desired sensors; an exhaustible power supply deters the robot from using all sensors all the time; and the robot has limited computational resources for turning raw sensor data into usable percepts. 1.1 Stateless Agents One solution to the hidden state problem is simply to avoid passing through the perceptually aliased states. This is the approach taken in Whitehead's Lion algorithm Whitehead, 1992]. Whenever the agent nds a state that delivers inconsistent reward, it sets that state's utility so low that the policy will …