Solving Partially Observable Problems by Evolution and Learning of Finite State Machines

Finite state machines (FSM) have been successfully used to implement the control of an agent to solve particular sequential tasks. Nevertheless, finite state machines must be hand-coded by the engineer, which might be very difficult for complex tasks. Researchers have used evolutionary techniques to evolve finite state machines and find automatic solutions to sequential tasks. Their approach consists on encoding the state-transition table defining a finite state machine in the genome. However, the search space of such approach tends to be innecesarily huge. In this article, we propose an alternative approach for the automatic design of finite state machines using artificial evolution and learning techniques: the SOS-algorithm. We have obtained very impresive results on experimental work solving partially observable problems.

[1]  Milos Hauskrecht,et al.  Planning and control in stochastic domains with imperfect information , 1997 .

[2]  Tomás Lang,et al.  Introduction to Digital Systems , 1998 .

[3]  Kee-Eung Kim,et al.  Learning Finite-State Controllers for Partially Observable Environments , 1999, UAI.

[4]  A. Cassandra,et al.  Exact and approximate algorithms for partially observable markov decision processes , 1998 .

[5]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[6]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[7]  Andrés Pérez Uribe,et al.  Structure-Adaptable Digital Neural Networks , 1999 .

[8]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[9]  Thomas G. Dietterich.  The MAXQ Method for Hierarchical Reinforcement Learning , 1998, ICML.

[10]  Lawrence J. Fogel,et al.  Intelligence Through Simulated Evolution: Forty Years of Evolutionary Programming , 1999 .

[11]  Jürgen Schmidhuber,et al.  HQ-Learning , 1997, Adapt. Behav..

[12]  Doina Precup,et al.  Theoretical Results on Reinforcement Learning with Temporally Abstract Options , 1998, ECML.

[13]  Christian Jacob,et al.  Illustrating Evolutionary Computation with Mathematica , 2001 .

[14]  Michael I. Jordan,et al.  Reinforcement Learning with Soft State Aggregation , 1994, NIPS.

[15]  Tom M. Mitchell,et al.  Reinforcement learning with hidden states , 1993 .

[16]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[17]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .