Reinforcement Learning with Approximation Spaces

This paper introduces a rough set approach to reinforcement learning by swarms of cooperating agents. The problem considered in this paper is how to guide reinforcement learning based on knowledge of acceptable behavior patterns. This is made possible by considering behavior patterns of swarms in the context of approximation spaces. Rough set theory introduced by Zdzislaw Pawlak in the early 1980s provides a ground for deriving pattern-based rewards within approximation spaces. Both conventional and approximation space-based forms of reinforcement comparison and the actor-critic method as well as two forms of the off-policy Monte Carlo learning control method are investigated in this article. The study of swarm behavior by collections of biologically-inspired bots is carried out in the context of an artificial ecosystem testbed. This ecosystem has an ethological basis that makes it possible to observe and explain the behavior of biological organisms that carries over into the study of reinforcement learning by interacting robotic devices. The results of ecosystem experiments with six forms of reinforcement learning are given. The contribution of this article is the presentation of several viable alternatives to conventional reinforcement learning methods defined in the context of approximation spaces.

[1]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[2]  Andrzej Skowron,et al.  Towards an Ontology of Approximate Reason , 2002, Fundam. Informaticae.

[3]  James F. Peters,et al.  Approximation Spaces for Hierarchical Intelligent Behavioral System Models , 2004, MSRAS.

[4]  Dirk P. Kroese,et al.  Simulation and the Monte Carlo method , 1981, Wiley series in probability and mathematical statistics.

[5]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[6]  Andrzej Skowron,et al.  Approximation Spaces and Information Granulation , 2004, Trans. Rough Sets.

[7]  Chris Gaskett,et al.  Q-Learning for Robot Control , 2002 .

[8]  Z. Pawlak Rough Sets: Theoretical Aspects of Reasoning about Data , 1991 .

[9]  Andrzej Skowron,et al.  Layered Learning for Concept Synthesis , 2004, Trans. Rough Sets.

[10]  Andrzej Skowron,et al.  Rough mereology: A new paradigm for approximate reasoning , 1996, Int. J. Approx. Reason..

[11]  James F. Peters,et al.  Line-crawling robot navigation: a rough neurocomputing approach , 2003 .

[12]  Andrzej Skowron,et al.  Calculi of Approximation Spaces , 2006, Fundam. Informaticae.

[13]  Christian P. Robert,et al.  Monte Carlo Statistical Methods (Springer Texts in Statistics) , 2005 .

[14]  Philip N. Lehner,et al.  Handbook of ethological methods , 1979 .

[15]  Andrzej Skowron,et al.  Rough Sets and Vague Concepts , 2004, Fundam. Informaticae.

[16]  James F. Peters,et al.  Approximation space for intelligent system design patterns , 2004, Eng. Appl. Artif. Intell..

[17]  James F. Peters,et al.  Reinforcement Learning with Pattern-based Rewards , 2005, Computational Intelligence.

[18]  Z. Pawlak Classification of objects by means of attributes , 1981 .

[19]  Andrzej Skowron,et al.  Rough Sets and Information Granulation , 2003, IFSA.

[20]  James F. Peters,et al.  Rough Ethograms: Study of Intelligent System Behavior , 2005, Intelligent Information Systems.

[21]  Jerzy W. Grzymala-Busse,et al.  Rough Sets , 1995, Commun. ACM.

[22]  N. Metropolis,et al.  The Monte Carlo method. , 1949 .

[23]  J. Stepaniuk Approximation Spaces, Reducts and Representatives , 1998 .

[24]  Anna Gomolinska,et al.  Rough Validity, Confidence, and Coverage of Rules in Approximation Spaces , 2005, Trans. Rough Sets.

[25]  Ryszard Engelking,et al.  Outline of general topology , 1968 .

[26]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[27]  J. Hammersley,et al.  Monte Carlo Methods , 1965 .

[28]  Andrew McCallum,et al.  Reinforcement learning with selective perception and hidden state , 1996 .

[29]  Marco Dorigo,et al.  Swarm intelligence: from natural to artificial systems , 1999 .

[30]  Francesco Mondada,et al.  Collective and Cooperative Group Behaviors: Biologically Inspired Experiments in Robotics , 1995, ISER.

[31]  Andrzej Skowron,et al.  Modelling Complex Patterns by Information Systems , 2005, Fundam. Informaticae.

[32]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[33]  Andrzej Skowron,et al.  Rough Set Methods in Approximation of Hierarchical Concepts , 2004, Rough Sets and Current Trends in Computing.

[34]  Reuven Y. Rubinstein,et al.  Simulation and the Monte Carlo Method , 1981 .

[35]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[36]  Doina Precup,et al.  Eligibility Traces for Off-Policy Policy Evaluation , 2000, ICML.

[37]  W. Reiher Hammersley, J. M., D. C. Handscomb: Monte Carlo Methods. Methuen & Co., London, and John Wiley & Sons, New York, 1964. VII + 178 S., Preis: 25 s , 1966 .

[38]  James F. Peters,et al.  Measuring Acceptance of Intelligent System Models , 2004, KES.

[39]  James F. Peters,et al.  Monte Carlo off-policy reinforcement learning: a rough set approach , 2005, Fifth International Conference on Hybrid Intelligent Systems (HIS'05).