Adaptive artificial datasets through learning classifier systems for classification tasks

In producing an artificial dataset, humans usually play a major role in creating and controlling the problem domain. In particular, humans set up and tune the problem’s difficulty. If humans can set up the difficulty levels appropriately, then learning systems can solve classification tasks successfully. This paper introduces an autonomous classification problem generation approach. The problem’s difficulty is adapted based on the classification agent’s performance within the defined attributes. An automated problem generator has been created to evolve simulated datasets whilst the classification agent, in this case a learning classifier system (LCS), attempts to learn the evolving datasets. The idea here is to tune the problem’s difficulty autonomously such that the problem’s characteristics may be determined effectively. Furthermore, this framework can empirically test the learning bounds of the classification agent whilst lowering human involvement. Initially, tabu search was integrated in the problem generator to discover the best combination of domain features in order to adjust the problem’s difficulty. In order to overcome stagnation in local optimum, a Pittsburgh-style LCSs, A-PLUS, was adapted for the first time to the problem generator. In this way, the effect of the problem’s characteristics, e.g. noise, which alter the classification agent’s performance, becomes human readable. Experiments confirm that the problem generator was able to tune the problem’s difficulty either to make the problem ‘harder’ or ‘easier’ so that it can either ‘increase’ or ‘decrease’ the classification agent’s performance.