Case simulation to assess learning systems

Abstract Learning forms an essential part of artificial intelligence applications. Databases of example “cases” are essential to the development and assessment of learning systems. Insufficient examples make it difficult or impossible to compare variations of the learning method. Sufficient examples are rarely available to assess a learning system thoroughly. Simulation represents a means of producing data based upon a defined system. In this paper, a simulation for assessing the characteristics of learning systems is described. The simulation aims to generate data as an actual knowledge-based system (KBS) observes and stores data. A notional model is developed to mirror what is known of part of the actual target domain of a particular KBS. Significant results materializing from simulated data include a quantitative comparison of learning and testing on the same and disjoint data sets. Simulated data is used to show that the use of the same data for learning and testing frequently reduces diagnostic accuracy when learnt knowledge is applied to new data.