Stacking Diverse Models to Achieve Reliable Error Response Distributions

Artificial Neural Networks (ANNs) can be useful for modeling real- world processes such as time series weather, financial, or chaotic data. The generalization and robustness of these models can be improved, and estimates of the modeling error distributions can be made, using a technique called Stacked Generalization (SG). SG uses a number of diverse models, each of which is trained and queried on independent cross validation subsets of the process data. The models are then combined in the stacking process to provide error estimates and improved accuracy. These improvements depend on the individual model response diversity between networks. Modified Series Association (MSA), an extension to SG, presents the various models with different input subspaces from the raw data as a catalyst for increased diversity. Model diversity is formulated, and an alternative model combination approach is derived from it, called Diversified Committee Machines (DCM). A framework for quantifying error estimation reliability is presented and discussed. Using this framework, the predictive accuracy of SG and DCM are compared in terms of both the modeled target function and the model's confidence interval about it. This is achieved through a new measure called the confidence coefficient. A benchmark problem is also introduced as a generic data set for future comparison between inductive learning machines.