Validation and Verification

(1)We will regard MSE as essentially analogous to RMSE, and concentrate on comparisonsbetween RMSE and MAE. While the choice between these two metrics may seeminconsequential, there are cases where they would lead to conflicting interpretations duringvalidation, especially for pattern classification networks. The following example is from(Twomey and Smith 1993a, 1995). The pattern classification problem is a well known two classclassification problem depicted in Figure 1. Two Gaussian distributions are classes A and B,where class A has a mean of 0 and a standard deviation of 1 and class B has a mean of 0 and astandard deviation of 2. There is considerable overlap between the classes, making this is astraightforward “hard” pattern classification task. Both training and testing sets contained equalnumbers of observations from each class.Figure 1 here.Typically, networks are trained to decreasingly lower training tolerances in an attempt toachieve the “best” performing network. This assumes that lower training tolerances equate withimproved performance, which is certainly not always the case. To examine the relativedifferences between RMSE and MAE, several networks were trained and tested in order tomeasure the effects of lowering the training tolerance for the two pattern classification problemwhere the correct output for class A is 0 and the correct output for class B is 1.A total of 13 different networks were trained, each to a different training tolerance.Training tolerances ranged from 0.08 to 0.6 for normalized input between 0 and 1. Duringtraining, a response was considered correct if the output was within a pre-specified absolutetraining tolerance of the target value (0 or 1). Training iterations continued for each networkMSEn