Expert classifiers and the ordered veracity-experience response (OVER) curve

The training of good generalizations must mitigate both memorization and arrogance. Memorization is characterized as being too timid in associating new observations with previous experience. Contrarily, arrogance is being too bold. In classification problems, memorization is traditionally assessed via error matrices and iterative error-based techniques such as cross validation. These techniques, however, do nothing to assess arrogance in classification. To identify arrogant classifications, we propose a confusion-based figure of merit which we shall call the ordered veracity-experience response curve, or OVER curve. To produce the OVER curve, one must employ expert classifiers. An expert is a special classifier - a relational computation with not only a mechanism for decision making but also a quantifiable skill level. In this paper, we define the elements of both the expert classifier and OVER curve and, then, demonstrate their utility using the multilayer perceptron.