Maximizing classifier yield for a given accuracy

We propose a novel and intuitive way to quantify the utility of a classifier in cases where automatic classification is deployed as partial replacement of human effort, but accuracy requirements exceed the capabilities of the classifier at hand. in our approach, a binary classifier is combined with a meta-classifier mapping all decisions of the first classifier that do not meet a pre-specified confidence level to a third category: for manual inspection. this ternary classifier can now be evaluated in terms of its yield, where yield is defined as the proportion of observations that can be classified automatically with a pre-specified minimum accuracy.