On deployable adversarial classification models

The application of machine learning methods to malware detection has opened up possibilities of generating large number of classification models that use different kind of features, learning algorithms, and proportion of training data. Performance on test data can be used as a model selection criteria for determining the best classification model from the large space of all classification models. The effectiveness of test data performance as a model selection criteria will depend on a performance target that can act as a stopping criteria for the generate and test methodology of classifier design. The methods of determining the performance target are mostly ad hoc, either based on experience or some heuristic. We propose a rigorous method of determining a performance target-the deployable performance-based on the bound on the expected consequential costs of misclassification where the expectation is over performance in addition to the decision outcome. We also give a method to determine deployable classification models-models whose estimation of generalization performance meets the deployable performance target.