The Principled Prediction-Problem Ontology: when black box algorithms are (not) appropriate

Black-box algorithms have had astonishing success in some settings. But their unpredictable brittleness has provoked serious concern and increased scrutiny. For any given black-box algorithm understanding where it might fail is extraordinarily challenging. In contrast, understanding which settings are not appropriate for black-box deployment requires no more than understanding simply how they are developed. We introduce a framework that isolates four problem-features -- measurement, adaptability, resilience, and agnosis -- which need to be carefully considered before selecting an algorithm. This paper lays out a principled framework, justified through careful decomposition of the system components used to develop black-box algorithms, for people to understand and discuss where black-box algorithms are appropriate and, more frequently, where they are not appropriate.