Cost-sensitive learning of hierarchical tree classifiers for large-scale image classification and novel category detection

In this paper, a cost-sensitive learning algorithm is developed to train hierarchical tree classifiers for large-scale image classification application (i.e., categorizing large-scale images into thousands of object classes). A visual tree is first constructed for organizing large numbers of object classes hierarchically and identifying inter-related learning tasks automatically. For the fine-grained object classes at the sibling leaf nodes, they share significant common visual properties but still contain subtle visual differences, thus a multi-task structural learning algorithm is developed to train their inter-related classifiers jointly to enhance their discrimination power. For the coarse-grained categories (i.e., groups of visually similar object classes) at the sibling non-leaf nodes, a hierarchical learning algorithm is developed to leverage tree structure (by adding two inter-level constraints) to train their inter-related classifiers jointly and control inter-level error propagation effectively. To achieve more robust detection of large numbers of object classes, a visual forest is learned by combining multiple visual trees (for different configurations) and their hierarchical tree classifiers. By penalizing various types of misclassification errors differently, a cost-sensitive learning approach is further developed to detect the appearances of new object classes accurately, and an incremental learning algorithm is developed to achieve more effective training of the discriminative classifiers for new object classes. Our experimental results have demonstrated that our cost-sensitive hierarchical learning algorithm can achieve very competitive results on both classification accuracy and computational efficiency as compared with other state-of-the-art techniques. HighlightsVisual tree to organize large-scale object classes hierarchically and determine inter-related learning tasks automatically.Multi-task structural learning for joint classifier training to enhance their discrimination power significantly.Hierarchical learning to leverage inter-level constraints for classifier training and limiting inter-level error propagation.Task and tree parallelism to scale up our hierarchical learning algorithm for large-scale image classification.Cost-sensitive learning and incremental learning for training and detecting for new object classes more effectively.