Simultaneous Learning of Trees and Representations for Extreme Classification, with Application to Language Modeling

This paper addresses the problem of multi-class classification with an extremely large number of classes, where the class predictor is learned jointly with the data representation, as is the case in language modeling problems. The predictor admits a hierarchical structure, which allows for efficient handling of settings that deal with a very large number of labels. The predictive power of the model however can heavily depend on the structure of the tree. We address this problem with an algorithm for tree construction and training that is based on a new objective function which favors balanced and easily-separable node partitions. We describe theoretical properties of this objective function and show that it gives rise to a boosting algorithm for which we provide a bound on classification error, i.e. we show that if the objective is weakly optimized in the internal nodes of the tree, then our algorithm will amplify this weak advantage to build a tree achieving any desired level of accuracy. We apply the algorithm to the task of language modeling by re-framing conditional density estimation as a variant of the hierarchical classification problem. We empirically demonstrate on text data that the proposed approach leads to high-quality trees in terms of perplexity and computational running time compared to its non-hierarchical counterpart.