A Contrastive Learning Approach for Compositional Zero-Shot Learning

An object can be in several states. For different states (attributes) the object could look dramatically different. Thus, the smart information retrieval systems of the future need to learn good state-object representations. Such a system should not only be able to recognize state-object compositions unseen during training but also be able to retrieve images based on multi-modal (image-text) query. In the literature, these tasks are treated separately. In this work, we propose a unified model, ContraNet, which leverages the rich semantics of the state-object to learn multimodal representation in a contrastive manner. We adopt a deep metric learning approach and learn a multimodal representation by pulling similar images and texts closer to each other and pushing apart different ones. Our autoencoder based model learns the text-aware representation of image which is suitable for both tasks. The reconstruction losses provide additional regularization for learning of the representation. Our approach outperforms the state-of-the-art (SOTA) methods on widely-used benchmarks. Specifically, on the task of state-object composition, ContraNet achieves 8.7% and 8.1% performance gain on UT-Zappos and MIT-States on best HM metric, respectively. For the image retrieval task, ContraNet surpasses the SOTA performance by 4% on MIT-States and 5.3% on Fashion200k.