Multi-Attribute Open Set Recognition

Open Set Recognition (OSR) extends image classification to an open-world setting, by simultaneously classifying known classes and identifying unknown ones. While conventional OSR approaches can detect Out-of-Distribution (OOD) samples, they cannot provide explanations indicating which underlying visual attribute(s) (e.g., shape, color or background) cause a specific sample to be unknown. In this work, we introduce a novel problem setup that generalizes conventional OSR to a multi-attribute setting, where multiple visual attributes are simultaneously recognized. Here, OOD samples can be not only identified but also categorized by their unknown attribute(s). We propose simple extensions of common OSR baselines to handle this novel scenario. We show that these baselines are vulnerable to shortcuts when spurious correlations exist in the training dataset. This leads to poor OOD performance which, according to our experiments, is mainly due to unintended crossattribute correlations of the predicted confidence scores. We provide an empirical evidence showing that this behavior is consistent across different baselines on both synthetic and real world datasets.

[1]  Ziming Zhang,et al.  Class-attribute inconsistency learning for novelty detection , 2022, Pattern Recognit..

[2]  A. Vedaldi,et al.  Open-Set Recognition: A Good Closed-Set Classifier is All You Need , 2021, ICLR.

[3]  Yonghong Tian,et al.  Adversarial Reciprocal Points Learning for Open Set Recognition , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Chaithanya Kumar Mummadi,et al.  DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  Alessandro Sperduti,et al.  Conditional Variational Capsule Network for Open Set Recognition , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  De-Chuan Zhan,et al.  Learning Placeholders for Open-Set Recognition , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Federico Tombari,et al.  Learning Graph Embeddings for Compositional Zero-shot Learning , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Muhammad Ferjad Naeem,et al.  Open World Compositional Zero-Shot Learning , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Songcan Chen,et al.  Recent Advances in Open Set Recognition: A Survey , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Aaron C. Courville,et al.  Systematic generalisation with group invariant predictions , 2021, ICLR.

[11]  Uwe von Lukas,et al.  Towards Combined Open Set Recognition and Out-of-Distribution Detection for Fine-grained Classification , 2021, VISIGRAPP.

[12]  Shiliang Pu,et al.  Learning Open Set Network with Discriminative Reciprocal Points , 2020, ECCV.

[13]  Andrew Kyle Lampinen,et al.  What shapes feature representations? Exploring datasets, architectures, and training , 2020, NeurIPS.

[14]  Gal Chechik,et al.  A causal view of compositional zero-shot recognition , 2020, NeurIPS.

[15]  M. Bethge,et al.  Shortcut learning in deep neural networks , 2020, Nature Machine Intelligence.

[16]  Ang Li,et al.  Hybrid Models for Open Set Recognition , 2020, ECCV.

[17]  Xin Sun,et al.  Conditional Gaussian Distribution Learning for Open Set Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Tiejun Huang,et al.  P-ODN: Prototype-based Open Deep Network for Open Set Recognition , 2019, Scientific Reports.

[19]  Marc'Aurelio Ranzato,et al.  Task-Driven Modular Networks for Zero-Shot Compositional Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Vishal M. Patel,et al.  C2AE: Class Conditioned Auto-Encoder for Open-Set Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Takeshi Naemura,et al.  Classification-Reconstruction Learning for Open-Set Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Matthias Bethge,et al.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness , 2018, ICLR.

[23]  Weng-Keen Wong,et al.  Open Set Learning with Counterfactual Images , 2018, ECCV.

[24]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Kristen Grauman,et al.  Attributes as Operators , 2018, ECCV.

[26]  Rahil Garnavi,et al.  Generative OpenMax for Multi-Class Open Set Classification , 2017, BMVC.

[27]  Martial Hebert,et al.  From Red Wine to Red Tomato: Composition with Context , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Kristen Grauman,et al.  Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Terrance E. Boult,et al.  Towards Open Set Deep Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Kristen Grauman,et al.  Fine-Grained Visual Comparisons with Local Learning , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[32]  Anderson Rocha,et al.  Toward Open Set Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.