Prototype adjustment for zero shot classification

Abstract Zero shot classification addresses the problem of classifying unseen classes with seen class samples. Current zero shot learning methods mostly focus on learning the mapping function from image feature space to semantic space which is extremely important. However, these methods assume the seen and unseen class prototypes are fixed. A class prototype is referred to the semantic representation of a class. The semantic representation is represented by the attributes or word vectors which may be inaccurate and not discriminative. We attempt to find new prototypes that are more accurate for the zero shot classification tasks. In this paper, we proposed a Prototype adjustment method for the zero shot classification tasks (PAZSC) by adjusting the prototypes and learning the mapping function from image feature space to semantic space, simultaneously. The adjusted prototypes are more separable and discriminative for the zero shot classification tasks. A joint optimization function is proposed to learn the new prototypes and the mapping function. What is more, there is a domain shift problem in zero shot classification tasks caused by the disjointed seen and unseen images. We further learn a more generalizable mapping function to alleviate the domain shift problem. We have experimented on the state-of-the-art zero shot learning datasets, demonstrating that our PAZSL method has good performance.

[1]  Shuang Wu,et al.  Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Tao Xiang,et al.  Zero-Shot Learning on Semantic Class Prototype Graph , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Wei-Lun Chao,et al.  Synthesized Classifiers for Zero-Shot Learning , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Pietro Perona,et al.  The Caltech-UCSD Birds-200-2011 Dataset , 2011 .

[5]  Shaogang Gong,et al.  Transductive Multi-view Embedding for Zero-Shot Recognition and Annotation , 2014, ECCV.

[6]  Cees Snoek,et al.  COSTA: Co-Occurrence Statistics for Zero-Shot Classification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[8]  Andrew Y. Ng,et al.  Zero-Shot Learning Through Cross-Modal Transfer , 2013, NIPS.

[9]  James Hays,et al.  SUN attribute database: Discovering, annotating, and recognizing scene attributes , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Bernt Schiele,et al.  Evaluating knowledge transfer and zero-shot learning in a large-scale setting , 2011, CVPR 2011.

[11]  Shaogang Gong,et al.  Zero-shot object recognition by semantic manifold distance , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Daoqiang Zhang,et al.  Attribute relation learning for zero-shot classification , 2014, Neurocomputing.

[13]  Christoph H. Lampert,et al.  Attribute-Based Classification for Zero-Shot Visual Object Categorization , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Marc'Aurelio Ranzato,et al.  DeViSE: A Deep Visual-Semantic Embedding Model , 2013, NIPS.

[15]  Richard S. Zemel,et al.  Prototypical Networks for Few-shot Learning , 2017, NIPS.

[16]  Xiao Li,et al.  Zero-shot classification by transferring knowledge and preserving data structure , 2017, Neurocomputing.

[17]  Philip H. S. Torr,et al.  An embarrassingly simple approach to zero-shot learning , 2015, ICML.

[18]  Xiao Li,et al.  Learning unseen visual prototypes for zero-shot classification , 2018, Knowl. Based Syst..

[19]  Pietro Perona,et al.  One-shot learning of object categories , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Wei Liu,et al.  Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Yi Yang,et al.  Learning Discriminative Latent Attributes for Zero-Shot Classification , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Zhongfei Zhang,et al.  Zero-Shot Learning via Latent Space Encoding , 2017, IEEE Transactions on Cybernetics.

[23]  Cordelia Schmid,et al.  Label-Embedding for Image Classification , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Radha Poovendran,et al.  Activity Recognition Using a Combination of Category Components and Local Models for Video Surveillance , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[25]  Samy Bengio,et al.  Zero-Shot Learning by Convex Combination of Semantic Embeddings , 2013, ICLR.

[26]  Bernt Schiele,et al.  Evaluation of output embeddings for fine-grained image classification , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Kristen Grauman,et al.  Zero-shot recognition with unreliable attributes , 2014, NIPS.

[28]  Abhinav Gupta,et al.  Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Shaogang Gong,et al.  Unsupervised Domain Adaptation for Zero-Shot Learning , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[30]  Christoph H. Lampert,et al.  Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Jianmin Wang,et al.  Learning Predictable and Discriminative Attributes for Visual Recognition , 2015, AAAI.

[32]  Tianbao Yang,et al.  Learning Attributes Equals Multi-Source Domain Generalization , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Tao Xiang,et al.  Learning to Compare: Relation Network for Few-Shot Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Ramakant Nevatia,et al.  DECK: Discovering Event Composition Knowledge from Web Images for Zero-Shot Event Detection and Recounting in Videos , 2017, AAAI.

[35]  Luc Van Gool,et al.  One-Shot Video Object Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Meng Wang,et al.  Zero-Shot Learning via Attribute Regression and Class Prototype Rectification , 2018, IEEE Transactions on Image Processing.

[37]  Chee Seng Chan,et al.  Recognizing unknown objects with attributes relationship model , 2015, Expert Syst. Appl..

[38]  Dacheng Tao,et al.  Selective Zero-Shot Classification with Augmented Attributes , 2018, ECCV.

[39]  Babak Saleh,et al.  Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions , 2013, 2013 IEEE International Conference on Computer Vision.

[40]  Zhongfei Zhang,et al.  Zero-shot learning with Multi-Battery Factor Analysis , 2017, Signal Process..

[41]  David A. Forsyth,et al.  Describing objects by their attributes , 2009, CVPR.

[42]  Wei-Lun Chao,et al.  An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild , 2016, ECCV.

[43]  Bernt Schiele,et al.  What helps where – and why? Semantic relatedness for knowledge transfer , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  Shih-Fu Chang,et al.  Designing Category-Level Attributes for Discriminative Visual Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Xun Xu,et al.  Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation , 2016, ECCV.

[46]  Ming Shao,et al.  Generative Zero-Shot Learning via Low-Rank Embedded Semantic Dictionary , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Qiang Yu,et al.  Random Forest Classifier for Zero-Shot Learning Based on Relative Attribute , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Silvio Savarese,et al.  Recognizing human actions by attributes , 2011, CVPR 2011.

[49]  Bernt Schiele,et al.  Latent Embeddings for Zero-Shot Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).