Fine-Grained Visual Comparisons with Local Learning

Given two images, we want to predict which exhibits a particular visual attribute more than the other-even when the two images are quite similar. Existing relative attribute methods rely on global ranking functions; yet rarely will the visual cues relevant to a comparison be constant for all data, nor will humans' perception of the attribute necessarily permit a global ordering. To address these issues, we propose a local learning approach for fine-grained visual comparisons. Given a novel pair of images, we learn a local ranking model on the fly, using only analogous training comparisons. We show how to identify these analogous pairs using learned metrics. With results on three challenging datasets-including a large newly curated dataset for fine-grained comparisons-our method outperforms stateof-the-art methods for relative attribute prediction.

[1]  Vincent Conitzer,et al.  Improved Bounds for Computing Kemeny Rankings , 2006, AAAI.

[2]  Pascal Vincent,et al.  K-Local Hyperplane and Convex Distance Nearest Neighbor Algorithms , 2001, NIPS.

[3]  Larry S. Davis,et al.  Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance , 2011, 2011 International Conference on Computer Vision.

[4]  Ankur Datta,et al.  Hierarchical ranking of facial attributes , 2011, Face and Gesture 2011.

[5]  Andrew W. Moore,et al.  Locally Weighted Learning , 1997, Artificial Intelligence Review.

[6]  Shaogang Gong,et al.  Cumulative Attribute Space for Age and Crowd Density Estimation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Kristen Grauman,et al.  Relative attributes , 2011, 2011 International Conference on Computer Vision.

[8]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[9]  Mark S. Nixon,et al.  Enriching Texture Analysis with Semantic Data , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Shree K. Nayar,et al.  Attribute and simile classifiers for face verification , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[11]  Meinolf Sellmann,et al.  Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems: 6th International Conference, CPAIOR 2009 Pittsburgh, PA, USA, May 27-31, 2009 Proceedings , 2009, CPAIOR.

[12]  Jitendra Malik,et al.  SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[13]  Prateek Jain,et al.  Fast image search for learned metrics , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[15]  Kevin Duh,et al.  Learning to rank with partially-labeled data , 2008, SIGIR '08.

[16]  Harry Shum,et al.  Query Dependent Ranking Using K-nearest Neighbor * , 2022 .

[17]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[18]  Hsin-Hsi Chen,et al.  Query-Dependent Rank Aggregation with Local Models , 2011, AIRS.

[19]  Abhinav Gupta,et al.  Constrained Semi-Supervised Learning Using Attributes and Comparative Attributes , 2012, ECCV.

[20]  Jitendra Malik,et al.  Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[21]  Léon Bottou,et al.  Local Learning Algorithms , 1992, Neural Computation.

[22]  Inderjit S. Dhillon,et al.  Information-theoretic metric learning , 2006, ICML '07.

[23]  Alexei A. Efros,et al.  Recognition by association via learning per-exemplar distances , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Craig Boutilier.  Preference Elicitation and Preference Learning in Social Choice , 2011, CPAIOR.

[26]  Alexander C. Berg,et al.  Automatic Attribute Discovery and Characterization from Noisy Web Data , 2010, ECCV.

[27]  Serge J. Belongie,et al.  Relative ranking of facial attractiveness , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[28]  Mark S. Nixon,et al.  Using comparative human descriptions for soft biometrics , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[29]  Avinava Dubey,et al.  Efficient and Accurate Local Learning for Ranking , 2009 .

[30]  Adriana Kovashka,et al.  WhittleSearch: Image search with relative attribute feedback , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Arijit Biswas,et al.  Simultaneous Active Learning of Classifiers & Attributes via Relative Feedback , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Shiguang Shan,et al.  Relative Forest for Attribute Prediction , 2012, ACCV.

[33]  Pietro Perona,et al.  Visual Recognition with Humans in the Loop , 2010, ECCV.

[34]  Yuan Yao,et al.  Statistical ranking and combinatorial Hodge theory , 2008, Math. Program..