Multi-Feature Fusion Method Based on Salient Object Detection for Beauty Product Retrieval

Beauty and Personal care product retrieval has attracted more and more attention due to its wide application value. However, due to the diversity of data and the complexity of image background, this task is very challenging. In this paper, we propose a multi-feature fusion method based on salient object detection to improve retrieval performance. The key of our method is to extract the foreground objects of the query set by using the salient object detection network, so as to eliminate the background interference. Then the foreground target images and dataset are put into the multi-classification networks to extract multiple fusion features for retrieval. We use the perfect-500k dataset for experiments, and the results show that our method is effective. Our method ranked 2st in the Grand Challenge of AI Meets Beauty in ACM Multimedia 2020 with a MAP score of 0.43729. We released our code on GitHub:github.com/R-M-Yan/ACMMM2020AIMeetBeauty.

[1]  Ali Borji,et al.  Salient Object Detection: A Benchmark , 2015, IEEE Transactions on Image Processing.

[2]  Huchuan Lu,et al.  Learning to Detect Salient Objects with Image-Level Supervision , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Huchuan Lu,et al.  Attentive Feedback Network for Boundary-Aware Salient Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Giorgos Tolias,et al.  Fine-Tuning CNN Image Retrieval with No Human Annotation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[7]  Yannis Avrithis,et al.  Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Dahua Lin,et al.  PolyNet: A Pursuit of Structural Diversity in Very Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[10]  Sabine Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Zhenguo Yang,et al.  Regional Maximum Activations of Convolutions with Attention for Cross-domain Beauty and Personal Care Product Retrieval , 2018, ACM Multimedia.

[12]  Yael Pritch,et al.  Saliency filters: Contrast based filtering for salient region detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Zhuowen Tu,et al.  Deeply Supervised Salient Object Detection with Short Connections , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[15]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[16]  Jiawei Wang,et al.  The Retrieval of the Beautiful: Self-Supervised Salient Object Detection for Beauty Product Retrieval , 2019, ACM Multimedia.

[17]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Cordelia Schmid,et al.  Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search , 2008, ECCV.

[19]  Haoran Xie,et al.  Cross-domain Beauty Item Retrieval via Unsupervised Embedding Learning , 2019, ACM Multimedia.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Qi Tian,et al.  SIFT Meets CNN: A Decade Survey of Instance Retrieval , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Chao Gao,et al.  BASNet: Boundary-Aware Salient Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Kai Xu,et al.  Beauty Product Image Retrieval Based on Multi-Feature Fusion and Feature Aggregation , 2018, ACM Multimedia.

[24]  Chee Seng Chan,et al.  Unprecedented Usage of Pre-trained CNNs on Beauty Product , 2018, ACM Multimedia.

[25]  Svetlana Lazebnik,et al.  Multi-scale Orderless Pooling of Deep Convolutional Activation Features , 2014, ECCV.

[26]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[29]  Qi Tian,et al.  Image Classification and Retrieval are ONE , 2015, ICMR.

[30]  Ronan Sicre,et al.  Particular object retrieval with integral max-pooling of CNN activations , 2015, ICLR.

[31]  Kai Xu,et al.  Improving cross-dimensional weighting pooling with multi-scale feature fusion for image retrieval , 2019, Neurocomputing.

[32]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[33]  Lingyun Yu,et al.  Beauty Product Retrieval Based on Regional Maximum Activation of Convolutions with Generalized Attention , 2019, ACM Multimedia.