A Hybrid Framework for Matching Printing Design Files to Product Photos

We propose a real-time image matching framework, which is hybrid in the sense that it uses both hand-crafted features and deep features obtained from a well-tuned deep convolutional network. The matching problem, which we concentrate on, is specific to a certain application, that is, printing design to product photo matching. Printing designs are any kind of template image files, created using a design tool, thus are perfect image signals. However, photographs of a printed product suffer many unwanted effects, such as uncontrolled shooting angle, uncontrolled illumination, occlusions, printing deficiencies in color, camera noise, optic blur, et cetera. For this purpose, we create an image set that includes printing design and corresponding product photo pairs with collaboration of an actual printing facility. Using this image set, we benchmark various hand-crafted and deep features for matching performance and propose a framework in which deep learning is utilized with highest contribution, but without disabling real-time operation using an ordinary desktop computer.

[1]  Touqeer Ahmad,et al.  Comparison of semantic segmentation approaches for horizon/sky line detection , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[2]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[3]  Hui Zhang,et al.  Image segmentation evaluation: A survey of unsupervised methods , 2008, Comput. Vis. Image Underst..

[4]  Jun Li,et al.  A novel approach for visual Saliency detection and segmentation based on objectness and top-down attention , 2017, 2017 2nd International Conference on Image, Vision and Computing (ICIVC).

[5]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[6]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Gerald Schaefer.  UCID-RAW – A Colour Image Database in Raw Format , 2017 .

[8]  Vivek Jain,et al.  A Survey : On Content Based Image Retrieval , 2013 .

[9]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Andrea Vedaldi,et al.  Understanding Image Representations by Measuring Their Equivariance and Equivalence , 2014, International Journal of Computer Vision.

[11]  Xu Yang,et al.  A novel approach for visual Saliency detection and segmentation based on objectness and top-down attention , 2017 .

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[14]  Martin Thoma,et al.  A Survey of Semantic Segmentation , 2016, ArXiv.

[15]  Ludmila I. Kuncheva,et al.  On the optimality of Naïve Bayes with dependent binary features , 2006, Pattern Recognit. Lett..

[16]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[17]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[18]  Deng Cai,et al.  A Revisit on Deep Hashings for Large-scale Content Based Image Retrieval , 2017, ArXiv.

[19]  Huafeng Wang,et al.  Deep Learning for Image Retrieval: What Works and What Doesn't , 2015, 2015 IEEE International Conference on Data Mining Workshop (ICDMW).

[20]  Joni-Kristian Kämäräinen,et al.  A comparison of local feature detectors and descriptors for visual object categorization by intra-class repeatability and matching , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[21]  Yu Liu,et al.  A review of semantic segmentation using deep neural networks , 2017, International Journal of Multimedia Information Retrieval.

[22]  Seungmin Rho,et al.  Medical image semantic segmentation based on deep learning , 2017, Neural Computing and Applications.

[23]  Esa Rahtu,et al.  Siamese network features for image matching , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[24]  David Stutz,et al.  Neural Codes for Image Retrieval , 2015 .

[25]  Jie Lin,et al.  A practical guide to CNNs and Fisher Vectors for image instance retrieval , 2015, Signal Process..

[26]  Martin Jägersand,et al.  Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[27]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[28]  Thomas Martin Deserno,et al.  The CLEF 2005 Cross-Language Image Retrieval Track , 2003, CLEF.

[29]  José García Rodríguez,et al.  A Review on Deep Learning Techniques Applied to Semantic Segmentation , 2017, ArXiv.

[30]  Harald Sack,et al.  Does one size really fit all?: Evaluating classifiers in bag-of-visual-words classification , 2014, i-KNOW '14.

[31]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[33]  Serge J. Belongie,et al.  Learning deep representations for ground-to-aerial geolocalization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Wei Sun,et al.  Methods and datasets on semantic segmentation: A review , 2018, Neurocomputing.

[35]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[36]  James Ze Wang,et al.  Content-based image retrieval: approaches and trends of the new age , 2005, MIR '05.

[37]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[38]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[39]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[41]  Mahmood Fathy,et al.  Semantic Video Segmentation: A Review on Recent Approaches , 2018, ArXiv.