Query-by-example HDR image retrieval based on CNN

Due to the expension of High Dynamic Range (HDR) imaging applications into various aspects of daily life, an efficient retrieval system, tailored to this type of data, has become a pressing challenge. In this paper, the reliability of Convolutional Neural Networks (CNN) descriptor and its investigation for HDR image retrieval are studied. The main idea consists in exploring the use of CNN to determine HDR image descriptor. Specifically, a Perceptually Uniform (PU) encoding is initially applied to the HDR content to map the luminance values in a perceptually uniform scale. Afterward, the CNN features, using Fully Connected (FC) layer activation, are extracted and classified by applying the Support Vector Machines (SVM) algorithm. Experimental evaluation demonstrates that the CNN descriptor, using the VGG19 network, achieves satisfactory results for describing HDR images on public available datasets such as PascalVoc2007, Cifar-10 and Wang. The experimental results also show that the features, after a PU processing, are more descriptive than those directly extracted from HDR contents. Finally, we show the superior performance of the proposed method against a recent state-of-the-art technique.

[1]  Albert Gordo,et al.  End-to-End Learning of Deep Visual Representations for Image Retrieval , 2016, International Journal of Computer Vision.

[2]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[3]  Ronan Sicre,et al.  Particular object retrieval with integral max-pooling of CNN activations , 2015, ICLR.

[4]  Rae-Hong Park,et al.  Tone mapping with contrast preservation and lightness correction in high dynamic range imaging , 2016, Signal Image Video Process..

[5]  Quan Quan,et al.  A multi-phase blending method with incremental intensity for training detection networks , 2020, The Visual Computer.

[6]  Heng Tao Shen,et al.  Unified Binary Generative Adversarial Network for Image Retrieval and Compression , 2020, International Journal of Computer Vision.

[7]  Hans-Peter Seidel,et al.  High Dynamic Range Imaging , 2015 .

[8]  Gregory Ward Larson,et al.  LogLuv Encoding for Full-Gamut, High-Dynamic Range Images , 1998, J. Graphics, GPU, & Game Tools.

[9]  Victor S. Lempitsky,et al.  Neural Codes for Image Retrieval , 2014, ECCV.

[10]  Atsuto Maki,et al.  Visual Instance Retrieval with Deep Convolutional Networks , 2014, ICLR.

[11]  Manuel Menezes de Oliveira Neto,et al.  High-quality brightness enhancement functions for real-time reverse tone mapping , 2009, The Visual Computer.

[12]  Hans-Peter Seidel,et al.  Extending quality metrics to full luminance range images , 2008, Electronic Imaging.

[13]  Manuel Menezes de Oliveira Neto,et al.  High-Quality Reverse Tone Mapping for a Wide Range of Exposures , 2014, 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images.

[14]  Alberto Del Bimbo,et al.  Image Retrieval using Multi-scale CNN Features Pooling , 2020, ICMR.

[15]  Yoshihiro Kanamori,et al.  Deep reverse tone mapping , 2017, ACM Trans. Graph..

[16]  Pavel Zemcík,et al.  Evaluation of feature point detection in high dynamic range imagery , 2016, J. Vis. Commun. Image Represent..

[17]  Diego Gutierrez,et al.  Dynamic range expansion based on image statistics , 2015, Multimedia Tools and Applications.

[18]  Kurt Debattista,et al.  HDR video past, present and future: A perspective , 2017, Signal Process. Image Commun..

[19]  Svetlana Lazebnik,et al.  Multi-scale Orderless Pooling of Deep Convolutional Activation Features , 2014, ECCV.

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Noel E. O'Connor,et al.  Bags of Local Convolutional Features for Scalable Instance Search , 2016, ICMR.

[22]  Greg Ward.  II.5 – REAL PIXELS , 1991 .

[23]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[24]  Alberto Del Bimbo,et al.  Fisher Encoded Convolutional Bag-of-Windows for Efficient Image Retrieval and Social Image Tagging , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[25]  Giorgos Tolias,et al.  Fine-Tuning CNN Image Retrieval with No Human Annotation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Qixiang Ye,et al.  Orientation robust object detection in aerial images using deep convolutional neural network , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[28]  Simon Osindero,et al.  Cross-Dimensional Weighting for Aggregated Deep Convolutional Features , 2015, ECCV Workshops.

[29]  Yiteng Pan,et al.  Learning social representations with deep autoencoder for recommender system , 2020, World Wide Web.

[30]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[31]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[32]  Azza Ouled Zaid,et al.  Color Based HDR Image Retrieval using HSV Histogram and Color Moments , 2018, 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA).

[33]  Jiwen Lu,et al.  Learning Compact Binary Descriptors with Unsupervised Deep Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Trevor Darrell,et al.  Part-Based R-CNNs for Fine-Grained Category Detection , 2014, ECCV.

[35]  Fazhi He,et al.  DRCDN: learning deep residual convolutional dehazing networks , 2019, The Visual Computer.

[36]  Jinsong Zhang,et al.  Learning High Dynamic Range from Outdoor Panoramas , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Miroslaw Bober,et al.  REMAP: Multi-Layer Entropy-Guided Pooling of Dense CNN Features for Image Retrieval , 2019, IEEE Transactions on Image Processing.

[38]  Meha Hachani,et al.  A new indexing method of HDR images using color histograms , 2017, International Conference on Machine Vision.

[39]  Heng Tao Shen,et al.  Hierarchical LSTMs with Adaptive Attention for Visual Captioning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Francesco Banterle,et al.  Inverse tone mapping , 2006, GRAPHITE '06.

[41]  Shengdong Zhang,et al.  NLDN: Non-local dehazing network for dense haze removal , 2020, Neurocomputing.

[42]  Gabriel Eilertsen,et al.  HDR image reconstruction from a single exposure using deep CNNs , 2017, ACM Trans. Graph..

[43]  Trevor Darrell,et al.  Learning with Recursive Perceptual Representations , 2012, NIPS.

[44]  Larry S. Davis,et al.  Exploiting local features from deep networks for image retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Giuseppe Valenzise,et al.  An evaluation of HDR image matching under extreme illumination changes , 2016, 2016 Visual Communications and Image Processing (VCIP).

[46]  Giuseppe Valenzise,et al.  Evaluation of Feature Detection in HDR Based Imaging Under Changes in Illumination Conditions , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[47]  Azza Ouled Zaid,et al.  HDR image retrieval by using color-based descriptor and tone mapping operator , 2019, The Visual Computer.

[48]  Patrick Le Callet,et al.  High Dynamic Range Video - From Acquisition, to Display and Applications , 2016 .

[49]  Azza Ouled Zaid,et al.  A New Retrieval System Based on Low Dynamic Range Expansion and SIFT Descriptor , 2018, 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP).