Pixel-Level Non-local Image Smoothing With Objective Evaluation

Recently, image smoothing has gained increasing attention due to its prerequisite role in other image processing tasks, e.g., image enhancement and editing. However, the evaluation of image smoothing algorithms is usually performed by subjective observation on images without corresponding ground truths. To promote the development of image smoothing algorithms, in this paper, we construct a novel Nankai Smoothing (NKS) dataset containing 200 images blended by versatile structure images and natural textures. The structure images are inherently smooth and naturally taken as ground truths. On our NKS dataset, we comprehensively evaluate 14 popular image smoothing algorithms. Moreover, we propose a Pixel-level Non-Local Smoothing (PNLS) method to well preserve the structure of the smoothed images, by exploiting the pixel-level non-local self-similarity prior of natural images. Extensive experiments on several benchmark datasets demonstrate that our PNLS outperforms previous algorithms on the image smoothing task. Ablation studies also reveal the work mechanism of our PNLS on image smoothing. To further show its effectiveness, we apply our PNLS on several applications such as semantic region smoothing, detail/edge enhancement, and image abstraction. The dataset and code are available at https://github.com/zal0302/PNLS.

[1]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Nick Barnes,et al.  Deep Texture and Structure Aware Filtering Network for Image Smoothing , 2017, ECCV.

[3]  Wei Liu,et al.  Semi-Global Weighted Least Squares in Image Filtering , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  David Zhang,et al.  Patch Group Based Nonlocal Self-Similarity Prior Learning for Image Denoising , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  David Zhang,et al.  A Hybrid l1-l0 Layer Decomposition Model for Tone Mapping , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Ling Shao,et al.  Noisy-As-Clean: Learning Unsupervised Denoising from the Corrupted Image , 2019, ArXiv.

[7]  Jia Xu,et al.  Fast Image Processing with Fully-Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Lei Zhang,et al.  A Benchmark for Edge-Preserving Image Smoothing , 2019, IEEE Transactions on Image Processing.

[10]  Qi Zhang,et al.  Rolling Guidance Filter , 2014, ECCV.

[11]  Shiming Xiang,et al.  Segment Graph Based Image Filtering: Fast Structure-Preserving Smoothing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Jiaya Jia,et al.  Convolutional Neural Pyramid for Image Processing , 2017, ArXiv.

[13]  Kunal N. Chaudhury,et al.  Fast Adaptive Bilateral Filtering , 2018, IEEE Transactions on Image Processing.

[14]  Jiaolong Yang,et al.  Image smoothing via unsupervised learning , 2018, ACM Trans. Graph..

[15]  Xianming Liu,et al.  Connecting Image Denoising and High-Level Vision Tasks via Deep Learning , 2018, IEEE Transactions on Image Processing.

[16]  David Zhang,et al.  Partial Deconvolution With Inaccurate Blur Kernel , 2018, IEEE Transactions on Image Processing.

[17]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[18]  Narendra Ahuja,et al.  Deep Joint Image Filtering , 2016, ECCV.

[19]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[20]  W. Zuo,et al.  Deep Learning on Image Denoising: An overview , 2019, Neural Networks.

[21]  Raanan Fattal,et al.  Edge-avoiding wavelets and their applications , 2009, ACM Trans. Graph..

[22]  David Zhang,et al.  Multi-channel Weighted Nuclear Norm Minimization for Real Color Image Denoising , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Minh N. Do,et al.  Fast Global Image Smoothing Based on Weighted Least Squares , 2014, IEEE Transactions on Image Processing.

[24]  Ling Shao,et al.  STAR: A Structure and Texture Aware Retinex Model , 2019, IEEE Transactions on Image Processing.

[25]  Ling Shao,et al.  RANet: Ranking Attention Network for Fast Video Object Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[26]  Richard Szeliski,et al.  Noise Estimation from a Single Image , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  Zhen Ji,et al.  Edge-Preserving Texture Suppression Filter Based on Joint Filtering Schemes , 2013, IEEE Transactions on Multimedia.

[28]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Gang Wang,et al.  Tree Filtering: Efficient Structure-Preserving Smoothing With a Minimum Spanning Tree , 2014, IEEE Transactions on Image Processing.

[30]  Bo Wang,et al.  Scale-Aware Edge-Preserving Image Filtering via Iterative Global Optimization , 2018, IEEE Transactions on Multimedia.

[31]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[32]  I. Daubechies,et al.  Factoring wavelet transforms into lifting steps , 1998 .

[33]  David Zhang,et al.  External Prior Guided Internal Prior Learning for Real-World Noisy Image Denoising , 2017, IEEE Transactions on Image Processing.

[34]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[35]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[36]  Zeev Farbman,et al.  Edge-preserving decompositions for multi-scale tone and detail manipulation , 2008, ACM Trans. Graph..

[37]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[38]  Ling Shao,et al.  NLH: A Blind Pixel-Level Non-Local Method for Real-World Image Denoising , 2019, IEEE Transactions on Image Processing.

[39]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[40]  Cewu Lu,et al.  Combining sketch and tone for pencil drawing production , 2012, NPAR '12.

[41]  Jingdong Wang,et al.  Salient Object Detection: A Discriminative Regional Feature Integration Approach , 2013, International Journal of Computer Vision.

[42]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Wangmeng Zuo,et al.  Attention-guided CNN for image denoising , 2020, Neural Networks.

[44]  A. Haar.  Zur Theorie der orthogonalen Funktionensysteme , 1910 .

[45]  Renjie Liao,et al.  Deep Edge-Aware Filters , 2015, ICML.

[46]  David Zhang,et al.  Real-world Noisy Image Denoising: A New Benchmark , 2018, ArXiv.

[47]  Ling Shao,et al.  Conditional Variational Image Deraining , 2020, IEEE Transactions on Image Processing.

[48]  Li Xu,et al.  Structure extraction from texture via relative total variation , 2012, ACM Trans. Graph..

[49]  B. Gooch,et al.  Real-time video abstraction , 2006, ACM Trans. Graph..

[50]  W. Sweldens.  The Lifting Scheme: A Custom - Design Construction of Biorthogonal Wavelets "Industrial Mathematics , 1996 .

[51]  Ming-Hsuan Yang,et al.  Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network , 2016, ECCV.

[52]  Thomas S. Huang,et al.  Non-Local Recurrent Network for Image Restoration , 2018, NeurIPS.

[53]  David Zhang,et al.  A Trilateral Weighted Sparse Coding Scheme for Real-World Image Denoising , 2018, ECCV.

[54]  Szymon Rusinkiewicz,et al.  Multiscale shape and detail enhancement from multi-light image collections , 2007, ACM Trans. Graph..

[55]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.