A generalized hybrid nonconvex variational regularization model for staircase reduction in image restoration

Abstract Total variation (TV) regularization model has the excellent performance in noise-removing and edge-preserving. However, it often yields staircase artifacts in the smooth region of the restorations. To attack this problem, we propose a generalized hybrid nonconvex variational regularization model in this paper, which utilizes two nonconvex regularizers to impose the priors to the two different components of the images, respectively. One is nonconvex TV regularizer that measures the piecewise constant component of the input image, and the other is nonconvex Laplacian regularizer that models the piecewise smooth component. New model inherits the advantages of the nonconvex regularization and the first- and second-order hybrid variational regularization, which can well remove the noises while preserving edges and reducing staircase artifacts. To solve this nonconvex minimization problem efficiently, we propose a first-order algorithm based on alternating direction method of multipliers (ADMM) combining with majorization–minimization (MM) scheme. In addition, a sufficient condition for the convergence of the proposed algorithm is provided. In the experiments, we compare the proposed model with several state-of-the-art image denoising models, numerical results illustrate the effectiveness of the proposed model and algorithm for both synthetic and real images in terms of peak signal to noise ratio (PSNR) and structural similarity (SSIM) indexes.