Tensor based approach for inpainting of video containing sparse text

Videos received from certain sources may contain irrelevant contents which might reduce the amount of information conveyed by it. This paper proposes an effective tensor based video inpainting approach to improve the quality of these videos by removing and replacing unwanted contents with relevant information. The proposed method employs reweighted tensor decomposition technique to identify and discard the inappropriate sparse components of the video data. These sparse components are substituted with proper contents by utilising spatio-temporal consistency through reweighted tensor completion. The replacement is carried out in such a way that the resulting video possesses superior temporal consistency and visual credibility. The proposed method is applied to sparse text removal of videos having dynamic content in various extents and is found that our method outperforms its counterparts.

[1]  Lieven De Lathauwer,et al.  A survey of tensor methods , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[2]  Vicent Caselles,et al.  Exemplar-Based Image Inpainting Using Multiscale Graph Cuts , 2013, IEEE Transactions on Image Processing.

[3]  Carla D. Martin,et al.  An Order-p Tensor Factorization with Applications in Imaging , 2013, SIAM J. Sci. Comput..

[4]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[5]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[6]  Weisi Lin,et al.  Image Quality Assessment Using Multi-Method Fusion , 2013, IEEE Transactions on Image Processing.

[7]  Akshay Krishnamurthy,et al.  Low-Rank Matrix and Tensor Completion via Adaptive Sampling , 2013, NIPS.

[8]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[9]  Guillermo Sapiro,et al.  Video inpainting of occluding and occluded objects , 2005, IEEE International Conference on Image Processing 2005.

[10]  Caiming Zhang,et al.  Patch Grouping SVD-Based Denoising Aggregation Patch Grouping SVD-Based Denoising Aggregation Back Projection Noisy Image , 2015 .

[11]  Louis Laborelli,et al.  Missing data correction in still images and image sequences , 2002, MULTIMEDIA '02.

[12]  Wotao Yin,et al.  Parallel matrix factorization for low-rank tensor completion , 2013, ArXiv.

[13]  Chi-Keung Tang,et al.  Image repairing: robust image synthesis by adaptive ND tensor voting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[14]  Hongyu Zhao,et al.  Low-Rank Modeling and Its Applications in Image Analysis , 2014, ACM Comput. Surv..

[15]  Patrick Pérez,et al.  Video Inpainting of Complex Scenes , 2014, SIAM J. Imaging Sci..

[16]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[17]  Zuowei Shen,et al.  Robust Video Restoration by Joint Sparse and Low Rank Matrix Approximation , 2011, SIAM J. Imaging Sci..

[18]  Eric L. Miller,et al.  Tensor-Based Formulation and Nuclear Norm Regularization for Multienergy Computed Tomography , 2013, IEEE Transactions on Image Processing.

[19]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Pierre Comon,et al.  Tensors : A brief introduction , 2014, IEEE Signal Processing Magazine.

[21]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[22]  Rae-Hong Park,et al.  Red-eye detection and correction using inpainting in digital photographs , 2009, IEEE Transactions on Consumer Electronics.

[23]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[24]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, 2006 International Conference on Image Processing.

[25]  Liangpei Zhang,et al.  Total-Variation-Regularized Low-Rank Matrix Factorization for Hyperspectral Image Restoration , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Jian Sun,et al.  Statistics of Patch Offsets for Image Completion , 2012, ECCV.

[27]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[28]  Yong-Sheng Chen,et al.  Virtual Contour Guided Video Object Inpainting Using Posture Mapping and Retrieval , 2011, IEEE Transactions on Multimedia.

[29]  Misha Elena Kilmer,et al.  Novel Methods for Multilinear Data Completion and De-noising Based on Tensor-SVD , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[31]  Bin Dong,et al.  An Edge Driven Wavelet Frame Model for Image Restoration , 2017, Applied and Computational Harmonic Analysis.

[32]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[33]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[34]  Sudhish N. George,et al.  Twist tensor total variation regularized-reweighted nuclear norm based tensor completion for video missing area recovery , 2018, Inf. Sci..

[35]  Tao Ding,et al.  A Rank Minimization Approach to Video Inpainting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[36]  Oliver Grau,et al.  How Not to Be Seen — Object Removal from Videos of Crowded Scenes , 2012, Comput. Graph. Forum.

[37]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[38]  Nikos Komodakis,et al.  Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning , 2007, IEEE Transactions on Image Processing.

[39]  Tai-Pang Wu,et al.  Video repairing under variable illumination using cyclic motions , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Kaoru Arakawa,et al.  Removal of film scratches using exemplar-based inpainting with directional median filter , 2012, 2012 International Symposium on Communications and Information Technologies (ISCIT).

[41]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[42]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[43]  T. Chan,et al.  Image inpainting by correspondence maps: A deterministic approach , 2003 .

[44]  Wei Liu,et al.  Tensor Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Tensors via Convex Optimization , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[46]  Donald Goldfarb,et al.  Robust Low-Rank Tensor Recovery: Models and Algorithms , 2013, SIAM J. Matrix Anal. Appl..

[47]  Qionghai Dai,et al.  Reweighted Low-Rank Matrix Recovery and its Application in Image Restoration , 2014, IEEE Transactions on Cybernetics.

[48]  Christine Guillemot,et al.  Video Inpainting With Short-Term Windows: Application to Object Removal and Error Concealment , 2015, IEEE Transactions on Image Processing.

[49]  Wensheng Zhang,et al.  The Twist Tensor Nuclear Norm for Video Completion , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[50]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[51]  Zemin Zhang,et al.  Exact Tensor Completion Using t-SVD , 2015, IEEE Transactions on Signal Processing.

[52]  Misha Elena Kilmer,et al.  Third-Order Tensors as Operators on Matrices: A Theoretical and Computational Framework with Applications in Imaging , 2013, SIAM J. Matrix Anal. Appl..