Spatiotemporal Colorization of Video Using 3D Steerable Pyramids

We propose a new technique for video colorization based on spatiotemporal color propagation in the 3D video volume, utilizing the dominant orientation response obtained from the steerable pyramid decomposition of the video. The volumetric color diffusion from the sources that are marked by scribbles occurs across spatiotemporally smooth regions, and the prevention of leakage is facilitated by the spatiotemporal discontinuities in the output of steerable filters, representing the object boundaries and motion boundaries. Unlike most existing methods, our approach dispenses with the need of motion vectors for interframe color transfer and provides a general framework for image and video colorization. The experimental results establish the effectiveness of the proposed approach in colorizing videos having different types of motion and visual content even in the presence of occlusion, in terms of accuracy and computational requirements.

[1]  S. Kollias,et al.  Dense saliency-based spatiotemporal feature points for action recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  J. E. Bendz,et al.  Visually Improved Understanding of Three-Dimensionally Propagating Electromagnetic Fields in Wireless Networks , 2007 .

[3]  Ling Shao,et al.  Spatio-temporal steerable pyramid for human action recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[4]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Sang Uk Lee,et al.  Prioritized image and video colorization based on Gaussian pyramid of gradient images , 2012, J. Electronic Imaging.

[6]  Neucimar J. Leite,et al.  Rotation-Invariant and Scale-Invariant Steerable Pyramid Decomposition for Texture Image Retrieval , 2007 .

[7]  Aurélie Bugeau,et al.  Variational Exemplar-Based Image Colorization , 2014, IEEE Transactions on Image Processing.

[8]  Guillermo Sapiro,et al.  Fast image and video colorization using chrominance blending , 2006, IEEE Transactions on Image Processing.

[9]  Hanqiu Sun,et al.  Video Colorization Using Parallel Optimization in Feature Space , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Enhua Wu,et al.  Colorization Using the Rotation-Invariant Feature Space , 2011, IEEE Computer Graphics and Applications.

[11]  Jan J. Koenderink.  Color Spaces , 2000, Color Imaging Conference.

[12]  Camillo J. Taylor,et al.  Hierarchically-constrained optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ma Lizhuang,et al.  An automatic image and video colorization algorithm based on pattern continuity , 2012, 2012 International Conference on Audio, Language and Image Processing.

[14]  Minglun Gong,et al.  Integrated foreground segmentation and boundary matting for live videos. , 2015, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[15]  Yukihiko Yamashita,et al.  Self-similarity-based image colorization , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[16]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Aurélie Bugeau,et al.  Luminance-Hue Specification in the RGB Space , 2015, SSVM.

[18]  Konstantinos G. Derpanis,et al.  Three-dimensional nth derivative of Gaussian separable steerable filters , 2005, IEEE International Conference on Image Processing 2005.

[19]  Dani Lischinski,et al.  Colorization by example , 2005, EGSR '05.

[20]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[21]  Jirí Zára,et al.  Unsupervised colorization of black-and-white cartoons , 2004, NPAR '04.

[22]  Sang Uk Lee,et al.  Image and video colorization based on prioritized source propagation , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[23]  Sumana Gupta,et al.  Colorization of grayscale images and videos using a semiautomatic approach , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[24]  Takahiko Horiuchi,et al.  Colorization algorithm for grayscale image by propagating seed pixels , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[25]  Toshihiro Furukawa,et al.  Image colorization based on the mixed l0/l1 norm minimization , 2012, 2012 19th IEEE International Conference on Image Processing.

[26]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..

[27]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[28]  Guillermo Sapiro,et al.  A Geodesic Framework for Fast Interactive Image and Video Segmentation and Matting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[29]  Richard P. Wildes,et al.  The Applicability of Spatiotemporal Oriented Energy Features to Region Tracking , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Lei Li,et al.  An Interactive Framework for Video Colorization , 2013, 2013 Seventh International Conference on Image and Graphics.

[31]  Ying Wu,et al.  Scribble Tracker: A Matting-Based Approach for Robust Tracking , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.