Illumination-aware live videos background replacement using antialiasing optimization

We propose a real-time illumination-aware live videos background replacement approach with antialiasing optimization on GPU in this paper. The aim of background replacement for live videos is to substitute the current real-time backgrounds with specially-chosen background images. Here we assume that the camera is stationary and the beginning of the video is only with a pure background scene. We propose the colored locality sensitive histograms (CLSH) considering the influence of other pixels to each pixel in every color channel to improve the performance of background segmentation, which makes the segmentation results robust enough to illumination differences. With the segmentation results, we then introduce a blocked real-time matting approach to enhance the accuracy of the objects’ boundary. Finally, to make the video composition more realistic, we propose a local antialiasing method to recover the distortions on edges. Compared with existing background replacement methods, our approach does not require costly blue/green screen or depth camera, but can produce more reliable video composition results. We have applied hardware GPU parallelism to speed up the live background replacement. Our illumination-aware video background replacement runs very efficiently in real-time, which can be applied for various video applications. The experimental results have shown the efficiency and high-quality rendering of our video background replacement in real-time.

[1]  Thierry Bouwmans,et al.  BGS Library: A Library Framework for Algorithm’s Evaluation in Foreground/Background Segmentation , 2014 .

[2]  Matthias Nießner,et al.  Efficient GPU rendering of subdivision surfaces using adaptive quadtrees , 2016, ACM Trans. Graph..

[3]  Zeev Farbman,et al.  Coordinates for instant image cloning , 2009, ACM Trans. Graph..

[4]  Antoine Vacavant,et al.  A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos , 2014, Comput. Vis. Image Underst..

[5]  Michael Cohen,et al.  Soft scissors: an interactive tool for realtime high quality matting , 2007, SIGGRAPH 2007.

[6]  Yangmi Lim,et al.  Video background replacement using a genetic algorithm , 2008 .

[7]  Yao Lu,et al.  Coherent Parametric Contours for Interactive Video Object Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Gour C. Karmakar,et al.  Efficient Probabilistic Spatio-Temporal Video Object Segmentation , 2007, 6th IEEE/ACIS International Conference on Computer and Information Science (ICIS 2007).

[9]  Ruigang Yang,et al.  Automatic Real-Time Video Matting Using Time-of-Flight Camera and Multichannel Poisson Equations , 2012, International Journal of Computer Vision.

[10]  Thierry Chateau,et al.  A Benchmark Dataset for Outdoor Foreground/Background Extraction , 2012, ACCV Workshops.

[11]  John G. Eyles,et al.  PixelFlow: high-speed rendering using image composition , 1992, SIGGRAPH.

[12]  Guillaume-Alexandre Bilodeau,et al.  SuBSENSE: A Universal Change Detection Method With Local Adaptive Sensitivity , 2015, IEEE Transactions on Image Processing.

[13]  Carlo Tomasi,et al.  Alpha estimation in natural images , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[14]  Guillaume-Alexandre Bilodeau,et al.  A Self-Adjusting Approach to Change Detection Based on Background Word Consensus , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[15]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[16]  Dani Lischinski,et al.  Slippage-free background replacement for hand-held video , 2014, ACM Trans. Graph..

[17]  Tony DeRose,et al.  Feature-adaptive GPU rendering of Catmull-Clark subdivision surfaces , 2012, TOGS.

[18]  P. KaewTrakulPong,et al.  An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection , 2002 .

[19]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[20]  Rubén Heras Evangelio,et al.  Adaptively Splitted GMM With Feedback Improvement for the Task of Background Subtraction , 2014, IEEE Transactions on Information Forensics and Security.

[21]  John M. Hannah,et al.  Alpha channel estimation in high resolution images and image sequences , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[22]  Pascal Barla,et al.  Surface flows for image-based shading design , 2012, ACM Trans. Graph..

[23]  Zhe Zhu,et al.  3D modeling and motion parallax for improved videoconferencing , 2016, Computational Visual Media.

[24]  Marcus A. Magnor,et al.  Sampling based scene-space video processing , 2015, ACM Trans. Graph..

[25]  Gerhard Rigoll,et al.  Background segmentation with feedback: The Pixel-Based Adaptive Segmenter , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[26]  Shi-Min Hu,et al.  Robust background identification for dynamic video editing , 2016, ACM Trans. Graph..

[27]  G.C. Karmakar,et al.  Automatic video background replacement using shape-based probabilistic spatio-temporal object segmentation , 2007, 2007 6th International Conference on Information, Communications & Signal Processing.

[28]  Wonjun Kim,et al.  Illumination-Invariant Background Subtraction: Comparative Review, Models, and Prospects , 2017, IEEE Access.

[29]  Thierry Bouwmans,et al.  Traditional and recent approaches in background modeling for foreground detection: An overview , 2014, Comput. Sci. Rev..

[30]  Marc Van Droogenbroeck,et al.  Deep background subtraction with scene-specific convolutional neural networks , 2016, 2016 International Conference on Systems, Signals and Image Processing (IWSSIP).

[31]  Dani Lischinski,et al.  JumpCut , 2015, ACM Trans. Graph..

[32]  M. Ibrahim Sezan,et al.  Video background replacement without a blue screen , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[33]  Jian Sun,et al.  A global sampling method for alpha matting , 2011, CVPR 2011.

[34]  Kwan-Liu Ma,et al.  Parallel Volume Rendering Using Binary-Swap Image Composition , 2008, International Conference on Computer Graphics and Interactive Techniques.

[35]  Marc Van Droogenbroeck,et al.  ViBe: A Universal Background Subtraction Algorithm for Video Sequences , 2011, IEEE Transactions on Image Processing.

[36]  Rynson W. H. Lau,et al.  Robust Object Tracking via Locality Sensitive Histograms , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[37]  Yun Zhang,et al.  Efficient Video Cutout by Paint Selection , 2015, Journal of Computer Science and Technology.

[38]  Mingliang Chen,et al.  Spatiotemporal GMM for Background Subtraction with Superpixel Hierarchy , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Thierry Bouwmans,et al.  Foreground Detection Using the Choquet Integral , 2008, 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services.

[40]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[41]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Baoxin Li,et al.  Adaptive video background replacement , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[43]  Manuel Menezes de Oliveira Neto,et al.  Shared Sampling for Real‐Time Alpha Matting , 2010, Comput. Graph. Forum.