Superpixel-Based Causal Multisensor Video Fusion

Video surveillance systems have become extremely important recently. It has been observed that information extracted from a single spectrum video is often insufficient in adverse conditions like low illumination, shadowing, smoke, dust, unstable background, and camouflage. Real-time video processing systems also need to be very fast where future frames are often unavailable at the time of processing the current frame. In this paper, we propose a superpixel-based causal multisensor video fusion algorithm suitable for real-time surveillance tasks. We develop new superpixel level spatial and temporal saliency models. Novel superpixel level multiple fusion rules are also designed to obtain the fused output. Comprehensive comparisons with several existing works clearly indicate the benefit of our solution.

[1]  Xiaoyi Jiang,et al.  Causal Video Segmentation Using Superseeds and Graph Matching , 2015, GbRPR.

[2]  James W. Davis,et al.  Background-subtraction using contour-based fusion of thermal and visible imagery , 2007, Comput. Vis. Image Underst..

[3]  Narendra Ahuja,et al.  Fusion of frequency and spatial domain information for motion analysis , 2004, ICPR 2004.

[4]  Michael A. Goodrich,et al.  Fused visible and infrared video for use in Wilderness Search and Rescue , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[5]  Amit K. Roy-Chowdhury,et al.  Re-Identification in the Function Space of Feature Warps , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Jean-Baptiste Thomas,et al.  Background subtraction with multispectral video sequences , 2014 .

[7]  Sridha Sridharan,et al.  Multi-spectral fusion for surveillance systems , 2008, Comput. Electr. Eng..

[8]  Jan Noyes,et al.  Task-based scanpath assessment of multi-sensor video fusion in complex scenarios , 2010, Inf. Fusion.

[9]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Timothy F. Cootes,et al.  Dynamic image fusion performance evaluation , 2007, 2007 10th International Conference on Information Fusion.

[11]  Henk J. A. M. Heijmans,et al.  A new quality metric for image fusion , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[12]  Chong-Wah Ngo,et al.  Concept-Driven Multi-Modality Fusion for Video Search , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Long Wang,et al.  A novel video fusion framework using surfacelet transform , 2012 .

[14]  Shuai Ding,et al.  Image Fusion Algorithm Based on Nonsubsampled Contourlet Transform , 2013 .

[15]  Liang Xu,et al.  Infrared-visible video fusion based on motion-compensated wavelet transforms , 2015, IET Image Process..

[16]  John See,et al.  Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition , 2015, PloS one.

[17]  Youjie Zhou,et al.  Multiscale Superpixels and Supervoxels Based on Hierarchical Edge-Weighted Centroidal Voronoi Tessellation , 2015, IEEE Transactions on Image Processing.

[18]  Xuelong Li,et al.  Lazy Random Walks for Superpixel Segmentation , 2014, IEEE Transactions on Image Processing.

[19]  Larry S. Davis,et al.  W4: Real-Time Surveillance of People and Their Activities , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  M. N. Shanmukha Swamy,et al.  Camouflaged target detection using real-time video fusion algorithm based on multi-scale transforms , 2014, 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE).

[21]  Oliver Rockinger,et al.  Image sequence fusion using a shift-invariant wavelet transform , 1997, Proceedings of International Conference on Image Processing.

[22]  Xiang Zhang,et al.  Superpixel-Based Spatiotemporal Saliency Detection , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Zhiwen Yu,et al.  A Bayesian Model for Crowd Escape Behavior Detection , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Ankur P. Parikh,et al.  Algorithms for Graph Similarity and Subgraph Matching , 2011 .

[25]  Sridha Sridharan,et al.  An Efficient and Robust System for Multiperson Event Detection in Real-World Indoor Surveillance Scenes , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[26]  Ding Liu,et al.  Image Fusion Using Higher Order Singular Value Decomposition , 2012, IEEE Transactions on Image Processing.

[27]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Xi Chen,et al.  A saliency detection model using aggregation degree of color and texture , 2015, Signal Process. Image Commun..

[29]  Wonjun Kim,et al.  Video Saliency Detection Using Contrast of Spatiotemporal Directional Coherence , 2014, IEEE Signal Processing Letters.

[30]  Long Wang,et al.  Multisensor video fusion based on higher order singular value decomposition , 2015, Inf. Fusion.

[31]  Yao Zhao,et al.  Frame Fusion for Video Copy Detection , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  Alessia Saggese,et al.  Real-Time Fire Detection for Video-Surveillance Applications Using a Combination of Experts Based on Color, Shape, and Motion , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Guillaume-Alexandre Bilodeau,et al.  An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications , 2012, Comput. Vis. Image Underst..

[34]  Xin He,et al.  Saliency detection based on integrated features , 2014, Neurocomputing.

[35]  Leonard McMillan,et al.  Multispectral Bilateral Video Fusion , 2007, IEEE Transactions on Image Processing.

[36]  Long Wang,et al.  Multisensor video fusion based on spatial-temporal salience detection , 2013, Signal Process..