GPU based Parallel Optimization for Real Time Panoramic Video Stitching

Abstract Panoramic video is a sort of video recorded at the same point of view to record the full scene. With the development of video surveillance and the requirement for 3D converged video surveillance in smart cities, CPU and GPU are required to possess strong processing abilities to make panoramic video. The traditional panoramic products depend on post processing, which results in high power consumption, low stability and unsatisfying performance in real time. In order to solve these problems, we propose a real-time panoramic video stitching framework. The framework we propose mainly consists of three algorithms, L-ORB image feature extraction algorithm, feature point matching algorithm based on LSH and GPU parallel video stitching algorithm based on CUDA. The experiment results show that the algorithm mentioned can improve the performance in the stages of feature extraction of images stitching and matching, the running speed of which is 11.3 times than that of the traditional ORB algorithm and 641 times than that of the traditional SIFT algorithm. Based on analyzing the GPU resources occupancy rate of each resolution image stitching, we further propose a stream parallel strategy to maximize the utilization of GPU resources. Compared with the L-ORB algorithm, the efficiency of this strategy is improved by 1.6–2.5 times, and it can make full use of GPU resources. The performance of the system accomplished in the paper is 29.2 times than that of the former embedded one, while the power dissipation is reduced to 10 W.

[1]  Xue Li,et al.  Deep Attention-Based Spatially Recursive Networks for Fine-Grained Visual Recognition , 2019, IEEE Transactions on Cybernetics.

[2]  Ling Shao,et al.  Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval , 2018, IEEE Transactions on Image Processing.

[3]  Lin Wu,et al.  Robust Subspace Clustering for Multi-View Data by Exploiting Correlation Consensus , 2015, IEEE Transactions on Image Processing.

[4]  Jiri Matas,et al.  Matching with PROSAC - progressive sample consensus , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[5]  Zhang Qi,et al.  Global Topology Based Image Stitching Using Hierarchical Triangulation , 2012 .

[6]  Meng Wang,et al.  3-D PersonVLAD: Learning Deep Global Representations for Video-Based Person Reidentification , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Zhe Wang,et al.  Multi-Probe LSH: Efficient Indexing for High-Dimensional Similarity Search , 2007, VLDB.

[8]  Yunquan Zhang,et al.  HartSift: A High-Accuracy and Real-Time SIFT Based on GPU , 2017, 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS).

[9]  Shihwa Lee,et al.  Seamless and Fast Panoramic Image Stitching , 2012, 2012 IEEE International Conference on Consumer Electronics (ICCE).

[10]  Zhen Qin,et al.  Social Grouping for Multi-Target Tracking and Head Pose Estimation in Video , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[12]  Andrea Toma,et al.  Breaking the diffusion limit with super-hydrophobic delivery of molecules to plasmonic nanofocusing SERS structures , 2011 .

[13]  Jonathan Brandt,et al.  Transform coding for fast approximate nearest neighbor search in high dimensions , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[15]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[16]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[17]  Vincent Lepetit,et al.  BRIEF: Binary Robust Independent Elementary Features , 2010, ECCV.

[18]  Douglas Chai,et al.  Feature-based panoramic image stitching , 2016, 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV).

[19]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[20]  Changjiang Liu,et al.  Real-Time 3D Road Scene Based on Virtual-Real Fusion Method , 2015, IEEE Sensors Journal.

[21]  Tal Hassner,et al.  The CUDA LATCH Binary Descriptor: Because Sometimes Faster Means Better , 2016, ECCV Workshops.

[22]  Jonathan T. Barron,et al.  Jump: virtual reality video , 2016, ACM Trans. Graph..

[23]  Lin Wu,et al.  Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval , 2017, IEEE Transactions on Image Processing.

[24]  Vladimir Loncar,et al.  CUDA programs for solving the time-dependent dipolar Gross-Pitaevskii equation in an anisotropic trap , 2016, Comput. Phys. Commun..

[25]  Lin Wu,et al.  Multiview Spectral Clustering via Structured Low-Rank Matrix Factorization , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Jiang Jie,et al.  Multi-Scale Image Mosaic Using Features from Edge , 2011 .

[27]  Lin Wu,et al.  Deep adaptive feature embedding with local sample distributions for person re-identification , 2017, Pattern Recognit..

[28]  Shaozhang Niu,et al.  A novel panoramic image stitching algorithm based on ORB , 2017, 2017 International Conference on Applied System Innovation (ICASI).

[29]  Yingen Xiong,et al.  Fast panorama stitching for high-quality panoramic images on mobile phones , 2010, IEEE Transactions on Consumer Electronics.

[30]  Nicole Immorlica,et al.  Locality-sensitive hashing scheme based on p-stable distributions , 2004, SCG '04.

[31]  Zhang Da-kun,et al.  A Distributed Parallel Algorithm for SIFT Feature Extraction , 2012 .

[32]  Lin Wu,et al.  Unsupervised Metric Fusion Over Multiview Data by Graph Random Walk-Based Cross-View Diffusion , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Lin Wu,et al.  What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification , 2017, Pattern Recognit..

[34]  Yingjie Xia,et al.  Towards improving quality of video-based vehicle counting method for traffic flow estimation , 2016, Signal Process..

[35]  Li Tao,et al.  Real-Time Panoramic Video Stitching Based on GPU Acceleration Using Local ORB Feature Extraction , 2017 .

[36]  Xiaogang Wang,et al.  Intelligent multi-camera video surveillance: A review , 2013, Pattern Recognit. Lett..

[37]  Xiyang Zhi,et al.  Realization of CUDA-based real-time registration and target localization for high-resolution video images , 2016, Journal of Real-Time Image Processing.

[38]  W. Eric L. Grimson,et al.  Correspondence-free multi-camera activity analysis and scene modeling , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[40]  Michael Riegler,et al.  Tiling in Interactive Panoramic Video: Approaches and Evaluation , 2016, IEEE Transactions on Multimedia.

[41]  Lin Wu,et al.  Where-and-When to Look: Deep Siamese Attention Networks for Video-Based Person Re-Identification , 2018, IEEE Transactions on Multimedia.