Survey of Compressed Domain Video Summarization Techniques

Video summarization is the method of extracting key frames or clips from a video to generate a synopsis of the content of the video. Generally, video is compressed before storing or transmitting it in most of the practical applications. Traditional techniques require the videos to be decoded to summarize them, which is a tedious job. Instead, compressed domain video processing can be used for summarizing videos by partially decoding them. A classification and analysis of various summarization techniques are presented in this article with special focus on compressed domain techniques along with a discussion on machine-learning-based techniques that can be applied to summarize the videos.

[1]  Banshidhar Majhi,et al.  A multi-view video synopsis framework , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[2]  Michael Lam,et al.  Unsupervised Video Summarization with Adversarial LSTM Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Bohyung Han,et al.  Personalized video summarization with human in the loop , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[4]  James M. Rehg,et al.  Gaze-enabled egocentric video summarization via constrained submodular maximization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Ke Zhang,et al.  Summary Transfer: Exemplar-Based Subset Selection for Video Summarization , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Yale Song,et al.  Action Recognition by Hierarchical Sequence Summarization , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Junseok Kwon,et al.  A unified framework for event summarization and rare event detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Anastasios Tefas,et al.  Summarization of human activity videos via low-rank approximation , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Eric P. Xing,et al.  Unsupervised Object-Level Video Summarization with Online Motion Auto-Encoder , 2018, Pattern Recognit. Lett..

[10]  Debi Prosad Dogra,et al.  Summarization of Neonatal Video EEG for Seizure and Artifact Detection , 2011, 2011 Third National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics.

[11]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[12]  Yang Yi,et al.  Key frame extraction based on visual attention model , 2012, J. Vis. Commun. Image Represent..

[13]  Klara Nahrstedt,et al.  Multicamera Summarization of Rehabilitation Sessions in Home Environment , 2017, ACM Multimedia.

[14]  Amit K. Roy-Chowdhury,et al.  Sparse modeling for topic-oriented video summarization , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Sebastian Boring,et al.  #EpicPlay: crowd-sourcing sports video highlights , 2012, CHI.

[16]  Yuzhen Niu,et al.  Video summagator: an interface for video summarization and navigation , 2012, CHI.

[17]  Eugenio Culurciello,et al.  Convolutional Clustering for Unsupervised Learning , 2015, ArXiv.

[18]  Aboul Ella Hassanien,et al.  SVM-based soccer video summarization system , 2011, 2011 Third World Congress on Nature and Biologically Inspired Computing.

[19]  Muhammad Shakir,et al.  Video Summarization: Techniques and Classification , 2012, ICCVG.

[20]  Farzad Zargari,et al.  Compressed Domain Video Abstraction Based on I-Frame of HEVC Coded Videos , 2018, Circuits, Systems, and Signal Processing.

[21]  Jurandy Almeida,et al.  VISON: VIdeo Summarization for ONline applications , 2012, Pattern Recognit. Lett..

[22]  Yanan Zhao,et al.  Performance evaluation of H.265/MPEG-HEVC encoders for 4K video sequences , 2014, Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific.

[23]  José María Martínez Sanchez,et al.  A Framework for Scalable Summarization of Video , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Marco Pellegrini,et al.  VISTO: visual storyboard for web video browsing , 2007, CIVR '07.

[25]  Bin Zhao,et al.  HSA-RNN: Hierarchical Structure-Adaptive RNN for Video Summarization , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Sung Wook Baik,et al.  Feature aggregation based visual attention model for video summarization , 2014, Comput. Electr. Eng..

[27]  Ahmed K. Elmagarmid,et al.  InsightVideo: toward hierarchical video content organization for efficient browsing, summarization and retrieval , 2005, IEEE Transactions on Multimedia.

[28]  Ruimin Hu,et al.  Fast Synopsis for Moving Objects Using Compressed Video , 2014, IEEE Signal Processing Letters.

[29]  Faouzi Kossentini,et al.  Video object summarization in the MPEG-4 compressed domain , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[30]  Mariusz Duplaga,et al.  Algorithm for Video Summarization of Bronchoscopy Procedures , 2011, Biomedical engineering online.

[31]  Silvio Jamil Ferzoli Guimarães,et al.  Graph-Based Hierarchical Video Summarization Using Global Descriptors , 2014, 2014 IEEE 26th International Conference on Tools with Artificial Intelligence.

[32]  Weisi Lin,et al.  Scene-Based Movie Summarization Via Role-Community Networks , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  George Ghinea,et al.  What do you wish to see? A summarization system for movies based on user preferences , 2015, Inf. Process. Manag..

[34]  Arnaldo de Albuquerque Araújo,et al.  VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method , 2011, Pattern Recognit. Lett..

[35]  Chong-Wah Ngo,et al.  Video summarization and scene detection by graph modeling , 2005, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  John Adcock,et al.  Video summarization preserving dynamic content , 2007, TVS '07.

[37]  Noboru Babaguchi,et al.  Video Summarization for Large Sports Video Archives , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[38]  B. S. Manjunath,et al.  Multicamera Video Summarization from Optimal Reconstruction , 2010, ACCV Workshops.

[39]  Dim P. Papadopoulos,et al.  Video Summarization Using a Self-Growing and Self-Organized Neural Gas Network , 2011, MIRAGE.

[40]  Javier Iparraguirre,et al.  Speeded-Up Video Summarization Based on Local Features , 2013, 2013 IEEE International Symposium on Multimedia.

[41]  Luc Van Gool,et al.  Video summarization by learning submodular mixtures of objectives , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  David Menotti,et al.  GPUs and Multicore CPUs Implementations of a Static Video Summarization , 2014, CIARP.

[43]  Youness Tabii,et al.  Video Summarization: Techniques and Applications , 2015 .

[44]  Jiebo Luo,et al.  Towards Scalable Summarization of Consumer Videos Via Sparse Dictionary Selection , 2012, IEEE Transactions on Multimedia.

[45]  Mark S. Drew,et al.  Clustering of compressed illumination-invariant chromaticity signatures for efficient video summarization , 2003, Image Vis. Comput..

[46]  Luca Viganò,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2015, IWSEC 2015.

[47]  Ming-Hsuan Yang,et al.  Learning Video-Story Composition via Recurrent Neural Network , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[48]  Hélio Pedrini,et al.  VISCOM: A robust video summarization approach using color co-occurrence matrices , 2016, Multimedia Tools and Applications.

[49]  Yong Jae Lee,et al.  Discovering important people and objects for egocentric video summarization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Miska M. Hannuksela,et al.  H.264/AVC in wireless environments , 2003, IEEE Trans. Circuits Syst. Video Technol..

[51]  B. S. Manjunath,et al.  Multicamera video summarization and anomaly detection from activity motifs , 2014, TOSN.

[52]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[53]  Wei-Hao Lin,et al.  News video classification using SVM-based multimodal classifiers and combination strategies , 2002, MULTIMEDIA '02.

[54]  Ahmet M. Kondoz,et al.  Flexible generation of video summaries from layered video bit-streams , 2008, 2008 15th IEEE International Conference on Image Processing.

[55]  Wei-Ta Chu,et al.  A User Experience Model for Home Video Summarization , 2009, MMM.

[56]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[57]  Ananda S. Chowdhury,et al.  Scalable Video Summarization Using Skeleton Graph and Random Walk , 2014, 2014 22nd International Conference on Pattern Recognition.

[58]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[59]  Ahmet M. Kondoz,et al.  Dynamic layout of visual summaries for scalable video , 2008, 2008 International Workshop on Content-Based Multimedia Indexing.

[60]  Wei Jiang,et al.  Memorable and rich video summarization , 2017, J. Vis. Commun. Image Represent..

[61]  Chia-han Lee,et al.  Low complexity on-line video summarization with Gaussian mixture model based clustering , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[62]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[63]  Markus Schedl,et al.  The effect of different video summarization models on the quality of video recommendation based on low-level visual features , 2017, CBMI.

[64]  David Menotti,et al.  Speeding up a Video Summarization Approach Using GPUs and Multicore CPUs , 2014, ICCS.

[65]  Yujie Li,et al.  Extracting key frames from first-person videos in the common space of multiple sensors , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[66]  Tao Mei,et al.  Video Summarization by Learning Deep Side Semantic Embedding , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[67]  Theodora A. Varvarigou,et al.  Video summarization guiding evaluative rectification for industrial activity recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[68]  David Dagan Feng,et al.  Real-Time Storyboard Generation for H.264/AVC Compressed Videos , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[69]  Xue Yang,et al.  A novel video summarization algorithm , 2011, 2011 International Conference on Multimedia Technology.

[70]  Xi Wang,et al.  Fast Summarization of User-Generated Videos: Exploiting Semantic, Emotional, and Quality Clues , 2016, IEEE MultiMedia.

[71]  Fumin Shen,et al.  Spatial and temporal scoring for egocentric video summarization , 2016, Neurocomputing.

[72]  Adriano M. Pereira,et al.  A video summarization approach based on the emulation of bottom-up mechanisms of visual attention , 2017, Journal of Intelligent Information Systems.

[73]  Ying Zhang,et al.  Dynamic Multi-video Summarization of Sensor-Rich Videos in Geo-Space , 2013, MMM.

[74]  Shohreh Kasaei,et al.  Event Detection and Summarization in Soccer Videos Using Bayesian Network and Copula , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[75]  Priyanka Sharma,et al.  KSUMM: A Compressed Domain Technique for Video Summarization Using Partial Decoding of Videos , 2018, Communications in Computer and Information Science.

[76]  Jianping Fan,et al.  Exploring video content structure for hierarchical summarization , 2004, Multimedia Systems.

[77]  Regunathan Radhakrishnan,et al.  Video Summarization Using Mpeg-7 Motion Activity and Audio Descriptors , 2003 .

[78]  Yung-Yu Chuang,et al.  NTU TRECVID-2007 fast rushes summarization system , 2007, TVS '07.

[79]  Shih-Fu Chang,et al.  Real-time personalized sports video filtering and summarization , 2001, MULTIMEDIA '01.

[80]  Emiru Tsunoo,et al.  Hierarchical Recurrent Neural Network for Story Segmentation , 2017, INTERSPEECH.

[81]  Mubarak Shah,et al.  A 3-dimensional sift descriptor and its application to action recognition , 2007, ACM Multimedia.

[82]  Hélio Pedrini,et al.  A Video Summarization Method Based on Spectral Clustering , 2013, CIARP.

[83]  Christophe De Vleeschouwer,et al.  Personalized Summarization of Broadcasted Soccer Videos with Adaptive Fast-Forwarding , 2013, INTETAIN.

[84]  Chen Li,et al.  Automatic Movie Summarization Based on the Visual-Audio Features , 2014, 2014 IEEE 17th International Conference on Computational Science and Engineering.

[85]  Kenny Davila,et al.  Whiteboard Video Summarization via Spatio-Temporal Conflict Minimization , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[86]  Jan-Michael Frahm,et al.  Hysteroscopy video summarization and browsing by estimating the physician's attention on video segments , 2012, Medical Image Anal..

[87]  José María Martínez Sanchez,et al.  An integrated approach to summarization and adaptation using H.264/MPEG-4 SVC , 2009, Signal Process. Image Commun..

[88]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[89]  Wesley De Neve,et al.  A compressed-domain approach for shot boundary detection on H.264/AVC bit streams , 2008, Signal Process. Image Commun..

[90]  Engin Mendi,et al.  Summarization of MPEG Compressed Video Sequences , 2011 .

[91]  Jurandy Almeida,et al.  Online video summarization on compressed domain , 2013, J. Vis. Commun. Image Represent..

[92]  Kristen Grauman,et al.  Story-Driven Summarization for Egocentric Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[93]  Jung Hwan Oh,et al.  Video Abstraction , 2009, Encyclopedia of Database Systems.

[94]  Hamid Reza Pourreza,et al.  Flexible soccer video summarization in compressed domain , 2013, ICCKE 2013.

[95]  Wolfgang Effelsberg,et al.  Video abstracting , 1997, CACM.

[96]  Ke Zhang,et al.  Video Summarization with Long Short-Term Memory , 2016, ECCV.

[97]  Cordelia Schmid,et al.  A Spatio-Temporal Descriptor Based on 3D-Gradients , 2008, BMVC.

[98]  Bernard Mérialdo,et al.  Multi-video summarization based on Video-MMR , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[99]  Ling Shao,et al.  Deep Learning for Pattern Recognition , 2019, Pattern Recognit. Lett..

[100]  Jiasong Zhu,et al.  Learning deep semantic attributes for user video summarization , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[101]  Jurandy Almeida,et al.  Rapid Video Summarization on Compressed Video , 2010, 2010 IEEE International Symposium on Multimedia.

[102]  Mohan S. Kankanhalli,et al.  Compressed Domain Summarization of Digital Video , 2001, IEEE Pacific Rim Conference on Multimedia.

[103]  R. Venkatesh Babu,et al.  A survey on compressed domain video analysis techniques , 2014, Multimedia Tools and Applications.

[104]  Harish Katti,et al.  Affective Video Summarization and Story Board Generation Using Pupillary Dilation and Eye Gaze , 2011, 2011 IEEE International Symposium on Multimedia.

[105]  Tao Mei,et al.  Near-lossless semantic video summarization and its applications to video analysis , 2013, TOMCCAP.

[106]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[107]  Shi-Min Hu,et al.  Visual storylines: Semantic visualization of movie sequence , 2012, Comput. Graph..

[108]  Cordelia Schmid,et al.  Action Recognition with Improved Trajectories , 2013, 2013 IEEE International Conference on Computer Vision.

[109]  Shih-Fu Chang,et al.  Survey of compressed-domain features used in audio-visual indexing and analysis , 2003, J. Vis. Commun. Image Represent..

[110]  Sung Wook Baik,et al.  Visual saliency models for summarization of diagnostic hysteroscopy videos in healthcare systems , 2016, SpringerPlus.

[111]  Zhi-Hua Zhou,et al.  Multi-View Video Summarization , 2010, IEEE Transactions on Multimedia.

[112]  Shih-Fu Chang,et al.  Tools for compressed-domain video indexing and editing , 1996, Electronic Imaging.

[113]  Kiyoharu Aizawa,et al.  Evaluation of video summarization for a large number of cameras in ubiquitous home , 2005, MULTIMEDIA '05.

[114]  Ananda S. Chowdhury,et al.  Video Key Frame Extraction through Canonical Correlation Analysis and Graph Modularity , 2013, PReMI.

[115]  Nagia M. Ghanem,et al.  VSCAN: An Enhanced Video Summarization Using Density-Based Spatial Clustering , 2013, ICIAP.

[116]  Patrick Gros,et al.  Constraint Satisfaction Programming for Video Summarization , 2013, 2013 IEEE International Symposium on Multimedia.

[117]  V. Ghini,et al.  An audio-video summarization scheme based on audio and video analysis , 2006, CCNC 2006. 2006 3rd IEEE Consumer Communications and Networking Conference, 2006..

[118]  Ananda S. Chowdhury,et al.  Video key frame extraction through dynamic Delaunay clustering with a structural constraint , 2013, J. Vis. Commun. Image Represent..

[119]  Jenq-Neng Hwang,et al.  Object-based video abstraction for video surveillance systems , 2002, IEEE Trans. Circuits Syst. Video Technol..

[120]  Vlad I. Morariu,et al.  Summarizing While Recording: Context-Based Highlight Detection for Egocentric Videos , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[121]  Hossein Mobahi,et al.  Deep learning from temporal coherence in video , 2009, ICML '09.

[122]  Chong-Wah Ngo,et al.  Summarizing Rushes Videos by Motion, Object, and Event Understanding , 2012, IEEE Transactions on Multimedia.

[123]  Ying Zhang,et al.  Aesthetics-Guided Summarization from Multiple User Generated Videos , 2015, ACM Trans. Multim. Comput. Commun. Appl..

[124]  George Ghinea,et al.  A novel user-centered design for personalized video summarization , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[125]  Fumiko Satoh,et al.  Learning personalized video highlights from detailed MPEG-7 metadata , 2002, Proceedings. International Conference on Image Processing.

[126]  Klaus Schöffmann,et al.  Fast Segmentation of H.264/AVC Bitstreams for On-Demand Video Summarization , 2008, MMM.

[127]  José María Martínez Sanchez,et al.  Scalable Comic-Like Video Summaries and Layout Disturbance , 2012, IEEE Transactions on Multimedia.

[128]  Robert Laganière,et al.  Video summarization of surveillance cameras , 2016, 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[129]  Wei-Ta Chu,et al.  Editing by Viewing: Automatic Home Video Summarization by Viewing Behavior Analysis , 2011, IEEE Transactions on Multimedia.

[130]  Mohan S. Kankanhalli,et al.  Semantic video summarization in compressed domain MPEG video , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[131]  Masaru Sugano,et al.  Generic Summarization Technology for Consumer Video , 2004, PCM.

[132]  Joo-Hwee Lim,et al.  Summarization of Egocentric Videos: A Comprehensive Survey , 2017, IEEE Transactions on Human-Machine Systems.

[133]  Jhing-Fa Wang,et al.  A Novel Video Summarization Based on Mining the Story-Structure and Semantic Relations Among Concept Entities , 2009, IEEE Transactions on Multimedia.

[134]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[135]  Chih-Jen Lin,et al.  Large-Scale Video Summarization Using Web-Image Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[136]  Uma Mudenagudi,et al.  Multilevel Framework for Summarization of Surveillance Videos , 2014, 2014 Fifth International Conference on Signal and Image Processing.

[137]  Lei Sun,et al.  The dynamic VideoBook: A hierarchical summarization for surveillance video , 2013, 2013 IEEE International Conference on Image Processing.

[138]  N. Nikolaidis,et al.  Video shot detection and condensed representation. a review , 2006, IEEE Signal Processing Magazine.

[139]  Tijo Thomas,et al.  Video summarization by clustering using euclidean distance , 2011, 2011 International Conference on Signal Processing, Communication, Computing and Networking Technologies.

[140]  Xiao Qin,et al.  Frog: A Framework for Context-Based File Systems , 2015, TOS.

[141]  Chu Luo Video Summarization for Object Tracking in the Internet of Things , 2014, 2014 Eighth International Conference on Next Generation Mobile Apps, Services and Technologies.

[142]  Harry W. Agius,et al.  Video summarisation: A conceptual framework and survey of the state of the art , 2008, J. Vis. Commun. Image Represent..

[143]  Jinchang Ren,et al.  Activity-driven content adaptation for effective video summarization , 2010, J. Vis. Commun. Image Represent..

[144]  Nicu Sebe,et al.  Exploiting facial expressions for affective video summarisation , 2009, CIVR '09.

[145]  Shaohui Mei,et al.  L2,0 constrained sparse dictionary selection for video summarization , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[146]  Mohan S. Kankanhalli,et al.  Video Summarization Using R-Sequences , 2000, Real Time Imaging.

[147]  Tao Mei,et al.  Detecting shot boundary with sparse coding for video summarization , 2017, Neurocomputing.