Methods and Challenges in Shot Boundary Detection: A Review

The recent increase in the number of videos available in cyberspace is due to the availability of multimedia devices, highly developed communication technologies, and low-cost storage devices. These videos are simply stored in databases through text annotation. Content-based video browsing and retrieval are inefficient due to the method used to store videos in databases. Video databases are large in size and contain voluminous information, and these characteristics emphasize the need for automated video structure analyses. Shot boundary detection (SBD) is considered a substantial process of video browsing and retrieval. SBD aims to detect transition and their boundaries between consecutive shots; hence, shots with rich information are used in the content-based video indexing and retrieval. This paper presents a review of an extensive set for SBD approaches and their development. The advantages and disadvantages of each approach are comprehensively explored. The developed algorithms are discussed, and challenges and recommendations are presented.

[1]  Ramesh C. Jain,et al.  Knowledge-guided parsing in video databases , 1993, Electronic Imaging.

[2]  Akio Nagasaka,et al.  Automatic Video Indexing and Full-Video Search for Object Appearances , 1991, VDB.

[3]  Awais Adnan,et al.  Shot Boundary Detection Using Sorted Color Histogram Polynomial Curve , 2013 .

[4]  Duy-Dinh Le,et al.  A text segmentation based approach to video shot boundary detection , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[5]  Dong-Sik Jang,et al.  Gradual shot boundary detection using localized edge blocks , 2006, Multimedia Tools and Applications.

[6]  Ting Liu,et al.  Video Segmentation via Temporal Pattern Classification , 2007, IEEE Transactions on Multimedia.

[7]  David Pye,et al.  Audio-visual segmentation for content-based retrieval , 1998, ICSLP.

[8]  Kishor Bhoyar,et al.  COLOR IMAGE SEGMENTATION BASED ON JND COLOR HISTOGRAM , 2010 .

[9]  Kazimierz Choros Reduction of Faulty Detected Shot Cuts and Cross Dissolve Effects in Video Segmentation Process of Different Categories of Digital Videos , 2011, Trans. Comput. Collect. Intell..

[10]  P. Kanungo,et al.  A motion and illumination resilient framework for automatic shot boundary detection , 2017, Signal, Image and Video Processing.

[11]  Jeongwoo Son,et al.  Study on the effect of frame size and color histogram bins on the shot boundary detection performance , 2016, 2016 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia).

[12]  John S. Boreczky,et al.  Comparison of video shot boundary detection techniques , 1996, J. Electronic Imaging.

[13]  Seong-Whan Lee,et al.  Automatic video parsing using shot boundary detection and camera operation analysis , 2001, Pattern Recognit..

[14]  Huadong Ma,et al.  To accelerate shot boundary detection by reducing detection region and scope , 2012, Multimedia Tools and Applications.

[15]  Osama Masoud,et al.  Real-Time Detection of Camera Tampering , 2006, 2006 IEEE International Conference on Video and Signal Based Surveillance.

[16]  Sudipta Roy,et al.  A Genetic Algorithm and Fuzzy Logic Approach for Video Shot Boundary Detection , 2016, Comput. Intell. Neurosci..

[17]  Alan Hanjalic,et al.  Shot-boundary detection: unraveled and resolved? , 2002, IEEE Trans. Circuits Syst. Video Technol..

[18]  Syed Abdul Rahman Al-Haddad,et al.  Fast Recursive Computation of Krawtchouk Polynomials , 2018, Journal of Mathematical Imaging and Vision.

[19]  Rainer Lienhart,et al.  Reliable Transition Detection in Videos: A Survey and Practitioner's Guide , 2001, Int. J. Image Graph..

[20]  S. Domnic,et al.  Walsh–Hadamard Transform Kernel-Based Feature Vector for Shot Boundary Detection , 2014, IEEE Transactions on Image Processing.

[21]  Ramin Zabih,et al.  A feature-based algorithm for detecting and classifying production effects , 1999, Multimedia Systems.

[22]  Shuwen Zhang,et al.  Shot Boundary Detection based on Multilevel Difference of Colour Histograms , 2016, 2016 First International Conference on Multimedia and Image Processing (ICMIP).

[23]  Nitin J. Janwe,et al.  Video shot boundary detection based on JND color histogram , 2013, 2013 IEEE Second International Conference on Image Information Processing (ICIIP-2013).

[24]  Anni Cai,et al.  A robust shot transition detection method based on support vector machine in compressed domain , 2007, Pattern Recognit. Lett..

[25]  Wei Xiong,et al.  Automatic video data structuring through shot partitioning and key-frame computing , 1997, Machine Vision and Applications.

[26]  Shiguo Lian,et al.  Automatic video temporal segmentation based on multiple features , 2011, Soft Comput..

[27]  Qi Tian,et al.  A unified framework for semantic shot classification in sports video , 2002, IEEE Transactions on Multimedia.

[28]  Syed Abdul Rahman Al-Haddad,et al.  Signal compression and enhancement using a new orthogonal-polynomial-based discrete transform , 2017, IET Signal Process..

[29]  Frédéric Dufaux,et al.  Efficient, robust, and fast global motion estimation for video coding , 2000, IEEE Trans. Image Process..

[30]  Joni-Kristian Kämäräinen,et al.  Video Shot Boundary Detection using Visual Bag-of-Words , 2013, VISAPP.

[31]  Ramin Zabih,et al.  A feature-based algorithm for detecting and classifying scene breaks , 1995, MULTIMEDIA '95.

[32]  Toby P. Breckon,et al.  Fundamentals of Digital Image Processing: A Practical Approach with Examples in Matlab , 2011 .

[33]  Majid Mirmehdi,et al.  Video cut detection using frequency domain correlation , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[34]  Boon-Lock Yeo,et al.  Rapid scene analysis on compressed video , 1995, IEEE Trans. Circuits Syst. Video Technol..

[35]  Zhao Huan,et al.  Shot Boundary Detection Based on Mutual Information and Canny Edge Detector , 2008, 2008 International Conference on Computer Science and Software Engineering.

[36]  Tudor Barbu,et al.  Novel automatic video cut detection technique using Gabor filtering , 2009, Comput. Electr. Eng..

[37]  Jeho Nam,et al.  Combined audio and visual streams analysis for video sequence segmentation , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[38]  Jie Zheng,et al.  An efficient algorithm for video shot boundary detection , 2004, Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004..

[39]  A. Murat Tekalp,et al.  Efficient Filtering and Clustering Methods for Temporal Video Segmentation and Visual Summarization , 1998, J. Vis. Commun. Image Represent..

[40]  Siddhartha Bhattacharyya,et al.  Hybrid soft computing approaches to content based video retrieval: A brief review , 2016, Appl. Soft Comput..

[41]  Abdul Hameed,et al.  A novel framework of shot boundary detection for uncompressed videos , 2009, 2009 International Conference on Emerging Technologies.

[42]  King Ngi Ngan,et al.  High accuracy flashlight scene determination for shot boundary detection , 2003, Signal Process. Image Commun..

[43]  Xueming Qian,et al.  Effective Fades and Flashlight Detection Based on Accumulating Histogram Difference , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[44]  Bhushan D. Patil,et al.  Shot Boundary Detection using Radon Projection Method , 2013 .

[45]  Jin Liu,et al.  An adaptive video shot segmentation scheme based on dual-detection model , 2013, Neurocomputing.

[46]  Siddhartha Bhattacharyya,et al.  Detection of Gradual Transition in Videos: Approaches and Applications , 2017 .

[47]  Bede Liu,et al.  Temporal segmentation of video using frame and histogram space , 2000, IEEE Transactions on Multimedia.

[48]  Li Li,et al.  A Survey on Visual Content-Based Video Indexing and Retrieval , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[49]  Ruxandra Tapu,et al.  Video Segmentation and Structuring for Indexing Applications , 2011, Int. J. Multim. Data Eng. Manag..

[50]  Serkan Kiranyaz,et al.  A perceptual scheme for fully automatic video shot boundary detection , 2014, Signal Process. Image Commun..

[51]  Paul Over,et al.  TRECVID 2005 - An Overview , 2005, TRECVID.

[52]  Paul England,et al.  Comparison of automatic video segmentation algorithms , 1996, Other Conferences.

[53]  Patrick Bouthemy,et al.  A unified approach to shot change detection and camera motion characterization , 1999, IEEE Trans. Circuits Syst. Video Technol..

[54]  Malay Kumar Kundu,et al.  Video shot boundary detection using multiscale geometric analysis of nsct and least squares support vector machine , 2018, Multimedia Tools and Applications.

[55]  Mahmood Fathy,et al.  Video Shot Boundary Detection Using Generalized Eigenvalue Decomposition and Gaussian Transition Detection , 2009, Comput. Informatics.

[56]  Chong-Wah Ngo,et al.  Video partitioning by temporal slice coherency , 2001, IEEE Trans. Circuits Syst. Video Technol..

[57]  Shohreh Kasaei,et al.  Event Detection and Summarization in Soccer Videos Using Bayesian Network and Copula , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[58]  Tian-Yuan Shih THE REVERSIBILITY OF 6 GEOMETRIC COLOR SPACES , 1995 .

[59]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[60]  T. Vlachos Cut detection in video sequences using phase correlation , 2000, IEEE Signal Processing Letters.

[61]  Sarah V. Porter,et al.  Video Segmentation and Indexing using Motion Estimation , 2004 .

[62]  Alexander Wong,et al.  Shot Boundary Detection Using Genetic Algorithm Optimization , 2011, 2011 IEEE International Symposium on Multimedia.

[63]  José Manuel Menéndez,et al.  A unified model for techniques on video-shot transition detection , 2005, IEEE Transactions on Multimedia.

[64]  Kazimierz Choros,et al.  Improved Video Scene Detection Using Player Detection Methods in Temporally Aggregated TV Sports News , 2014, ICCCI.

[65]  Silvio Jamil Ferzoli Guimarães,et al.  Gradual transition detection based on bipartite graph matching approach , 2009, 2009 IEEE International Workshop on Multimedia Signal Processing.

[66]  Ioannis Pitas,et al.  Information theory-based shot cut/fade detection and video summarization , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[67]  Xiaokang Yang,et al.  CNN-based shot boundary detection and video annotation , 2015, 2015 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting.

[68]  Gholamreza Haffari,et al.  Data-dependent dissimilarity measure: an effective alternative to geometric distance measures , 2017, Knowledge and Information Systems.

[69]  Majid Mirmehdi,et al.  Temporal video segmentation and classification of edit effects , 2003, Image Vis. Comput..

[70]  Nobuyuki Yagi,et al.  Shot Boundary Detection at TRECVID 2007 , 2007, TRECVID.

[71]  Abd. Rahman Ramli,et al.  Image edge detection operators based on orthogonal polynomials , 2017 .

[72]  Hamidreza Rashidy Kanan,et al.  AVCD-FRA: A novel solution to automatic video cut detection using fuzzy-rule-based approach , 2013, Comput. Vis. Image Underst..

[73]  Behzad Shahraray,et al.  Scene change detection and content-based sampling of video sequences , 1995, Electronic Imaging.

[74]  Suchada Sitjongsataporn,et al.  Multi-Modal Visual Features-Based Video Shot Boundary Detection , 2017, IEEE Access.

[75]  Y.-N. Li,et al.  Fast video shot boundary detection framework employing pre-processing techniques , 2009, IET Image Process..

[76]  Nicole Vincent,et al.  Efficient and robust shot change detection , 2007, Journal of Real-Time Image Processing.

[77]  Syed Abdul Rahman Al-Haddad,et al.  Low-Distortion MMSE Speech Enhancement Estimator Based on Laplacian Prior , 2017, IEEE Access.

[78]  Jie Zhao,et al.  Effective Dissolve Detection Based on Accumulating Histogram Difference and the Support Point , 2010, 2010 First International Conference on Pervasive Computing, Signal Processing and Applications.

[79]  Liang-Hua Chen,et al.  A Supervised Learning Approach to Flashlight Detection , 2017, Cybern. Syst..

[80]  Fiaz Hussain,et al.  Video Indexing: A Survey , 2014 .

[81]  Sudipta Roy,et al.  Video shot boundary detection: A review , 2015, 2015 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT).

[82]  Bhabatosh Chanda,et al.  A Model-Based Shot Boundary Detection Technique Using Frame Transition Parameters , 2012, IEEE Transactions on Multimedia.

[83]  Shih-Fu Chang,et al.  Scene change detection in an MPEG-compressed video sequence , 1995, Electronic Imaging.

[84]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[85]  Ullas Gargi,et al.  Performance characterization of video-shot-change detection methods , 2000, IEEE Trans. Circuits Syst. Video Technol..

[86]  John M. Gauch,et al.  Story tracking in video news broadcasts , 2004 .

[87]  Eric Bruno,et al.  Video shot detection based on linear prediction of motion , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[88]  Rainer Lienhart,et al.  Comparison of automatic shot boundary detection algorithms , 1998, Electronic Imaging.

[89]  Adnan M. Alattar Detecting and compressing dissolve regions in video sequences with a DVI multimedia image compression algorithm , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[90]  King Ngi Ngan,et al.  Integrated shot boundary detection using object-based technique , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[91]  Eid Emary,et al.  Abrupt Cut Detection in News Videos Using Dominant Colors Representation , 2016, AISI.

[92]  Özgür Ulusoy,et al.  Fuzzy color histogram-based video segmentation , 2010, Comput. Vis. Image Underst..

[93]  Monisha Sharma,et al.  Comparative Study of Block Matching Algorithm and Dual Tree Complex Wavelet Transform for Shot Detection in Videos , 2014, 2014 International Conference on Electronic Systems, Signal Processing and Computing Technologies.

[94]  Min Gyo Chung,et al.  A scene boundary detection method , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[95]  Ba Tu Truong,et al.  New enhancements to cut, fade, and dissolve detection processes in video segmentation , 2000, ACM Multimedia.

[96]  Tsuhan Chen,et al.  Learning to Segment a Video to Clips Based on Scene and Camera Motion , 2012, ECCV.

[97]  Ralph Weischedel,et al.  PERFORMANCE MEASURES FOR INFORMATION EXTRACTION , 2007 .

[98]  Ioannis Pitas,et al.  Video Shot Boundary Detection and Condensed Representation : A Review , 2006 .

[99]  Bo Zhang,et al.  A novel shot boundary detection framework , 2005, Visual Communications and Image Processing.

[100]  S. Domnic,et al.  Edge Strength Extraction using Orthogonal Vectors for Shot Boundary Detection , 2012 .

[101]  Siba Haidar,et al.  The SAMOVA Shot Boundary Detection for TRECVID Evaluation 2004 , 2004, TRECVID.

[102]  Kikukawa Takeshi,et al.  Development of an Automatic Summary Editing System for the Audio Visual Resources. , 1992 .

[103]  Th. Hermes,et al.  Automatic Shot Boundary Detecting Using Adaptive Thresholds , 2003, TRECVID.

[104]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[105]  Raimondo Schettini,et al.  Dynamic storyboards for video content summarization , 2006, MIR '06.

[106]  Andries Petrus Engelbrecht,et al.  A Cooperative approach to particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[107]  Riccardo Leonardi,et al.  Scene break detection: a comparison , 1998, Proceedings Eighth International Workshop on Research Issues in Data Engineering. Continuous-Media Databases and Applications.

[108]  Ramakant Nevatia,et al.  Video event classification with temporal partitioning , 2015, 2015 12th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[109]  Ramesh C. Jain,et al.  Dynamic vision , 1988, [1988 Proceedings] 9th International Conference on Pattern Recognition.

[110]  Ramesh C. Jain,et al.  Production model based digital video segmentation , 1995, Multimedia Tools and Applications.

[111]  Moncef Gabbouj,et al.  Demo. Video shot boundary detection by structural analysis of local image features , 2011, WIAMIS 2011.

[112]  Jessika Weiss,et al.  Vision Science Photons To Phenomenology , 2016 .

[113]  Dong Wang,et al.  Learning a Contextual Multi-Thread Model for Movie/TV Scene Segmentation , 2013, IEEE Transactions on Multimedia.

[114]  Rita Cucchiara,et al.  Linear Transition Detection as a Unified Shot Detection Approach , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[115]  Fernando Díaz-de-María,et al.  Temporal segmentation and keyframe selection methods for user-generated video search-based annotation , 2015, Expert Syst. Appl..

[116]  Shin'ichi Satoh,et al.  A Framework for Video Segmentation using Global and Local Features , 2013, Int. J. Pattern Recognit. Artif. Intell..

[117]  Rainer Lienhart,et al.  Reliable dissolve detection , 2001, IS&T/SPIE Electronic Imaging.

[118]  Li Huan,et al.  A Method for Fast Shot Boundary Detection Based on SVM , 2008, 2008 Congress on Image and Signal Processing.

[119]  James A. Thom,et al.  Temporal video segmentation: detecting the end-of-act in circus performance videos , 2015, Multimedia Tools and Applications.

[120]  Yong Shi,et al.  Fast Video Shot Boundary Detection Based on SVD and Pattern Matching , 2013, IEEE Transactions on Image Processing.

[121]  Syed Abdul Rahman Al-Haddad,et al.  On Computational Aspects of Tchebichef Polynomials for Higher Polynomial Order , 2017, IEEE Access.

[122]  Amr Ahmed Video Representation and Processing for Multimedia Data Mining , 2009, Semantic Mining Technologies for Multimedia Databases.

[123]  Hui Fang,et al.  A fuzzy logic approach for detection of video shot boundaries , 2006, Pattern Recognit..

[124]  John R. Kender,et al.  Computational approaches to temporal sampling of video sequences , 2007, TOMCCAP.

[125]  K. P. Uma,et al.  Kirsch Directional Derivatives Based Shot Boundary Detection: An Efficient and Accurate Method , 2015 .

[126]  Mahmood Fathy,et al.  Video Shot Boundary Detection Using QR-Decomposition and Gaussian Transition Detection , 2010, EURASIP J. Adv. Signal Process..

[127]  Betim Çiço,et al.  Computer vision approaches based on deep learning and neural networks: Deep neural networks for video analysis of human pose estimation , 2017, 2017 6th Mediterranean Conference on Embedded Computing (MECO).

[128]  N. Nikolaidis,et al.  Video shot detection and condensed representation. a review , 2006, IEEE Signal Processing Magazine.

[129]  Yanming Zou,et al.  A Temporal Video Segmentation and Summary Generation Method Based on Shots' Abrupt and Gradual Transition Boundary Detecting , 2010, 2010 Second International Conference on Communication Software and Networks.

[130]  T. Kanade,et al.  Color information for region segmentation , 1980 .

[131]  Magda B. Fayk,et al.  Particle swarm optimisation based video abstraction , 2010 .

[132]  Kazimierz Choros,et al.  False and Miss Detections in Temporal Segmentation of TV Sports News Videos - Causes and Remedies , 2014, MISSI.

[133]  László Böszörményi,et al.  State-of-the-art and future challenges in video scene detection: a survey , 2013, Multimedia Systems.

[134]  Ioannis Pitas,et al.  Video shot-boundary detection using singular-value decomposition and statistical tests , 2007, J. Electronic Imaging.

[135]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[136]  Soyoung Park,et al.  Effect of adaptive thresholding on shot boundary detection performance , 2016, 2016 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia).

[137]  Lawrence G. Roberts,et al.  Machine Perception of Three-Dimensional Solids , 1963, Outstanding Dissertations in the Computer Sciences.

[138]  Minaz J. Parmar,et al.  MAC-REALM: A Video Content Feature Extraction and Modelling Framework , 2015, Computer/law journal.

[139]  R. Priya,et al.  A comprehensive review of significant researches on content based indexing and retrieval of visual information , 2013, Frontiers of Computer Science.

[140]  Bo Zhang,et al.  A Formal Study of Shot Boundary Detection , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[141]  J. Tasic,et al.  Colour spaces: perceptual, historical and applicational background , 2003, The IEEE Region 8 EUROCON 2003. Computer as a Tool..

[142]  Irena Koprinska,et al.  Temporal video segmentation: A survey , 2001, Signal Process. Image Commun..

[143]  Somnath Sengupta,et al.  Switchable video error concealment using encoder driven scene transition detection and edge preserving SEC , 2013, Multimedia Tools and Applications.

[144]  Juan Chen,et al.  Modelling of content-aware indicators for effective determination of shot boundaries in compressed MPEG videos , 2010, Multimedia Tools and Applications.

[145]  Atreyi Kankanhalli,et al.  Automatic partitioning of full-motion video , 1993, Multimedia Systems.

[146]  Bhabatosh Chanda,et al.  A shot detection technique using linear regression of shot transition pattern , 2014, Multimedia Tools and Applications.

[147]  Aman Parnami,et al.  Indian Institute of Technology, Bombay at TRECVID 2006 , 2006, TRECVID.

[148]  G. Camara-Chavez,et al.  Shot Boundary Detection by a Hierarchical Supervised Approach , 2007, 2007 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services.

[149]  Balasubramanian Raman,et al.  A Hierarchical Shot Boundary Detection Algorithm Using Global and Local Features , 2016, CVIP.

[150]  King Ngi Ngan,et al.  An Object-Based Shot Boundary Detection Using Edge Tracing and Tracking , 2001, J. Vis. Commun. Image Represent..

[151]  Kazimierz Choros,et al.  Effectiveness of Video Segmentation Techniques for Different Categories of Videos , 2008, New Trends in Multimedia and Network Information Systems.

[152]  Diego Cabrera,et al.  Attribute clustering using rough set theory for feature selection in fault severity classification of rotating machinery , 2017, Expert Syst. Appl..

[153]  Nicole Vincent,et al.  A review of real-time segmentation of uncompressed video sequences for content-based search and retrieval , 2003, Real Time Imaging.

[154]  Sarp Ertürk,et al.  Modified phase-correlation based robust hard-cut detection with application to archive film , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[155]  Aroh Barjatya,et al.  Block Matching Algorithms For Motion Estimation , 2004 .

[156]  Rong Xie,et al.  Shot boundary detection using convolutional neural networks , 2016, 2016 Visual Communications and Image Processing (VCIP).