Visual search for musical performances and endoscopic videos

[ANGLES] This project explores the potential of LIRE, an en existing Content-Based Image Retrieval (CBIR) system, when used to retrieve medical videos. These videos are recording of the live streams used by surgeons during the endoscopic procedures, captured from inside of the subject. The growth of such video content stored in servers requires search engines capable to assist surgeons in their management and retrieval. In our tool, queries are formulated by visual examples and those allow surgeons to re-find shots taken during the procedure. This thesis presents an extension and adaptation of Lire for video retrieval based on visual features and late fusion. The results are assessed from two perspectives: a quantitative and qualitative one. While the quantitative one follows the standard practices and metrics for video retrieval, the qualitative assessment has been based on an empirical social study using a semi-interactive web-interface. In particular, a thinking aloud test was applied to analyze if the user expectations and requirements were fulfilled. Due to the scarcity of surgeons available for the qualitative tests, a second domain was also addressed: videos captured at musical performances. These type of videos has also experienced an exponential growth with the advent of affordable multimedia smart phones, available to a large audience. Analogously to the endoscopic videos, searching in a large data set of such videos is a challenging topic.

[1]  David Dagan Feng,et al.  Content-Based Medical Image Retrieval: A Survey of Applications to Multidimensional and Multimodality Data , 2013, Journal of Digital Imaging.

[2]  João Magalhães,et al.  Multimodal medical information retrieval with unsupervised rank fusion , 2015, Comput. Medical Imaging Graph..

[3]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[4]  Wei Tsang Ooi,et al.  The jiku mobile video dataset , 2013, MMSys.

[5]  Gwénolé Quellec,et al.  Wavelet optimization for content-based image retrieval in medical databases , 2010, Medical Image Anal..

[6]  Gwénolé Quellec,et al.  Real-time retrieval of similar videos with application to computer-aided retinal surgery , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[7]  Jianping Fan,et al.  Concept-oriented indexing of video databases: toward semantic sensitive retrieval and browsing , 2004, IEEE Transactions on Image Processing.

[8]  Henning Müller,et al.  Comparing fusion techniques for the ImageCLEF 2013 medical case retrieval task , 2015, Comput. Medical Imaging Graph..

[9]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[10]  Jayashree Kalpathy-Cramer,et al.  Multimodal medical image retrieval: image categorization to improve search precision , 2010, MIR '10.

[11]  Marc P. Schuyler.  The MPEG-4 Video Standard Verification Model , 2017 .

[12]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[13]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[14]  Hugo Jair Escalante,et al.  Late fusion of heterogeneous methods for multimedia image retrieval , 2008, MIR '08.

[15]  Yiannis S. Boutalis,et al.  Searching images with MPEG-7 (& MPEG-7-like) Powered Localized dEscriptors: The SIMPLE answer to effective Content Based Image Retrieval , 2014, 2014 12th International Workshop on Content-Based Multimedia Indexing (CBMI).

[16]  Cheng Thao,et al.  GoldMiner: a radiology image search engine. , 2007, AJR. American journal of roentgenology.

[17]  George R. Thoma,et al.  A Learning-Based Similarity Fusion and Filtering Approach for Biomedical Image Retrieval Using SVM Classification and Relevance Feedback , 2011, IEEE Transactions on Information Technology in Biomedicine.

[18]  Antoine Geissbühler,et al.  A Review of Content{Based Image Retrieval Systems in Medical Applications { Clinical Bene(cid:12)ts and Future Directions , 2022 .

[19]  Paul Clough,et al.  ImageCLEF: Experimental Evaluation in Visual Information Retrieval , 2010 .

[20]  Mathias Lux.  LIRE: open source image retrieval in Java , 2013, MM '13.

[21]  Henning Müller,et al.  Overview of the ImageCLEF 2013 Medical Tasks , 2013, CLEF.

[22]  Jing Huang,et al.  Image indexing using color correlograms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Yiannis S. Boutalis,et al.  CEDD: Color and Edge Directivity Descriptor: A Compact Descriptor for Image Indexing and Retrieval , 2008, ICVS.

[24]  Adil Alpkocak,et al.  DEMIR at ImageCLEFMed 2013: The Effects of Modality Classification to Information Retrieval , 2013, CLEF.

[25]  Chang-Tsun Li,et al.  A Content-Based Approach to Medical Image Database Retrieval , 2009, Database Technologies: Concepts, Methodologies, Tools, and Applications.

[26]  Wei Tsang Ooi,et al.  On Demand Retrieval of Crowdsourced Mobile Video , 2015, IEEE Sensors Journal.

[27]  Alan F. Smeaton,et al.  A Comparison of Score, Rank and Probability-Based Fusion Methods for Video Shot Retrieval , 2005, CIVR.