Correlation-Assisted Imbalance Multimedia Concept Mining and Retrieval

In the past decades, we have witnessed an explosion of multimedia data, especially with the development of social media websites and blooming popularity of smart devices. As a result, multimedia semantic concept mining and retrieval whose objective is to mine useful information from the large amount of multimedia data including texts, images, and videos has become more and more important. The huge amount of multimedia data and the semantic gap between low-level features and high-level semantic concepts have made it even more challenging. To address these challenges, the correlations among the classes can provide important context cues to help bridge the semantic gap. Meanwhile, many real-world datasets do not have uniform class distributions while the minority instances actually represent the concept of interests, like frauds in transactions, intrusions in network security, and unusual events in surveillance. Despite extensive research efforts, imbalanced concept retrieval remains one of the most challenging research problems in multimedia data mining. Different from existing frameworks regarding concept correlations among labels, this paper presents a novel concept correlation analysis model using the correlation between the retrieval scores and labels. Experimental results on the TRECVID benchmark datasets demonstrate that the proposed framework can enhance imbalanced concept mining and retrieval even with trivial scores from the minority class.

[1]  Si Wu,et al.  Improving support vector machine classifiers by modifying kernel functions , 1999, Neural Networks.

[2]  Shu-Ching Chen,et al.  Network intrusion detection through Adaptive Sub-Eigenspace Modeling in multiagent systems , 2007, ACM Trans. Auton. Adapt. Syst..

[3]  Xin Huang,et al.  User Concept Pattern Discovery Using Relevance Feedback And Multiple Instance Learning For Content-Based Image Retrieval , 2002, MDM/KDD.

[4]  Min Chen,et al.  A unified framework for image database clustering and content-based retrieval , 2004, MMDB '04.

[5]  Koen E. A. van de Sande,et al.  All vehicles are cars: subclass preferences in container concepts , 2012, ICMR '12.

[6]  Rangasami L. Kashyap,et al.  Augmented transition networks as video browsing models for multimedia databases and multimedia information systems , 1999, Proceedings 11th International Conference on Tools with Artificial Intelligence.

[7]  Mei-Ling Shyu,et al.  Utilizing Context Information to Enhance Content-Based Image Classification , 2011, Int. J. Multim. Data Eng. Manag..

[8]  Shu-Ching Chen,et al.  Video Semantic Concept Discovery using Multimodal-Based Association Classification , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[9]  Zhi-Hua Zhou,et al.  Exploratory Undersampling for Class-Imbalance Learning , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[10]  Min Chen,et al.  Spatio-Temporal Analysis for Human Action Detection and Recognition in Uncontrolled Environments , 2015, Int. J. Multim. Data Eng. Manag..

[11]  Choochart Haruechaiyasak,et al.  Category cluster discovery from distributed WWW directories , 2003, Inf. Sci..

[12]  Samira Pouyanfar,et al.  Semantic Event Detection Using Ensemble Deep Learning , 2016, 2016 IEEE International Symposium on Multimedia (ISM).

[13]  Qiang Ji,et al.  A Hierarchical Context Model for Event Recognition in Surveillance Video , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Xiuqi Li,et al.  Image Retrieval By Color , Texture , And Spatial Information , 2002 .

[15]  Min Chen,et al.  Video Semantic Event/Concept Detection Using a Subspace-Based Multimedia Data Mining Framework , 2008, IEEE Transactions on Multimedia.

[16]  Dennis Koelma,et al.  The ImageNet Shuffle: Reorganized Pre-training for Video Event Detection , 2016, ICMR.

[17]  Shu-Ching Chen,et al.  Enhancing Multimedia Imbalanced Concept Detection Using VIMP in Random Forests , 2016, 2016 IEEE 17th International Conference on Information Reuse and Integration (IRI).

[18]  Mark Johnston,et al.  Developing New Fitness Functions in Genetic Programming for Classification With Unbalanced Data , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  Georges Quénot,et al.  TRECVID 2015 - An Overview of the Goals, Tasks, Data, Evaluation Mechanisms and Metrics , 2011, TRECVID.

[20]  Min Chen,et al.  Efficient Imbalanced Multimedia Concept Retrieval by Deep Learning on Spark Clusters , 2017, Int. J. Multim. Data Eng. Manag..

[21]  Chengcui Zhang,et al.  An intelligent framework for spatio-temporal vehicle tracking , 2001, ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585).

[22]  Chengcui Zhang,et al.  Innovative Shot Boundary Detection for Video Indexing , 2005 .

[23]  Bingbing Ni,et al.  YouTubeEvent: On large-scale video event classification , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[24]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Rangasami L. Kashyap,et al.  Augmented Transition Network as a Semantic Model for Video Data , 2001 .

[26]  Chao Chen,et al.  Weighted Subspace Filtering and Ranking Algorithms for Video Concept Retrieval , 2011, IEEE MultiMedia.

[27]  Mei-Ling Shyu,et al.  Supporting Semantic Concept Retrieval with Negative Correlations in a Multimedia Big Data Mining System , 2016, Int. J. Semantic Comput..

[28]  C. P. Unsworth,et al.  Excessive Noise Injection Training of Neural Networks for Markerless Tracking in Obscured and Segmented Environments , 2006, Neural Computation.

[29]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[30]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[31]  Shu-Ching Chen,et al.  A Classifier Ensemble Framework for Multimedia Big Data Classification , 2016, 2016 IEEE 17th International Conference on Information Reuse and Integration (IRI).

[32]  Min Chen,et al.  Deep Learning for Imbalanced Multimedia Data Classification , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[33]  Gustavo E. A. P. A. Batista,et al.  A study of the behavior of several methods for balancing machine learning training data , 2004, SKDD.

[34]  Rangasami L. Kashyap,et al.  Temporal And Spatial Semantic Models For Multimedia Presentations , 1997 .

[35]  Mei-Ling Shyu,et al.  Enhancing Rare Class Mining in Multimedia Big Data by Concept Correlation , 2016, 2016 IEEE International Symposium on Multimedia (ISM).

[36]  Wu Qingfeng,et al.  An empirical study on ensemble selection for class-imbalance data sets , 2010, 2010 5th International Conference on Computer Science & Education.

[37]  Min Chen,et al.  Utilizing concept correlations for effective imbalanced data classification , 2014, Proceedings of the 2014 IEEE 15th International Conference on Information Reuse and Integration (IEEE IRI 2014).

[38]  Mei-Ling Shyu,et al.  Negative Correlation Discovery for Big Multimedia Data Semantic Concept Mining and Retrieval , 2016, 2016 IEEE Tenth International Conference on Semantic Computing (ICSC).

[39]  Shu-Ching Chen,et al.  Feature Selection Using Correlation and Reliability Based Scoring Metric for Video Semantic Detection , 2010, 2010 IEEE Fourth International Conference on Semantic Computing.

[40]  Min Chen,et al.  Image database retrieval utilizing affinity relationships , 2003, MMDB '03.

[41]  Mei-Ling Shyu,et al.  Weighted Association Rule Mining for Video Semantic Detection , 2010, Int. J. Multim. Data Eng. Manag..

[42]  Rangasami L. Kashyap,et al.  Identifying Overlapped Objects for Video Indexing and Modeling in Multimedia Database Systems , 2001, Int. J. Artif. Intell. Tools.

[43]  Yang Liu,et al.  Enhancing Multimedia Semantic Concept Mining and Retrieval by Incorporating Negative Correlations , 2014, 2014 IEEE International Conference on Semantic Computing.

[44]  Rangasami L. Kashyap,et al.  Semantic Models for Multimedia Database Searching and Browsing , 2000, Advances in Database Systems.

[45]  Su-Shing Chen,et al.  A bit-serial VLSI architecture for generating moments in real-time , 1993, IEEE Trans. Syst. Man Cybern..

[46]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[47]  Yi Yang,et al.  Fast and Accurate Content-based Semantic Search in 100M Internet Videos , 2015, ACM Multimedia.

[48]  Xiuqi Li,et al.  An effective content-based visual image retrieval system , 2002, Proceedings 26th Annual International Computer Software and Applications.

[49]  Ching-Te Chiu,et al.  Boosted multi-class object detection with parallel hardware implementation for real-time applications , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[50]  Rangasami L. Kashyap,et al.  Generalized Affinity-Based Association Rule Mining for Multimedia Database Queries , 2001, Knowledge and Information Systems.

[51]  Mark R. Segal,et al.  Machine Learning Benchmarks and Random Forest Regression , 2004 .

[52]  Mei-Ling Shyu,et al.  Effective Feature Space Reduction with Imbalanced Data for Semantic Concept Detection , 2008, 2008 IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (sutc 2008).

[53]  Mei-Ling Shyu,et al.  Concept-concept association information integration and multi-model collaboration for multimedia semantic concept detection , 2014, Inf. Syst. Frontiers.

[54]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[55]  Mei-Ling Shyu,et al.  Leveraging Concept Association Network for Multimedia Rare Concept Mining and Retrieval , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[56]  Joseph E. Beck,et al.  Naive Bayes Classifiers for User Modeling , 1999 .

[57]  Min Chen,et al.  A latent semantic indexing based method for solving multiple instance learning problem in region-based image retrieval , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).