Sparse coding with earth mover’s distance for multi-instance histogram representation

Sparse coding (SC) has been studied very well as a powerful data representation method. It attempts to represent the feature vector of a data sample by reconstructing it as the sparse linear combination of some basic elements, and a $$L_2$$L2 norm distance function is usually used as the loss function for the reconstruction error. In this paper, we investigate using SC as the representation method within multi-instance learning framework, where a sample is given as a bag of instances and further represented as a histogram of the quantized instances. We argue that for the data type of histogram, using $$L_2$$L2 norm distance is not suitable, and propose to use the earth mover’s distance (EMD) instead of $$L_2$$L2 norm distance as a measure of the reconstruction error. By minimizing the EMD between the histogram of a sample and the its reconstruction from some basic histograms, a novel sparse coding method is developed, which is refereed as SC-EMD. We evaluate its performances as a histogram representation method in tow multi-instance learning problems—abnormal image detection in wireless capsule endoscopy videos, and protein binding site retrieval. The encouraging results demonstrate the advantages of the new method over the traditional method using $$L_2$$L2 norm distance.

[1]  Xiangping Kang,et al.  A multi-instance ensemble learning model based on concept lattice , 2011, Knowl. Based Syst..

[2]  Jude W. Shavlik,et al.  Gleaner: Creating ensembles of first-order clauses to improve recall-precision curves , 2006, Machine Learning.

[3]  Ke Tang,et al.  Margin-Based Over-Sampling Method for Learning from Imbalanced Datasets , 2011, PAKDD.

[4]  Haibin Ling,et al.  An Efficient Earth Mover's Distance Algorithm for Robust Histogram Comparison , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Zhi-Hua Zhou,et al.  Multi-instance learning by treating instances as non-I.I.D. samples , 2008, ICML '09.

[6]  Zhi-Hua Zhou,et al.  Multi-Instance Multi-Label Learning with Application to Scene Classification , 2006, NIPS.

[7]  Changhe Yuan,et al.  An Improved Lower Bound for Bayesian Network Structure Learning , 2015, AAAI.

[8]  Michael D. Gordon,et al.  Recall-precision trade-off: A derivation , 1989, JASIS.

[9]  Bing Yu,et al.  A fuzzy TOPSIS model via chi-square test for information source selection , 2013, Knowl. Based Syst..

[10]  Jim Jing-Yan Wang,et al.  Support image set machine: Jointly learning representation and classifier for image set classification , 2015, Knowl. Based Syst..

[11]  Madan Gopal,et al.  Reduced one-against-all method for multiclass SVM classification , 2011, Expert Syst. Appl..

[12]  Zhang Yi,et al.  Learning locality-constrained collaborative representation for robust face recognition , 2012, Pattern Recognit..

[13]  Jiayu Tang,et al.  A Study of Quality Issues for Image Auto-Annotation With the Corel Dataset , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[15]  R. Raz,et al.  ProMate: a structure based prediction program to identify the location of protein-protein binding sites. , 2004, Journal of molecular biology.

[16]  Youfu Li,et al.  Adaptive weighted learning for linear regression problems via Kullback-Leibler divergence , 2013, Pattern Recognit..

[17]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[18]  Yunqian Ma,et al.  Practical selection of SVM parameters and noise estimation for SVM regression , 2004, Neural Networks.

[19]  Barbara Caputo,et al.  Recognizing human actions: a local SVM approach , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[20]  N. Cook Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[21]  Zhiwu Lu,et al.  Image annotation by semantic sparse recoding of visual content , 2012, ACM Multimedia.

[22]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[23]  Kamesh Namuduri,et al.  Video stabilization using principal component analysis and scale invariant feature transform in particle filter framework , 2009, IEEE Transactions on Consumer Electronics.

[24]  M. Ding,et al.  A ladder method for linear semi-infinite programming , 2013 .

[25]  Nathalie Japkowicz,et al.  Boosting support vector machines for imbalanced data sets , 2008, Knowledge and Information Systems.

[26]  R. Bagby,et al.  The twenty-item Toronto Alexithymia Scale--I. Item selection and cross-validation of the factor structure. , 1994, Journal of psychosomatic research.

[27]  Gholam R. Amin A comment on modified big-M method to recognize the infeasibility of linear programming models , 2010, Knowl. Based Syst..

[28]  Alistair A. Young,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2017, MICCAI 2017.

[29]  Qiu Chen,et al.  Face recognition using vector quantization histogram method , 2002, Proceedings. International Conference on Image Processing.

[30]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[31]  Shun-ichi Amari,et al.  The AIC Criterion and Symmetrizing the Kullback–Leibler Divergence , 2007, IEEE Transactions on Neural Networks.

[32]  Michael Lindenbaum,et al.  Nonnegative Matrix Factorization with Earth Mover's Distance Metric for Image Analysis , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Liang-Tien Chia,et al.  Laplacian Sparse Coding, Hypergraph Laplacian Sparse Coding, and Applications , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  P. Swain,et al.  Wireless capsule endoscopy: a comparison with push enteroscopy in patients with gastroscopy and colonoscopy negative gastrointestinal bleeding , 2003, Gut.

[35]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[36]  Yi Liu,et al.  One-against-all multi-class SVM classification using reliability measures , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[37]  Liang-Tien Chia,et al.  Local features are not lonely – Laplacian sparse coding for image classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Mei-Chen Yeh,et al.  Fast Human Detection Using a Cascade of Histograms of Oriented Gradients , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[39]  Connie M. Borror,et al.  A Hybrid SPC Method with the Chi‐Square Distance Monitoring Procedure for Large‐scale, Complex Process Data , 2006, Qual. Reliab. Eng. Int..

[40]  Jitendra Malik,et al.  SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[41]  Anne M. Denton,et al.  Data mining of vector–item patterns using neighborhood histograms , 2009, Knowledge and Information Systems.

[42]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[43]  Peter J. Bickel,et al.  The Earth Mover's distance is the Mallows distance: some insights from statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[44]  David R. Westhead,et al.  Improved prediction of protein-protein binding sites using a support vector machines approach. , 2005, Bioinformatics.

[45]  Shengping Zhang,et al.  Sparse coding based visual tracking: Review and experimental comparison , 2013, Pattern Recognit..

[46]  Changhe Yuan,et al.  Tightening Bounds for Bayesian Network Structure Learning , 2014, AAAI.

[47]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[48]  Yixin Chen,et al.  Image Categorization by Learning and Reasoning with Regions , 2004, J. Mach. Learn. Res..

[49]  Shouhuai Xu,et al.  Cross-layer detection of malicious websites , 2013, CODASPY.

[50]  Fei Guo,et al.  A novel multivariate performance optimization method based on sparse coding and hyper-predictor learning , 2015, Neural Networks.

[51]  Thierry Denoeux,et al.  A k-nearest neighbor classification rule based on Dempster-Shafer theory , 1995, IEEE Trans. Syst. Man Cybern..

[52]  Robert Powers,et al.  Protein NMR recall, precision, and F-measure scores (RPF scores): structure quality assessment measures based on information retrieval statistics. , 2005, Journal of the American Chemical Society.

[53]  Pierre Baldi,et al.  Assessing the accuracy of prediction algorithms for classification: an overview , 2000, Bioinform..

[54]  Chi-Ren Shyu,et al.  Fast protein binding site comparisons using visual words representation , 2012, Bioinform..

[55]  Ke Tang,et al.  Enhanced Maximum AUC Linear Classifier , 2010, 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery.

[56]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[57]  Ghassan Hamarneh,et al.  N-Sift: N-Dimensional Scale Invariant Feature Transform for Matching Medical Images , 2007, ISBI.

[58]  Shouhuai Xu,et al.  Trustworthy Information: Concepts and Mechanisms , 2010, WAIM.

[59]  John R. Hershey,et al.  Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[60]  Yuan Yan Tang,et al.  A Visual-Attention Model Using Earth Mover's Distance-Based Saliency Measurement and Nonlinear Feature Combination , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Tina Eliassi-Rad,et al.  Correcting evaluation bias of relational classifiers with network cross validation , 2010, Knowledge and Information Systems.

[62]  Sae Hwang Bag-of-Visual-Words Approach to Abnormal Image Detection in Wireless Capsule Endoscopy Videos , 2011, ISVC.

[63]  Shouhuai Xu,et al.  Adaptive Epidemic Dynamics in Networks , 2013, ACM Trans. Auton. Adapt. Syst..

[64]  Zhiwu Lu,et al.  Latent Semantic Learning by Efficient Sparse Coding with Hypergraph Regularization , 2011, AAAI.

[65]  Thomas Hofmann,et al.  Multi-Instance Multi-Label Learning with Application to Scene Classification , 2007 .

[66]  G. Wahba Smoothing noisy data with spline functions , 1975 .

[67]  Changhe Yuan,et al.  Finding Optimal Bayesian Network Structures with Constraints Learned from Data , 2014, UAI.

[68]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[69]  Jun Zhou,et al.  Discriminative sparse neighbor coding , 2016, Multimedia Tools and Applications.

[70]  Kemal Polat,et al.  A novel hybrid intelligent method based on C4.5 decision tree classifier and one-against-all approach for multi-class classification problems , 2009, Expert Syst. Appl..

[71]  Hans Burkhardt,et al.  Colour image retrieval based on DCT-domain vector quantisation index histograms , 2005 .

[72]  J. Ware,et al.  Cross-validation of item selection and scoring for the SF-12 Health Survey in nine countries: results from the IQOLA Project. International Quality of Life Assessment. , 1998, Journal of clinical epidemiology.

[73]  Chun Chen,et al.  Graph Regularized Sparse Coding for Image Representation , 2011, IEEE Transactions on Image Processing.

[74]  P. Tsai,et al.  Histogram-based reversible data hiding for vector quantisation-compressed images , 2009, IET Image Process..

[75]  Leida Li,et al.  Geometrically robust image watermarking using scale-invariant feature transform and Zernike moments , 2007 .

[76]  P. Swain,et al.  Wireless capsule endoscopy. , 2002, The Israel Medical Association journal : IMAJ.

[77]  Yihong Gong,et al.  Linear spatial pyramid matching using sparse coding for image classification , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[79]  C. Christopoulos,et al.  Efficient methods for encoding regions of interest in the upcoming JPEG2000 still image coding standard , 2000, IEEE Signal Processing Letters.

[80]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[81]  Shouhuai Xu,et al.  A Stochastic Model of Multivirus Dynamics , 2012, IEEE Transactions on Dependable and Secure Computing.

[82]  Jingyan Wang,et al.  Representing Data by Sparse Combination of Contextual Data Points for Classification , 2015, ISNN.

[83]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[84]  A. Broggi,et al.  Pedestrian Detection using Infrared images and Histograms of Oriented Gradients , 2006, 2006 IEEE Intelligent Vehicles Symposium.

[85]  Zhiwu Lu An iterative algorithm for entropy regularized likelihood learning on Gaussian mixture with automatic model selection , 2006, Neurocomputing.

[86]  Rui Xu,et al.  Survey of clustering algorithms , 2005, IEEE Transactions on Neural Networks.

[87]  Zhiwu Lu,et al.  Exhaustive and Efficient Constraint Propagation: A Graph-Based Learning Approach and Its Applications , 2011, International Journal of Computer Vision.

[88]  Majid Soleimani-Damaneh,et al.  Modified big-M method to recognize the infeasibility of linear programming models , 2008, Knowl. Based Syst..

[89]  N. Ye,et al.  Robustness of Chi‐square and Canberra distance metrics for computer intrusion detection , 2002 .

[90]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[91]  Zhiwu Lu,et al.  Spatial Markov Kernels for Image Categorization and Annotation , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[92]  Ivor W. Tsang,et al.  Core Vector Machines: Fast SVM Training on Very Large Data Sets , 2005, J. Mach. Learn. Res..

[93]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[94]  Shouhuai Xu,et al.  Characterizing Honeypot-Captured Cyber Attacks: Statistical Framework and Case Study , 2013, IEEE Transactions on Information Forensics and Security.

[95]  Arkadi Nemirovski,et al.  Robust solutions of Linear Programming problems contaminated with uncertain data , 2000, Math. Program..

[96]  Zhi-Hua Zhou,et al.  Multi-Instance Learning Based Web Mining , 2005, Applied Intelligence.

[97]  Gerardo Beni,et al.  A Validity Measure for Fuzzy Clustering , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[98]  Shouhuai Xu,et al.  A Characterization of Cybersecurity Posture from Network Telescope Data , 2014, INTRUST.

[99]  Marko Heikkilä,et al.  Description of interest regions with local binary patterns , 2009, Pattern Recognit..

[100]  Richard M. Jackson,et al.  Q-SiteFinder: an energy-based method for the prediction of protein-ligand binding sites , 2005, Bioinform..

[101]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[102]  Fady Alajaji,et al.  The Kullback-Leibler divergence rate between Markov sources , 2004, IEEE Transactions on Information Theory.

[103]  Shouhuai Xu,et al.  An evasion and counter-evasion study in malicious websites detection , 2014, 2014 IEEE Conference on Communications and Network Security.

[104]  Dong-Chul Park,et al.  Centroid neural network with Chi square distance measure for texture classification , 2009, 2009 International Joint Conference on Neural Networks.

[105]  Frans Coenen,et al.  Formalising optimal feature weight setting in case based diagnosis as linear programming problems , 2002, Knowl. Based Syst..

[106]  Giles M. Foody,et al.  Status of land cover classification accuracy assessment , 2002 .

[107]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[108]  A. May,et al.  The First Prospective Controlled Trial Comparing Wireless Capsule Endoscopy with Push Enteroscopy in Chronic Gastrointestinal Bleeding , 2002, Endoscopy.

[109]  Longin Jan Latecki,et al.  Improving SVM classification on imbalanced time series data sets with ghost points , 2011, Knowledge and Information Systems.