Multiple-Instance Learning for Medical Image and Video Analysis

Multiple-instance learning (MIL) is a recent machine-learning paradigm that is particularly well suited to medical image and video analysis (MIVA) tasks. Based solely on class labels assigned globally to images or videos, MIL algorithms learn to detect relevant patterns locally in images or videos. These patterns are then used for classification at a global level. Because supervision relies on global labels, manual segmentations are not needed to train MIL algorithms, unlike traditional single-instance learning (SIL) algorithms. Consequently, these solutions are attracting increasing interest from the MIVA community: since the term was coined by Dietterich et al. in 1997, 73 research papers about MIL have been published in the MIVA literature. This paper reviews the existing strategies for modeling MIVA tasks as MIL problems, recommends general-purpose MIL algorithms for each type of MIVA tasks, and discusses MIVA-specific MIL algorithms. Various experiments performed in medical image and video datasets are compiled in order to back up these discussions. This meta-analysis shows that, besides being more convenient than SIL solutions, MIL algorithms are also more accurate in many cases. In other words, MIL is the ideal solution for many MIVA tasks. Recent trends are discussed, and future directions are proposed for this emerging paradigm.

[1]  Anonymous Authors.  Empowering Multiple Instance Histopathology Cancer Diagnosis by Cell Graphs , 2014 .

[2]  Zhuowen Tu,et al.  Max-Margin Multiple-Instance Dictionary Learning , 2013, ICML.

[3]  Baoxin Li,et al.  Automatic classification and detection of clinically relevant images for diabetic retinopathy , 2008, SPIE Medical Imaging.

[4]  Marian Stewart Bartlett,et al.  Weakly supervised pain localization using multiple instance learning , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[5]  Robert P. W. Duin,et al.  Dissimilarity representations allow for building good classifiers , 2002, Pattern Recognit. Lett..

[6]  Jean Ponce,et al.  Automatic annotation of human actions in video , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  James R. Foulds,et al.  A review of multi-instance learning assumptions , 2010, The Knowledge Engineering Review.

[8]  Zhuowen Tu,et al.  Multiple clustered instance learning for histopathology cancer image classification, segmentation and clustering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Lauge Sørensen,et al.  Classification of COPD with Multiple Instance Learning , 2014, 2014 22nd International Conference on Pattern Recognition.

[10]  Kim L. Boyer,et al.  A min-max framework of cascaded classifier with multiple instance learning for computer aided diagnosis , 2009, CVPR.

[11]  Zhan Li,et al.  Mil-Cand: An Improved Framework for Computer Aided Nodule Detection Based on Multiple Instance Learning , 2012, CCPR.

[12]  Bernard Ghanem,et al.  Reduction of lymph tissue false positives in pulmonary embolism detection , 2008, Medical Imaging: Computer-Aided Diagnosis.

[13]  Thomas Gärtner,et al.  Multi-Instance Kernels , 2002, ICML.

[14]  Lauge Sørensen,et al.  Image Dissimilarity-Based Quantification of Lung Disease from CT , 2010, MICCAI.

[15]  Misha Denil,et al.  From Group to Individual Labels Using Deep Features , 2015, KDD.

[16]  Gwénolé Quellec,et al.  A multiple-instance learning framework for diabetic retinopathy screening , 2012, Medical Image Anal..

[17]  Christian Roux,et al.  Fouille d’images multi-instance et multi-résolution appliquée au dépistage de la rétinopathie diabétique , 2011 .

[18]  Gwénolé Quellec,et al.  A Polynomial Model of Surgical Gestures for Real-Time Retrieval of Surgery Videos , 2012, MCBR-CDS.

[19]  Liang Chen,et al.  Identification of Cerebral Small Vessel Disease Using Multiple Instance Learning , 2015, MICCAI.

[20]  David M. J. Tax,et al.  Characterizing Multiple Instance Datasets , 2015, SIMBAD.

[21]  Paul A. Viola,et al.  Multiple Instance Boosting for Object Detection , 2005, NIPS.

[22]  Jelena Kovacevic,et al.  Automated colitis detection from endoscopic biopsies as a tissue screening tool in diagnostic pathology , 2012, 2012 19th IEEE International Conference on Image Processing.

[23]  Xin Xu,et al.  Logistic Regression and Boosting for Labeled Bags of Instances , 2004, PAKDD.

[24]  Baoxin Li,et al.  Simpler Non-Parametric Methods Provide as Good or Better Results to Multiple-Instance Learning , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Bram van Ginneken,et al.  Multiple-instance learning for computer-aided detection of tuberculosis , 2014, Medical Imaging.

[26]  Peter L. Bartlett,et al.  Boosting Algorithms as Gradient Descent , 1999, NIPS.

[27]  Zhenhong Jia,et al.  Combining CNN and MIL to Assist Hotspot Segmentation in Bone Scintigraphy , 2015, ICONIP.

[28]  Mathieu Lamard,et al.  Multiple-Instance Learning for Anomaly Detection in Digital Mammography , 2016, IEEE Transactions on Medical Imaging.

[29]  Ronan Collobert,et al.  From image-level to pixel-level labeling with Convolutional Networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Wei Liu,et al.  Multi-instance learning for mass retrieval in digitized mammograms , 2012, Medical Imaging.

[31]  Jinbo Bi,et al.  Multiple Instance Learning of Pulmonary Embolism Detection with Geodesic Distance along Vascular Structure , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Heng-Da Cheng,et al.  Local-weighted Citation-kNN algorithm for breast ultrasound image classification , 2015 .

[33]  Gwénolé Quellec,et al.  Multiple-instance learning for breast cancer detection in mammograms , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[34]  Razvan C. Bunescu,et al.  Multiple instance learning for sparse positive bags , 2007, ICML '07.

[35]  Serge J. Belongie,et al.  Simultaneous Learning and Alignment: Multi-Instance and Multi-Pose Learning ? , 2008 .

[36]  Narendra Ahuja,et al.  MIS-Boost: Multiple Instance Selection Boosting , 2011, ArXiv.

[37]  Andrew P. Bradley,et al.  Multiple instance learning for breast MRI based on generic spatio-temporal features , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Xiaojing Ye,et al.  Coarse-to-fine classification via parametric and nonparametric models for computer-aided diagnosis , 2011, CIKM '11.

[39]  Kunio Doi,et al.  Computer-aided diagnosis in medical imaging: Historical review, current status and future potential , 2007, Comput. Medical Imaging Graph..

[40]  Rama Chellappa,et al.  Generalized Dictionaries for Multiple Instance Learning , 2015, International Journal of Computer Vision.

[41]  Chao Li,et al.  Multiple instance learning for computer aided detection and diagnosis of gastric cancer with dual-energy CT imaging , 2015, J. Biomed. Informatics.

[42]  Gwénolé Quellec,et al.  Weakly supervised classification of medical images , 2012, 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI).

[43]  Baoxin Li,et al.  Retrieving clinically relevant diabetic retinopathy images using a multi-class multiple-instance framework , 2013, Medical Imaging.

[44]  Yaniv Gal,et al.  Multiple Instance Learning for Breast Cancer Magnetic Resonance Imaging , 2014, 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[45]  Zhuowen Tu,et al.  Weakly supervised histopathology cancer image segmentation and classification , 2014, Medical Image Anal..

[46]  Di Zhao,et al.  A Novel Multiinstance Learning Approach for Liver Cancer Recognition on Abdominal CT Images Based on CPSO-SVM and IO , 2013, Comput. Math. Methods Medicine.

[47]  Murat Dundar,et al.  Bayesian multiple instance learning: automatic feature selection and inductive transfer , 2008, ICML '08.

[48]  Ethem Alpaydin,et al.  Single- vs. multiple-instance classification , 2015, Pattern Recognit..

[49]  Metin Nafi Gürcan,et al.  A Multiple Instance Learning Approach toward Optimal Classification of Pathology Slides , 2010, 2010 20th International Conference on Pattern Recognition.

[50]  Yixin Chen,et al.  MILES: Multiple-Instance Learning via Embedded Instance Selection , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Yilong Yin,et al.  Detecting malignant patients via modified boosted tree , 2010, Science China Information Sciences.

[52]  Georg Langs,et al.  Predicting Semantic Descriptions from Medical Images with Convolutional Neural Networks , 2015, IPMI.

[53]  Tomás Lozano-Pérez,et al.  A Framework for Multiple-Instance Learning , 1997, NIPS.

[54]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[55]  Qi Zhang,et al.  EM-DD: An Improved Multiple-Instance Learning Technique , 2001, NIPS.

[56]  Melih Kandemir,et al.  Instance Label Prediction by Dirichlet Process Multiple Instance Learning , 2014, UAI.

[57]  Bram van Ginneken,et al.  A Novel Multiple-Instance Learning-Based Approach to Computer-Aided Detection of Tuberculosis on Chest X-Rays , 2015, IEEE Transactions on Medical Imaging.

[58]  Mark Craven,et al.  Supervised versus multiple instance learning: an empirical comparison , 2005, ICML.

[59]  Jinbo Bi,et al.  Effective 3D object detection and regression using probabilistic segmentation features in CT images , 2011, CVPR 2011.

[60]  Daniel Rueckert,et al.  Multiple instance learning for classification of dementia in brain MRI , 2013, Medical Image Anal..

[61]  Zhi-Hua Zhou,et al.  Improve Multi-Instance Neural Networks through Feature Selection , 2004, Neural Processing Letters.

[62]  Anders Hast,et al.  Automated Classification of Glandular Tissue by Statistical Proximity Sampling , 2015, Int. J. Biomed. Imaging.

[63]  Jian Yin,et al.  Automated skin biopsy histopathological image annotation using multi-instance representation and learning , 2013, BMC Medical Genomics.

[64]  Trevor Darrell,et al.  Fully Convolutional Multi-Class Multiple Instance Learning , 2014, ICLR.

[65]  Melih Kandemir,et al.  Computer-aided diagnosis from weak supervision: A benchmarking study , 2015, Comput. Medical Imaging Graph..

[66]  Yang Gao,et al.  Multi-Instance Dictionary Learning for Detecting Abnormal Events in Surveillance Videos , 2014, Int. J. Neural Syst..

[67]  Zheng Bing,et al.  Medical image semantic annotation based on MIL , 2013, 2013 ICME International Conference on Complex Medical Engineering.

[68]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[69]  Ivor W. Tsang,et al.  A Convex Method for Locating Regions of Interest with Multi-instance Learning , 2009, ECML/PKDD.

[70]  R. Bharat Rao,et al.  Multiple-Instance Learning Improves CAD Detection of Masses in Digital Mammography , 2008, Digital Mammography / IWDM.

[71]  Jian Yin,et al.  Multi-instance learning for skin biopsy image features recognition , 2012, 2012 IEEE International Conference on Bioinformatics and Biomedicine.

[72]  Jun Wang,et al.  Solving the Multiple-Instance Problem: A Lazy Learning Approach , 2000, ICML.

[73]  Thomas Hofmann,et al.  Multi-Instance Multi-Label Learning with Application to Scene Classification , 2007 .

[74]  Marian Stewart Bartlett,et al.  Classification and weakly supervised pain localization using multiple segment representation , 2014, Image Vis. Comput..

[75]  Gwénolé Quellec,et al.  Real-time retrieval of similar videos with application to computer-aided retinal surgery , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[76]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[77]  Zhi-Hua Zhou.  Multi-Instance Learning : A Survey , 2004 .

[78]  Ronald M. Summers,et al.  Seeing Is Believing: Video Classification for Computed Tomographic Colonography Using Multiple-Instance Learning , 2012, IEEE Transactions on Medical Imaging.

[79]  Thomas Hofmann,et al.  Support Vector Machines for Multiple-Instance Learning , 2002, NIPS.

[80]  Zhuowen Tu,et al.  Context-Constrained Multiple Instance Learning for Histopathology Image Segmentation , 2012, MICCAI.

[81]  Jinbo Bi,et al.  Robust Parametric Modeling Approach Based on Domain Knowledge for Computer Aided Detection of Vertebrae Column Metastases in MRI , 2007, IPMI.

[82]  Xianglong Tang,et al.  Ultrasonic classification of breast tumors based on multi-instance learning , 2011, International Symposium on Multispectral Image Processing and Pattern Recognition.

[83]  Antoine Geissbühler,et al.  A Review of Content{Based Image Retrieval Systems in Medical Applications { Clinical Bene(cid:12)ts and Future Directions , 2022 .

[84]  Murat Dundar,et al.  Multiple-Instance Learning Algorithms for Computer-Aided Detection , 2008, IEEE Transactions on Biomedical Engineering.

[85]  Gwénolé Quellec,et al.  Real-time recognition of surgical tasks in eye surgery videos , 2014, Medical Image Anal..

[86]  Wynne Hsu,et al.  Abnormality Detection in Retinal Images , 2004 .

[87]  Heng-Da Cheng,et al.  Multiple-instance learning with global and local features for thyroid ultrasound image classification , 2014, 2014 7th International Conference on Biomedical Engineering and Informatics.

[88]  Fernando De la Torre,et al.  Gaussian Processes Multiple Instance Learning , 2010, ICML.

[89]  Alejandro F Frangi,et al.  Multiple instance cancer detection by boosting regularised trees , 2015 .

[90]  Heng-Da Cheng,et al.  Breast Ultrasound Image Classification Based on Multiple-Instance Learning , 2012, Journal of Digital Imaging.

[91]  Zhi-Hua Zhou,et al.  Multi-instance learning by treating instances as non-I.I.D. samples , 2008, ICML '09.

[92]  Baoxin Li,et al.  Classification of diabetic retinopathy images using multi-class multiple-instance learning based on color correlogram features , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[93]  Melih Kandemir,et al.  Digital pathology: Multiple instance learning can detect Barrett's cancer , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[94]  Yuhong Guo.  Max-margin Multiple-Instance Learning via Semidefinite Programming , 2009, ACML.

[95]  Melih Kandemir,et al.  Detection of Retinopathy of Prematurity using multiple instance learning , 2015, 2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[96]  Jiajun Wu,et al.  Deep multiple instance learning for image classification and auto-annotation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Lei Zhang,et al.  Mammogram microcalcification cluster detection by locating key instances in a Multi-Instance Learning framework , 2012, 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2012).

[98]  Boris Babenko.  Multiple Instance Learning: Algorithms and Applications , 2008 .

[99]  Gwénolé Quellec,et al.  Automatic detection of referral patients due to retinal pathologies through data mining , 2016, Medical Image Anal..

[100]  Xiang Zhou,et al.  Mining Medical Images , 2009 .

[101]  R. Bharat Rao,et al.  Mining Medical Records for Computer Aided Diagnosis , .

[102]  Hong Gu,et al.  Bayesian multi-instance multi-label learning using Gaussian process prior , 2012, Machine Learning.

[103]  Lauge Sørensen,et al.  Label Stability in Multiple Instance Learning , 2015, MICCAI.

[104]  Gwénolé Quellec,et al.  Real-Time Task Recognition in Cataract Surgery Videos Using Adaptive Spatiotemporal Polynomials , 2015, IEEE Transactions on Medical Imaging.

[105]  Baoxin Li,et al.  A machine-learning approach to retrieving diabetic retinopathy images , 2012, BCB '12.

[106]  Metin Nafi Gürcan,et al.  Computerized classification of intraductal breast lesions using histopathological images , 2011, IEEE Transactions on Biomedical Engineering.

[107]  Jian Yin,et al.  Augmenting Multi-Instance Multilabel Learning with Sparse Bayesian Models for Skin Biopsy Image Analysis , 2014, BioMed research international.

[108]  Dehua Zhao,et al.  Automated Classification of Human Histological Images, A Multiple-Instance Learning Approach , 2006, 2006 IEEE/NLM Life Science Systems and Applications Workshop.

[109]  Gwénolé Quellec,et al.  Multimedia data mining for automatic diabetic retinopathy screening , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[110]  Thomas Hofmann,et al.  Multiple Instance Learning for Computer Aided Diagnosis , 2007 .

[111]  Damon Wing Kee Wong,et al.  Effective Drusen Localization for Early AMD Screening using Sparse Multiple Instance Learning , 2015 .

[112]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[113]  Yixin Chen,et al.  Image Categorization by Learning and Reasoning with Regions , 2004, J. Mach. Learn. Res..

[114]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..

[115]  Yan Xu,et al.  Deep learning of feature representation with multiple instance learning for medical image analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[116]  Jaume Amores,et al.  Multiple instance classification: Review, taxonomy and comparative study , 2013, Artif. Intell..

[117]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.