Consensus Based Medical Image Segmentation Using Semi-Supervised Learning And Graph Cuts

Medical image segmentation requires consensus ground truth segmentations to be derived from multiple expert annotations. A novel approach is proposed that obtains consensus segmentations from experts using graph cuts (GC) and semi supervised learning (SSL). Popular approaches use iterative Expectation Maximization (EM) to estimate the final annotation and quantify annotator's performance. Such techniques pose the risk of getting trapped in local minima. We propose a self consistency (SC) score to quantify annotator consistency using low level image features. SSL is used to predict missing annotations by considering global features and local image consistency. The SC score also serves as the penalty cost in a second order Markov random field (MRF) cost function optimized using graph cuts to derive the final consensus label. Graph cut obtains a global maximum without an iterative procedure. Experimental results on synthetic images, real data of Crohn's disease patients and retinal images show our final segmentation to be accurate and more consistent than competing methods.

[1]  Wei Li,et al.  Learning Image Context for Segmentation of Prostate in CT-Guided Radiotherapy , 2011, MICCAI.

[2]  Joachim M. Buhmann,et al.  Computational modeling for assessment of IBD: To be or not to be? , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[3]  Dwarikanath Mahapatra,et al.  Semi-supervised learning and graph cuts for consensus based medical image segmentation , 2016, Pattern Recognit..

[4]  B Julesz,et al.  Inability of Humans to Discriminate between Visual Textures That Agree in Second-Order Statistics—Revisited , 1973, Perception.

[5]  O. Commowick,et al.  Incorporating Priors on Expert Performance Parameters for Segmentation Validation and Label Fusion: A Maximum a Posteriori STAPLE , 2010, MICCAI.

[6]  Zhuowen Tu,et al.  Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Mark Craven,et al.  An Analysis of Active Learning Strategies for Sequence Labeling Tasks , 2008, EMNLP.

[8]  Max A. Viergever,et al.  Adaptive local multi-atlas segmentation: Application to the heart and the caudate nucleus , 2010, Medical Image Anal..

[9]  Maria Petrou,et al.  Three-Dimensional Nonlinear Invisible Boundary Detection , 2006, IEEE Transactions on Image Processing.

[10]  Dwarikanath Mahapatra,et al.  Coherency Based Spatio-Temporal Saliency Detection for Video Object Segmentation , 2014, IEEE Journal of Selected Topics in Signal Processing.

[11]  Paul A. Yushkevich,et al.  Guiding Automatic Segmentation with Multiple Manual Segmentations , 2012, MICCAI.

[12]  M. Picco,et al.  Magnetic resonance for assessment of disease activity and severity in ileocolonic Crohn's disease , 2010 .

[13]  Joachim M. Buhmann,et al.  Crohn's disease tissue segmentation from abdominal MRI using semantic information and graph cuts , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[14]  Dwarikanath Mahapatra,et al.  MRF-Based Intensity Invariant Elastic Registration of Cardiac Perfusion Images Using Saliency Information , 2011, IEEE Transactions on Biomedical Engineering.

[15]  Joachim M. Buhmann,et al.  Automatic Detection and Segmentation of Crohn's Disease Tissues From Abdominal MRI , 2013, IEEE Transactions on Medical Imaging.

[16]  Dwarikanath Mahapatra,et al.  Segmentation of optic disc and optic cup in retinal fundus images using shape regression , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[17]  Dwarikanath Mahapatra,et al.  Skull Stripping of Neonatal Brain MRI: Using Prior Shape Information with Graph Cuts , 2012, Journal of Digital Imaging.

[18]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[19]  Dwarikanath Mahapatra,et al.  Landmark Detection in Cardiac MRI Using Learned Local Image Statistics , 2012, STACOM.

[20]  Horst Bischof,et al.  Semi-Supervised Random Forests , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Stefan Klein,et al.  Automatic segmentation of the prostate in 3D MR images by atlas matching using localized mutual information. , 2008, Medical physics.

[22]  Joachim M. Buhmann,et al.  Automatic cardiac RV segmentation using semantic information with graph cuts , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[23]  S. Roy,et al.  Retrieval of MR Kidney Images by Incorporating Shape Information in Histogram of Low Level Features , 2009 .

[24]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Joachim M. Buhmann,et al.  Joint segmentation and groupwise registration of cardiac DCE MRI using sparse data representations , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[26]  Zezhi Chen,et al.  Detecting and Classifying Linear Structures in Mammograms Using Random Forests , 2011, IPMI.

[27]  Joachim M. Buhmann,et al.  Active learning based segmentation of Crohn's disease using principles of visual saliency , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[28]  Joachim M. Buhmann,et al.  Boosting Convolutional Filters with Entropy Sampling for Optic Cup and Disc Image Segmentation from Fundus Images , 2015, MLMI.

[29]  Dwarikanath Mahapatra,et al.  Orientation Histograms as Shape Priors for Left Ventricle Segmentation Using Graph Cuts , 2011, MICCAI.

[30]  Nassir Navab,et al.  Learning from Multiple Experts with Random Forests: Application to the Segmentation of the Midbrain in 3D Ultrasound , 2013, MICCAI.

[31]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Mark W. Schmidt,et al.  Modeling annotator expertise: Learning when everybody knows a bit of something , 2010, AISTATS.

[33]  Daoqiang Zhang,et al.  Multimodal classification of Alzheimer's disease and mild cognitive impairment , 2011, NeuroImage.

[34]  Dwarikanath Mahapatra Groupwise registration of dynamic cardiac perfusion images using temporal dynamics and segmentation information , 2012, Medical Imaging: Image Processing.

[35]  H. Drummond,et al.  Increasing incidence of both juvenile-onset Crohn's disease and ulcerative colitis in Scotland , 2001, European journal of gastroenterology & hepatology.

[36]  Christophe Restif,et al.  Revisiting the Evaluation of Segmentation Results: Introducing Confidence Maps , 2007, MICCAI.

[37]  Paul M. Thompson,et al.  Robust Brain Extraction Across Datasets and Comparison With Publicly Available Methods , 2011, IEEE Transactions on Medical Imaging.

[38]  Joachim M. Buhmann,et al.  A Supervised Learning Based Approach to Detect Crohn's Disease in Abdominal MR Volumes , 2012, Abdominal Imaging.

[39]  Andrew Blake,et al.  Discriminative, Semantic Segmentation of Brain Tissue in MR Images , 2009, MICCAI.

[40]  Joachim M. Buhmann,et al.  Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation , 2017, Comput. Medical Imaging Graph..

[41]  Antonio Criminisi,et al.  Decision Forests for Computer Vision and Medical Image Analysis , 2013, Advances in Computer Vision and Pattern Recognition.

[42]  N. Kingsbury Complex Wavelets for Shift Invariant Analysis and Filtering of Signals , 2001 .

[43]  Paul A. Yushkevich,et al.  Multi-Atlas Segmentation with Joint Label Fusion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Joachim M. Buhmann,et al.  Cardiac LV and RV Segmentation Using Mutual Context Information , 2012, MLMI.

[45]  Dwarikanath Mahapatra,et al.  Joint Registration and Segmentation of Dynamic Cardiac Perfusion Images Using MRFs , 2010, MICCAI.

[46]  Stephen Lin,et al.  Sliding Window and Regression Based Cup Detection in Digital Fundus Images for Glaucoma Diagnosis , 2011, MICCAI.

[47]  Koenraad Van Leemput,et al.  Segmentation of image ensembles via latent atlases , 2010, Medical Image Anal..

[48]  A. Routray,et al.  An Active Snake Model for Classification of Extreme Emotions , 2006, 2006 IEEE International Conference on Industrial Technology.

[49]  Chengjun Liu,et al.  Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition , 2002, IEEE Trans. Image Process..

[50]  Jaap Stoker,et al.  Magnetic resonance imaging for evaluation of disease activity in Crohn’s disease: a systematic review , 2009, European Radiology.

[51]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[52]  Dwarikanath Mahapatra,et al.  Integrating Segmentation Information for Improved MRF-Based Elastic Image Registration , 2012, IEEE Transactions on Image Processing.

[53]  Joachim M. Buhmann,et al.  A Supervised Learning Approach for Crohn's Disease Detection Using Higher-Order Image Statistics and a Novel Shape Asymmetry Measure , 2013, Journal of Digital Imaging.

[54]  Klaus Schunk,et al.  Small Bowel Magnetic Resonance Imaging for Inflammatory Bowel Disease , 2002, Topics in magnetic resonance imaging : TMRI.

[55]  Joachim M. Buhmann,et al.  Computational Pathology: Challenges and Promises for Tissue Analysis , 2015, Comput. Medical Imaging Graph..

[56]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[57]  Hongmin Cai,et al.  Multiparametric tissue characterization of brain neoplasms and their recurrence using pattern classification of MR images. , 2008, Academic radiology.

[58]  Joachim M. Buhmann,et al.  Active learning based segmentation of Crohns disease from abdominal MRI , 2016, Comput. Methods Programs Biomed..

[59]  Hayit Greenspan,et al.  X-ray Categorization and Spatial Localization of Chest Pathologies , 2011, MICCAI.

[60]  RussLL L. Ds Vnlos,et al.  SPATIAL FREQUENCY SELECTIVITY OF CELLS IN MACAQUE VISUAL CORTEX , 2022 .

[61]  Jayanthi Sivaswamy,et al.  Drishti-GS: Retinal image dataset for optic nerve head(ONH) segmentation , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[62]  Jordi Rimola,et al.  Magnetic resonance imaging for evaluation of Crohn's disease: Validation of parameters of severity and quantitative index of activity , 2011, Inflammatory bowel diseases.

[63]  Dwarikanath Mahapatra,et al.  A CNN based neurobiology inspired approach for retinal image quality assessment , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[64]  Ullrich Köthe,et al.  Segmentation of SBFSEM Volume Data of Neural Tissue by Hierarchical Classification , 2008, DAGM-Symposium.

[65]  Max A. Viergever,et al.  Multi-Atlas-Based Segmentation With Local Decision Fusion—Application to Cardiac and Aortic Segmentation in CT Scans , 2009, IEEE Transactions on Medical Imaging.

[66]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[67]  Dwarikanath Mahapatra,et al.  Automatic Eye Type Detection in Retinal Fundus Image Using Fusion of Transfer Learning and Anatomical Features , 2016, 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[68]  Dwarikanath Mahapatra,et al.  Registration of dynamic renal MR images using neurobiological model of saliency , 2008, 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[69]  Mert R. Sabuncu,et al.  A Generative Model for Image Segmentation Based on Label Fusion , 2010, IEEE Transactions on Medical Imaging.

[70]  Daniel Rueckert,et al.  Multi-atlas based segmentation of brain images: Atlas selection and its effect on accuracy , 2009, NeuroImage.

[71]  H. Sebastian Seung,et al.  Selective Sampling Using the Query by Committee Algorithm , 1997, Machine Learning.

[72]  Dwarikanath Mahapatra,et al.  Illumination invariant tracking in office environments using neurobiology-saliency based particle filter , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[73]  Dorin Comaniciu,et al.  Four-Chamber Heart Modeling and Automatic Segmentation for 3-D Cardiac CT Volumes Using Marginal Space Learning and Steerable Features , 2008, IEEE Transactions on Medical Imaging.

[74]  Dwarikanath Mahapatra,et al.  Graph cut based automatic prostate segmentation using learned semantic information , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[75]  Achim Zeileis,et al.  BMC Bioinformatics BioMed Central Methodology article Conditional variable importance for random forests , 2008 .

[76]  Daniel Rueckert,et al.  Automatic anatomical brain MRI segmentation combining label propagation and decision fusion , 2006, NeuroImage.

[77]  M. Sans,et al.  Hydrocolonic sonography for evaluating inflammatory bowel disease. , 2001, AJR. American journal of roentgenology.

[78]  Torsten Rohlfing,et al.  Performance-based classifier combination in atlas-based image segmentation using expectation-maximization parameter estimation , 2004, IEEE Transactions on Medical Imaging.

[79]  Rich Caruana,et al.  An empirical comparison of supervised learning algorithms , 2006, ICML.

[80]  Dwarikanath Mahapatra,et al.  Using Saliency Features for Graphcut Segmentation of Perfusion Kidney Images , 2009 .

[81]  Jitendra Malik,et al.  Learning a classification model for segmentation , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[82]  Joachim M. Buhmann,et al.  A Model Development Pipeline for Crohn's Disease Severity Assessment from Magnetic Resonance Images , 2013, Abdominal Imaging.

[83]  Dwarikanath Mahapatra,et al.  Combining multiple expert annotations using semi-supervised learning and graph cuts for medical image segmentation , 2016, Comput. Vis. Image Underst..

[84]  Abdulmotaleb El-Saddik,et al.  A Real-Time Smart Assistant for Video Surveillance Through Handheld Devices , 2014, ACM Multimedia.

[85]  Shannon L. Risacher,et al.  Identifying AD-Sensitive and Cognition-Relevant Imaging Biomarkers via Joint Classification and Regression , 2011, MICCAI.

[86]  Joel G Fletcher,et al.  Prospective comparison of state-of-the-art MR enterography and CT enterography in small-bowel Crohn's disease. , 2009, AJR. American journal of roentgenology.

[87]  Joachim M. Buhmann,et al.  Crohn's disease segmentation from MRI using learned image priors , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[88]  D. Mahapatra,et al.  Analyzing Training Information From Random Forests for Improved Image Segmentation , 2014, IEEE Transactions on Image Processing.

[89]  Dwarikanath Mahapatra Joint Segmentation and Groupwise Registration of Cardiac Perfusion Images Using Temporal Information , 2012, Journal of Digital Imaging.

[90]  Dwarikanath Mahapatra,et al.  Segmentation of Optic Disc and Optic Cup in Retinal Fundus Images Using Coupled Shape Regression , 2016 .

[91]  P Glasziou,et al.  Meta-analytic methods for diagnostic test accuracy. , 1995, Journal of clinical epidemiology.

[92]  Ignas Budvytis,et al.  Semi-Supervised Video Segmentation Using Tree Structured Graphical Models , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Carlos Ortiz-de-Solorzano,et al.  Combination Strategies in Multi-Atlas Image Segmentation: Application to Brain MR Data , 2009, IEEE Transactions on Medical Imaging.

[94]  Joachim M. Buhmann,et al.  Visual Saliency Based Active Learning for Prostate MRI Segmentation , 2015, MLMI.

[95]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[96]  Dwarikanath Mahapatra,et al.  Cardiac MRI Segmentation Using Mutual Context Information from Left and Right Ventricle , 2013, Journal of Digital Imaging.

[97]  Jerry L. Prince,et al.  Robust Statistical Fusion of Image Labels , 2012, IEEE Transactions on Medical Imaging.

[98]  V. Caselles,et al.  A geometric model for active contours in image processing , 1993 .

[99]  Joel G Fletcher,et al.  Crohn Disease: mural attenuation and thickness at contrast-enhanced CT Enterography--correlation with endoscopic and histologic findings of inflammation. , 2006, Radiology.

[100]  Max A. Viergever,et al.  Label Fusion in Atlas-Based Segmentation Using a Selective and Iterative Method for Performance Level Estimation (SIMPLE) , 2010, IEEE Transactions on Medical Imaging.

[101]  Guido Gerig,et al.  Valmet: A New Validation Tool for Assessing and Improving 3D Object Segmentation , 2001, MICCAI.

[102]  Joachim M. Buhmann,et al.  Localizing and segmenting Crohn's disease affected regions in abdominal MRI using novel context features , 2013, Medical Imaging.

[103]  W. Sandborn,et al.  Inflammatory bowel disease: clinical aspects and established and evolving therapies , 2007, The Lancet.

[104]  Dwarikanath Mahapatra,et al.  Image Quality Classification for DR Screening Using Convolutional Neural Networks , 2016 .

[105]  Simon K Warfield,et al.  Segmentations of MRI images of the female pelvic floor: A study of inter‐ and intra‐reader reliability , 2011, Journal of magnetic resonance imaging : JMRI.

[106]  Joachim M. Buhmann,et al.  A field of experts model for optic cup and disc segmentation from retinal fundus images , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[107]  Juha Koikkalainen,et al.  Fast and robust multi-atlas segmentation of brain magnetic resonance images , 2010, NeuroImage.

[108]  Dwarikanath Mahapatra,et al.  An MRF framework for joint registration and segmentation of natural and perfusion images , 2010, 2010 IEEE International Conference on Image Processing.

[109]  Simon K. Warfield,et al.  Estimating A Reference Standard Segmentation With Spatially Varying Performance Parameters: Local MAP STAPLE , 2012, IEEE Transactions on Medical Imaging.

[110]  Bennett A. Landman,et al.  Robust Statistical Label Fusion Through Consensus Level, Labeler Accuracy, and Truth Estimation (COLLATE) , 2011, IEEE Transactions on Medical Imaging.

[111]  Zhang Li,et al.  Image Registration Based on Autocorrelation of Local Structure , 2016, IEEE Transactions on Medical Imaging.

[112]  Matthew Richardson,et al.  Learning with Knowledge from Multiple Experts , 2003, ICML.

[113]  Dwarikanath Mahapatra,et al.  Cardiac Image Segmentation from Cine Cardiac MRI Using Graph Cuts and Shape Priors , 2013, Journal of Digital Imaging.

[114]  Ying Sun,et al.  Rigid Registration of Renal Perfusion Images Using a Neurobiology-Based Visual Saliency Model , 2010, EURASIP J. Image Video Process..

[115]  Milos Hauskrecht,et al.  Learning classification models from multiple experts , 2013, J. Biomed. Informatics.

[116]  David D. Lewis,et al.  Heterogeneous Uncertainty Sampling for Supervised Learning , 1994, ICML.

[117]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[118]  Dwarikanath Mahapatra,et al.  Nonrigid Registration of Dynamic Renal MR Images Using a Saliency Based MRF Model , 2008, MICCAI.

[119]  Stefan Winkler,et al.  Motion saliency outweighs other low-level features while watching videos , 2008, Electronic Imaging.

[120]  Dwarikanath Mahapatra,et al.  Automatic Cardiac Segmentation Using Semantic Information from Random Forests , 2014, Journal of Digital Imaging.

[121]  Joachim M. Buhmann,et al.  Prostate MRI Segmentation Using Learned Semantic Knowledge and Graph Cuts , 2014, IEEE Transactions on Biomedical Engineering.

[122]  Zhao Liu,et al.  Biological Indexes Based Reflectional Asymmetry for Classifying Cutaneous Lesions , 2011, MICCAI.

[123]  Xiaojin Zhu,et al.  Semi-Supervised Learning , 2010, Encyclopedia of Machine Learning.

[124]  Raymond J. Mooney,et al.  Content-boosted collaborative filtering for improved recommendations , 2002, AAAI/IAAI.

[125]  Joachim M. Buhmann,et al.  Obtaining Consensus Annotations For Retinal Image Segmentation Using Random Forest And Graph Cuts , 2015 .

[126]  Joachim M. Buhmann,et al.  Weakly supervised semantic segmentation of Crohn's disease tissues from abdominal MRI , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[127]  Dwarikanath Mahapatra,et al.  Retinal Image Quality Classification Using Saliency Maps and CNNs , 2016, MLMI@MICCAI.