Two-Stream Compare and Contrast Network for Vertebral Compression Fracture Diagnosis

Differentiating Vertebral Compression Fractures (VCFs) associated with trauma and osteoporosis (benign VCFs) or those caused by metastatic cancer (malignant VCFs) are critically important for treatment decisions. So far, automatic VCFs diagnosis is solved in a two-step manner, i.e. first identify VCFs and then classify it into benign or malignant. In this paper, we explore to model VCFs diagnosis as a three-class classification problem, i.e. normal vertebrae, benign VCFs, and malignant VCFs. However, VCFs recognition and classification require very different features, and both tasks are characterized by high intra-class variation and high inter-class similarity. Moreover, the dataset is extremely class-imbalanced. To address the above challenges, we propose a novel Two-Stream Compare and Contrast Network (TSCCN) for VCFs diagnosis. This network consists of two streams, a recognition stream which learns to identify VCFs through comparing and contrasting between adjacent vertebra, and a classification stream which compares and contrasts between intra-class and inter-class to learn features for fine-grained classification. The two streams are integrated via a learnable weight control module which adaptively sets their contribution. The TSCCN is evaluated on a dataset consisting of 239 VCFs patients and achieves the average sensitivity and specificity of 92.56\% and 96.29\%, respectively.

[1]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[2]  Hyunjun Eun,et al.  Single-view 2D CNNs with fully automatic non-nodule categorization for false positive reduction in pulmonary nodule detection , 2018, Comput. Methods Programs Biomed..

[3]  Yuwei Zhang,et al.  Texture and Shape Biased Two-Stream Networks for Clothing Classification and Attribute Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Adam P. Harrison,et al.  Accurate Esophageal Gross Tumor Volume Segmentation in PET/CT using Two-Stream Chained 3D Deep Network Fusion , 2019, MICCAI.

[5]  Bin Yang,et al.  Learning to Reweight Examples for Robust Deep Learning , 2018, ICML.

[6]  Chen Huang,et al.  Learning Deep Representation for Imbalanced Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yang Song,et al.  Class-Balanced Loss Based on Effective Number of Samples , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Yang Song,et al.  Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Timothy F. Cootes,et al.  Classification of Osteoporotic Vertebral Fractures Using Shape and Appearance Modelling , 2017, MSKI@MICCAI.

[10]  Wei Shen,et al.  Multi-crop Convolutional Neural Networks for lung nodule malignancy suspiciousness classification , 2017, Pattern Recognit..

[11]  R. M. Rangayyan,et al.  Classification of vertebral compression fractures in magnetic resonance images using shape analysis , 2015, 2015 E-Health and Bioengineering Conference (EHB).

[12]  H. Cloft,et al.  Review of the Imaging Features of Benign Osteoporotic and Malignant Vertebral Compression Fractures , 2018, American Journal of Neuroradiology.

[13]  Ronald M. Summers,et al.  TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Yong Xia,et al.  Attention Residual Learning for Skin Lesion Classification , 2019, IEEE Transactions on Medical Imaging.

[15]  Dong Wang,et al.  Learning to Navigate for Fine-grained Classification , 2018, ECCV.

[16]  Ramesh Raskar,et al.  Pairwise Confusion for Fine-Grained Visual Classification , 2017, ECCV.

[17]  Nathalie Japkowicz,et al.  The Class Imbalance Problem: Significance and Strategies , 2000 .

[18]  Saeed Hassanpour,et al.  Deep neural networks for automatic detection of osteoporotic vertebral fractures on CT scans , 2018, Comput. Biol. Medicine.

[19]  Nikolas Lessmann,et al.  Iterative fully convolutional neural networks for automatic vertebra segmentation and identification , 2018, Medical Image Anal..

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Rangaraj M. Rangayyan,et al.  Semiautomatic Classification of Benign Versus Malignant Vertebral Compression Fractures Using Texture and Gray-Level Features in Magnetic Resonance Images , 2015, 2015 IEEE 28th International Symposium on Computer-Based Medical Systems.

[22]  Jinwoo Shin,et al.  M2m: Imbalanced Classification via Major-to-Minor Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  C. Cuénod,et al.  Acute vertebral collapse due to osteoporosis or malignancy: appearance on unenhanced and gadolinium-enhanced MR images. , 1996, Radiology.

[24]  K. Hayashi,et al.  Malignant and benign compression fractures: differentiation and diagnostic pitfalls on MRI. , 2004, Clinical radiology.

[25]  Shaogang Gong,et al.  Imbalanced Deep Learning by Minority Class Incremental Rectification , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Jun Zhao,et al.  Vertebrae Identification and Localization Utilizing Fully Convolutional Networks and a Hidden Markov Model , 2020, IEEE Transactions on Medical Imaging.

[27]  REVIEW OF MALIGNANT AND BENIGN FINDINGS OF COMPRESSION VERTEBRAL FRACTURES ON MRI , 2005 .

[28]  Colin Wei,et al.  Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss , 2019, NeurIPS.

[29]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Kun Zhao,et al.  Lung nodule classification by the combination of fusion classifier and cascaded convolutional neural networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[31]  Hui Zhao,et al.  Weakly Supervised Segmentation of Vertebral Bodies with Iterative Slice-Propagation , 2019, DART/MIL3ID@MICCAI.

[32]  Rangaraj M. Rangayyan,et al.  Classification of vertebral compression fractures in magnetic resonance images using spectral and fractal analysis , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[33]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[34]  Jan S. Kirschke,et al.  Grading Loss: A Fracture Grade-based Metric Loss for Vertebral Fracture Detection , 2020, MICCAI.

[35]  Qi Wu,et al.  Skin Lesion Classification in Dermoscopy Images Using Synergic Deep Learning , 2018, MICCAI.

[36]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Seetha Hari,et al.  Learning From Imbalanced Data , 2019, Advances in Computer and Electrical Engineering.

[38]  Lior Wolf,et al.  Compression fractures detection on CT , 2017, Medical Imaging.

[39]  Rangaraj M. Rangayyan,et al.  Recognition of vertebral compression fractures in magnetic resonance images using statistics of height and width , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[40]  A. Hofman,et al.  Review of radiological scoring methods of osteoporotic vertebral fractures for clinical and research settings , 2012, European Radiology.

[41]  Yanfeng Wang,et al.  Spatial Regularized Classification Network for Spinal Dislocation Diagnosis , 2019, MLMI@MICCAI.