Self-Supervised Longitudinal Neighbourhood Embedding

Longitudinal MRIs are often used to capture the gradual deterioration of brain structure and function caused by aging or neurological diseases. Analyzing this data via machine learning generally requires a large number of ground-truth labels, which are often missing or expensive to obtain. Reducing the need for labels, we propose a self-supervised strategy for representation learning named Longitudinal Neighborhood Embedding (LNE). Motivated by concepts in contrastive learning, LNE explicitly models the similarity between trajectory vectors across different subjects. We do so by building a graph in each training iteration defining neighborhoods in the latent space so that the progression direction of a subject follows the direction of its neighbors. This results in a smooth trajectory field that captures the global morphological change of the brain while maintaining the local continuity. We apply LNE to longitudinal T1w MRIs of two neuroimaging studies: a dataset composed of 274 healthy subjects, and Alzheimer’s Disease Neuroimaging Initiative (ADNI, N = 632). The visualization of the smooth trajectory vector field and superior performance on downstream tasks demonstrate the strength of the proposed method over existing self-supervised methods in extracting information associated with normal aging and in revealing impact of neurodegenerative disorders. The code is available at https://github. com/ouyangjiahong/longitudinal-neighbourhood-embedding.

[1]  Jun Wang,et al.  Deep Multi-modal Latent Representation Learning for Automated Dementia Diagnosis , 2019, MICCAI.

[2]  Daoqiang Zhang,et al.  Label-aligned multi-task feature learning for multimodal classification of Alzheimer’s disease and mild cognitive impairment , 2015, Brain Imaging and Behavior.

[3]  Phillip Isola,et al.  Contrastive Multiview Coding , 2019, ECCV.

[4]  V Dalca Adrian,et al.  Unsupervised Deep Learning for Bayesian Brain MRI Segmentation , 2019 .

[5]  Seong-Whan Lee,et al.  Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis , 2014, NeuroImage.

[6]  Stephan Günnemann,et al.  Diffusion Improves Graph Learning , 2019, NeurIPS.

[7]  Xiahai Zhuang,et al.  Diagnosis of Alzheimer’s Disease via Multi-Modality 3D Convolutional Neural Network , 2019, Front. Neurosci..

[8]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[9]  Kilian M. Pohl,et al.  Longitudinal Pooling & Consistency Regularization to Model Disease Progression From MRIs , 2021, IEEE Journal of Biomedical and Health Informatics.

[10]  Dinggang Shen,et al.  Latent Representation Learning for Alzheimer’s Disease Diagnosis With Incomplete Multi-Modality Neuroimaging and Genetic Data , 2019, IEEE Transactions on Medical Imaging.

[11]  Kaveh Hassani,et al.  Contrastive Multi-View Representation Learning on Graphs , 2020, ICML.

[12]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  M. Toepper,et al.  Dissociating Normal Aging from Alzheimer’s Disease: A View from Cognitive Neuroscience , 2017, Journal of Alzheimer's disease : JAD.

[15]  Mert R. Sabuncu,et al.  An Unsupervised Learning Model for Deformable Medical Image Registration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Jie Tang,et al.  Self-Supervised Learning: Generative or Contrastive , 2020, IEEE Transactions on Knowledge and Data Engineering.

[17]  Yong Fan,et al.  Non-rigid image registration using self-supervised fully convolutional networks without training data , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[18]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[19]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[20]  Kilian M. Pohl,et al.  Longitudinal self-supervised learning , 2021, Medical Image Anal..

[21]  Di Guo,et al.  Convolutional Neural Networks-Based MRI Image Analysis for the Alzheimer’s Disease Prediction From Mild Cognitive Impairment , 2018, Front. Neurosci..

[22]  Snehashis Roy,et al.  Longitudinal multiple sclerosis lesion segmentation: Resource and challenge , 2017, NeuroImage.

[23]  Mert R. Sabuncu,et al.  Unsupervised deep learning for Bayesian brain MRI segmentation , 2019, MICCAI.

[24]  Lloyd T. Elliott,et al.  Brain aging comprises many modes of structural and functional change with distinct genetic and biophysical associations , 2019, bioRxiv.

[25]  Daniel Rueckert,et al.  Multi-region analysis of longitudinal FDG-PET for the classification of Alzheimer's disease , 2012, NeuroImage.

[26]  Tanya P. Garcia,et al.  Statistical Approaches to Longitudinal Data Analysis in Neurodegenerative Diseases: Huntington’s Disease as a Model , 2017, Current Neurology and Neuroscience Reports.

[27]  Alzheimer's Disease Neuroimaging Initiative,et al.  Predicting Alzheimer’s conversion in mild cognitive impairment patients using longitudinal neuroimaging and clinical markers , 2020, Brain Imaging and Behavior.

[28]  Kilian M. Pohl,et al.  LSSL: Longitudinal Self-Supervised Learning , 2020, ArXiv.

[29]  Kaiming He,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Manhua Liu,et al.  RNN-based longitudinal analysis for diagnosis of Alzheimer's disease , 2019, Comput. Medical Imaging Graph..

[31]  J. Whitwell,et al.  Longitudinal imaging: change and causality , 2008, Current opinion in neurology.

[32]  Ehsan Adeli,et al.  Self-Supervised Representation Learning via Neighborhood-Relational Encoding , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[33]  Dinggang Shen,et al.  Landmark‐based deep multi‐instance learning for brain disease diagnosis , 2018, Medical Image Anal..

[34]  Mert R. Sabuncu,et al.  VoxelMorph: A Learning Framework for Deformable Medical Image Registration , 2018, IEEE Transactions on Medical Imaging.

[35]  Alexander Kolesnikov,et al.  Revisiting Self-Supervised Visual Representation Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Julien Mairal,et al.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments , 2020, NeurIPS.