CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting

In medical applications, deep learning methods are designed to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents – a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at https://github.com/MIPT-Oulu/CLIMAT.

[1]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[2]  Matthew B. Blaschko,et al.  Semixup: In- and Out-of-Manifold Regularization for Deep Semi-Supervised Knee Osteoarthritis Severity Grading From Plain Radiographs , 2020, IEEE Transactions on Medical Imaging.

[3]  A. Gelber,et al.  Osteoarthritis , 2020, Annals of Internal Medicine.

[4]  Richard Kijowski,et al.  Deep learning risk assessment models for predicting progression of radiographic medial joint space loss over a 48-MONTH follow-up period. , 2020, Osteoarthritis and cartilage.

[5]  A. Mobasheri,et al.  Multi-classifier prediction of knee osteoarthritis progression from incomplete imbalanced longitudinal data , 2019, Scientific Reports.

[6]  Simo Saarakkala,et al.  Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data , 2019, Scientific Reports.

[7]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[8]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[9]  Kevin Gimpel,et al.  Gaussian Error Linear Units (GELUs) , 2016, 1606.08415.

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[12]  J. Bosmans,et al.  Optimizing communication between the radiologist and the general practitioner. , 2013, JBR-BTR : organe de la Societe royale belge de radiologie (SRBR) = orgaan van de Koninklijke Belgische Vereniging voor Radiologie.

[13]  B. Heidari Knee osteoarthritis prevalence, risk factors, pathogenesis and features: Part I. , 2011, Caspian journal of internal medicine.

[14]  H. O. León,et al.  Intercondylar notch stenosis in degenerative arthritis of the knee. , 2005, Arthroscopy : the journal of arthroscopic & related surgery : official publication of the Arthroscopy Association of North America and the International Arthroscopy Association.

[15]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[16]  J. Kellgren,et al.  Radiological Assessment of Osteo-Arthrosis , 1957, Annals of the rheumatic diseases.

[17]  Egill A. Fridgeirsson,et al.  Transformer-Based Deep Survival Analysis , 2021, SPACA.