Simulating time to event prediction with spatiotemporal echocardiography deep learning

Integrating methods for time-to-event prediction with diagnostic imaging modalities is of considerable interest, as accurate estimates of survival requires accounting for censoring of individuals within the observation period. New methods for time-to-event prediction have been developed by extending the cox-proportional hazards model with neural networks. In this paper, to explore the feasibility of these methods when applied to deep learning with echocardiography videos, we utilize the Stanford EchoNet-Dynamic dataset with over 10,000 echocardiograms, and generate simulated survival datasets based on the expert annotated ejection fraction readings. By training on just the simulated survival outcomes, we show that spatiotemporal convolutional neural networks yield accurate survival estimates. Introduction Recent advances in diagnostic echocardiography AI systems have shown that not only is it possible to automate standard measures of cardiac function (ejection fraction), but even measure common biomarkers that are typically never measured via echocardiography scans (Hemoglobin, blood urea nitrogen).1–3 Our group has shown that predicting post-operative right ventricular failure in the setting of left ventricular assist device implantation is feasible by analyzing the full spatiotemporal density of information within each pre-operative echocardiogram.4 Others have now shown that predicting 1-year mortality is possible using just baseline echocardiography.5 Though this was developed using one of the largest echocardiographic datasets, the work was limited by the use of potentially under-parameterized neural networks, and treatment of right censored survival data as a binary event. Time-to-event predictions rather than simple binary predictions will find more relevance in the clinical setting when working with ‘prognostic’ rather than ‘diagnostic’ AI systems.6 Time-to-event analyses are characterized by the ability to predict event probabilities as a function of time, whereas binary classifiers can only provide predictions for one predetermined duration. One of the advantages of this is that it accounts for censoring of individuals within the observation period.7 This more accurately represents the clinical ground truth where individuals may be either lost to follow up, or have yet to develop an event of interest. Baseline Echocardiogram Numpy Pixel Array) VideoTensor Input (32,3,112,112) 0 5 10 1.905 1.910 1.915 1.920 0 5 10 1.905 1.910 1.915 1.920 0.1240 0.1245 0.1250 0.1255 0.1260 0.1265 mod el-1 mod elmod el-3 ++++++++++++++++++++++++++++ +++ +++++++ + + + 0.00 0.25 0.50 0.75 1.00 0 200 400 600 800 Time 7 x 7 x 7 2x c on v2 _x b lo ck s 2x c on v3 _x b lo ck s 2x c on v4 _x b lo ck s 2x c on v5 _x b lo ck s Av er ag e Po ol in g