Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

Computed Tomography (CT) scanners that are commonly-used in hospitals and medical centers nowadays produce low-resolution images, e.g. one voxel in the image corresponds to at most one-cubic millimeter of tissue. In order to accurately segment tumors and make treatment plans, radiologists and oncologists need CT scans of higher resolution. The same problem appears in Magnetic Resonance Imaging (MRI). In this paper, we propose an approach for the single-image super-resolution of 3D CT or MRI scans. Our method is based on deep convolutional neural networks (CNNs) composed of 10 convolutional layers and an intermediate upscaling layer that is placed after the first 6 convolutional layers. Our first CNN, which increases the resolution on two axes (width and height), is followed by a second CNN, which increases the resolution on the third axis (depth). Different from other methods, we compute the loss with respect to the ground-truth high-resolution image right after the upscaling layer, in addition to computing the loss after the last convolutional layer. The intermediate loss forces our network to produce a better output, closer to the ground-truth. A widely-used approach to obtain sharp results is to add Gaussian blur using a fixed standard deviation. In order to avoid overfitting to a fixed standard deviation, we apply Gaussian smoothing with various standard deviations, unlike other approaches. We evaluate the proposed method in the context of 2D and 3D super-resolution of CT and MRI scans from two databases, comparing it to related works from the literature and baselines based on various interpolation schemes, using <inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> scaling factors. The empirical study shows that our approach attains superior results to all other methods. Moreover, our subjective image quality assessment by human observers reveals that both doctors and regular annotators chose our method in favor of Lanczos interpolation in 97.55% cases for an upscaling factor of <inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> and in 96.69% cases for an upscaling factor of <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula>. In order to allow others to reproduce our state-of-the-art results, we provide our code as open source at <uri>https://github.com/lilygeorgescu/3d-super-res-cnn</uri>.