Performance improvement of weakly supervised fully convolutional networks by skip connections for brain structure segmentation

Abstract

Purpose For the planning and navigation of neurosurgery, we have developed a fully convolutional network (FCN)-based method for brain structure segmentation on magnetic resonance (MR) images. The capability of an FCN depends on the quality of the training data (i.e., raw data and annotation data) and network architectures. The improvement of annotation quality is a significant concern because it requires much labor for labeling organ regions. To address this problem, we focus on skip connection architectures and reveal which skip connections are effective for training FCNs using sparsely annotated brain images. Methods We tested 2D FCN architectures with four different types of skip connections. The first was a U-Net architecture with horizontal skip connections that transfer feature maps at the same scale from the encoder to the decoder. The second was a U-Net++ architecture with dense convolution layers and dense horizontal skip connections. The third was a full-resolution residual network (FRRN) architecture with vertical skip connections that pass feature maps between each downsampled scale path and the full-resolution scale path. The last one was a hybrid architecture with a combination of horizontal and vertical skip connections. We validated the effect of skip connections on medical image segmentation from sparse annotation based on these four FCN architectures, which were trained under the same conditions. Results For multi-class segmentation of the cerebrum, cerebellum, brainstem, and blood vessels from sparsely annotated MR images, we performed a comparative evaluation of segmentation performance among the above four FCN approaches: U-Net, U-Net++, FRRN, and hybrid architectures. The experimental results show that the horizontal skip connections in the U-Net architectures were effective for the segmentation of larger-sized objects, while the vertical skip connections in the FRRN architecture improved the segmentation of smaller-sized objects. The hybrid architecture with both horizontal and vertical skip connections achieved the best results of the four FCN architectures. We then performed an ablation study to explore which skip connections in the FRRN architecture contributed to the improved segmentation of blood vessels. In the ablation study, we compared the segmentation performance between architectures with a horizontal path (HP), a horizontal path and vertical up paths (HP+VUPs), a horizontal path and vertical down paths (HP+VDPs), and a horizontal path and vertical up and down paths (FRRN). We found that the vertical up paths were effective in improving the segmentation of smaller-sized objects. Conclusions This paper investigated which skip connection architectures were effective for multi-class brain segmentation from sparse annotation. Consequently, using vertical skip connections with horizontal skip connections allowed FCNs to improve segmentation performance. This article is protected by copyright. All rights reserved