Multi-Task Joint Learning Model for Segmenting and Classifying Tongue Images Using a Deep Neural Network

Abstract

Automatic tongue image segmentation and tongue classification are two crucial tongue characterization tasks in traditional Chinese medicine (TCM). Due to the complexity of automatic tongue segmentation and the fine-grained traits of tongue classification, both tasks are challenging. However, as discussed in the introduction section, these two tasks are interrelated, making them highly compatible with the idea of multitask joint learning (MTL). By sharing the underlying parameters and adding two different task objective functions, a MTL method for segmenting and classifying tongue images is proposed in this paper. Moreover, two state-of-the-art deep neural network variants (UNET and discriminative filter bank (DFL)) are fused into the MTL to perform the tongue segmentation and tongue classification tasks, respectively. To the best of our knowledge, our method is the first attempt to manage both tasks simultaneously with MTL. We conducted extensive experiments on reliable and quality assured datasets. The experimental results show that our joint method outperforms both the existing tongue segmentation methods and the existing tongue classification methods. Visualizations and ablation studies are provided to aid in understanding our approach, which suggest that our method is highly consistent with human perception