Self-Supervised Contrastive Video-Speech Representation Learning for Ultrasound
- 29 September 2020
- book chapter
- conference paper
- Published by Springer Science and Business Media LLC
- Vol. 12263, 534-543
- https://doi.org/10.1007/978-3-030-59716-0_51
Abstract
In medical imaging, manual annotations can be expensive to acquire and sometimes infeasible to access, making conventional deep learning-based models difficult to scale. As a result, it would be beneficial if useful representations could be derived from raw data without the need for manual annotations. In this paper, we propose to address the problem of self-supervised representation learning with multi-modal ultrasound video-speech raw data. For this case, we assume that there is a high correlation between the ultrasound video and the corresponding narrative speech audio of the sonographer. In order to learn meaningful representations, the model needs to identify such correlation and at the same time understand the underlying anatomical features. We designed a framework to model the correspondence between video and audio without any kind of human annotations. Within this framework, we introduce cross-modal contrastive learning and an affinity-aware self-paced learning scheme to enhance correlation modelling. Experimental evaluations on multi-modal fetal ultrasound video and audio show that the proposed approach is able to learn strong representations and transfers well to downstream tasks of standard plane detection and eye-gaze prediction.Keywords
This publication has 13 references indexed in Scilit:
- Audio-Visual Scene Analysis with Self-Supervised Multisensory FeaturesPublished by Springer Science and Business Media LLC ,2018
- Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese NetworksPublished by Springer Science and Business Media LLC ,2018
- What Do Different Evaluation Metrics Tell Us About Saliency Models?IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Unsupervised Representation Learning by Sorting SequencesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Look, Listen and LearnPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand UltrasoundIEEE Transactions on Medical Imaging, 2017
- Aggregated Residual Transformations for Deep Neural NetworksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2017
- Colorful Image ColorizationPublished by Springer Science and Business Media LLC ,2016
- Unsupervised Learning of Visual Representations Using VideosPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015