Comparison of manual and automatic segmentation methods for brain structures in the presence of space-occupying lesions: a multi-expert study
- 1 July 2011
- journal article
- research article
- Published by IOP Publishing in Physics in Medicine & Biology
- Vol. 56 (14), 4557-4577
- https://doi.org/10.1088/0031-9155/56/14/021
Abstract
The purpose of this work was to characterize expert variation in segmentation of intracranial structures pertinent to radiation therapy, and to assess a registration-driven atlas-based segmentation algorithm in that context. Eight experts were recruited to segment the brainstem, optic chiasm, optic nerves, and eyes, of 20 patients who underwent therapy for large space-occupying tumors. Performance variability was assessed through three geometric measures: volume, Dice similarity coefficient, and Euclidean distance. In addition, two simulated ground truth segmentations were calculated via the simultaneous truth and performance level estimation algorithm and a novel application of probability maps. The experts and automatic system were found to generate structures of similar volume, though the experts exhibited higher variation with respect to tubular structures. No difference was found between the mean Dice similarity coefficient (DSC) of the automatic and expert delineations as a group at a 5% significance level over all cases and organs. The larger structures of the brainstem and eyes exhibited mean DSC of approximately 0.8-0.9, whereas the tubular chiasm and nerves were lower, approximately 0.4-0.5. Similarly low DSCs have been reported previously without the context of several experts and patient volumes. This study, however, provides evidence that experts are similarly challenged. The average maximum distances (maximum inside, maximum outside) from a simulated ground truth ranged from (-4.3, +5.4) mm for the automatic system to (-3.9, +7.5) mm for the experts considered as a group. Over all the structures in a rank of true positive rates at a 2 mm threshold from the simulated ground truth, the automatic system ranked second of the nine raters. This work underscores the need for large scale studies utilizing statistically robust numbers of patients and experts in evaluating quality of automatic algorithms.Keywords
This publication has 37 references indexed in Scilit:
- An atlas-navigated optimal medial axis and deformable model algorithm (NOMAD) for the segmentation of the optic nerves and chiasm in MR and CT imagesMedical Image Analysis, 2011
- Evaluation of Automatic Atlas-Based Lymph Node Segmentation for Head-and-Neck CancerInternational Journal of Radiation Oncology*Biology*Physics, 2010
- An evaluation of four automatic methods of segmenting the subcortical structures in the brainNeuroImage, 2009
- Automatic Segmentation of Whole Breast Using Atlas Approach and Deformable Image RegistrationInternational Journal of Radiation Oncology*Biology*Physics, 2009
- Evaluation of an atlas-based automatic segmentation software for the delineation of brain organs at risk in a radiation therapy clinical contextRadiotherapy and Oncology, 2008
- Evaluation of Lung MDCT Nodule Annotation Across Radiologists and MethodsAcademic Radiology, 2006
- Atlas-Based Segmentation of Pathological MR Brain Images Using a Model of Lesion GrowthIEEE Transactions on Medical Imaging, 2004
- An overlap invariant entropy measure of 3D medical image alignmentPattern Recognition, 1999
- Morphometric analysis of white matter lesions in MR images: method and validationIEEE Transactions on Medical Imaging, 1994
- Measures of the Amount of Ecologic Association Between SpeciesEcology, 1945