Multi-Center Colonoscopy Quality Measurement Utilizing Natural Language Processing
- 1 April 2015
- journal article
- research article
- Published by Ovid Technologies (Wolters Kluwer Health) in The American Journal of Gastroenterology
- Vol. 110 (4), 543-552
- https://doi.org/10.1038/ajg.2015.51
Abstract
An accurate system for tracking of colonoscopy quality and surveillance intervals could improve the effectiveness and cost-effectiveness of colorectal cancer (CRC) screening and surveillance. The purpose of this study was to create and test such a system across multiple institutions utilizing natural language processing (NLP). From 42,569 colonoscopies with pathology records from 13 centers, we randomly sampled 750 paired reports. We trained (n=250) and tested (n=500) an NLP-based program with 19 measurements that encompass colonoscopy quality measures and surveillance interval determination, using blinded, paired, annotated expert manual review as the reference standard. The remaining 41,819 nonannotated documents were processed through the NLP system without manual review to assess performance consistency. The primary outcome was system accuracy across the 19 measures. A total of 176 (23.5%) documents with 252 (1.8%) discrepant content points resulted from paired annotation. Error rate within the 500 test documents was 31.2% for NLP and 25.4% for the paired annotators (P=0.001). At the content point level within the test set, the error rate was 3.5% for NLP and 1.9% for the paired annotators (P=0.04). When eight vaguely worded documents were removed, 125 of 492 (25.4%) were incorrect by NLP and 104 of 492 (21.1%) by the initial annotator (P=0.07). Rates of pathologic findings calculated from NLP were similar to those calculated by annotation for the majority of measurements. Test set accuracy was 99.6% for CRC, 95% for advanced adenoma, 94.6% for nonadvanced adenoma, 99.8% for advanced sessile serrated polyps, 99.2% for nonadvanced sessile serrated polyps, 96.8% for large hyperplastic polyps, and 96.0% for small hyperplastic polyps. Lesion location showed high accuracy (87.0-99.8%). Accuracy for number of adenomas was 92%. NLP can accurately report adenoma detection rate and the components for determining guideline-adherent colonoscopy surveillance intervals across multiple sites that utilize different methods for reporting colonoscopy findings.Keywords
This publication has 34 references indexed in Scilit:
- Validation of Case Finding Algorithms for Hepatocellular Cancer From Administrative Data and Electronic Health Records Using Natural Language ProcessingMedical Care, 2016
- Automated Detection Using Natural Language Processing of Radiologists Recommendations for Additional Imaging of Incidental FindingsAnnals of Emergency Medicine, 2013
- Assessment of adenoma detection rate benchmarks in women versus menGastrointestinal Endoscopy, 2013
- Modifiable endoscopic factors that influence the adenoma detection rate in colorectal cancer screening colonoscopiesGastrointestinal Endoscopy, 2012
- A study of transportability of an existing smoking status detection module across institutions.2012
- Serrated Lesions of the Colorectum: Review and Recommendations From an Expert PanelThe American Journal of Gastroenterology, 2012
- Guidelines for Colonoscopy Surveillance After Screening and Polypectomy: A Consensus Update by the US Multi-Society Task Force on Colorectal CancerGastroenterology, 2012
- The effect of colonoscopy preparation quality on adenoma detection ratesGastrointestinal Endoscopy, 2012
- Developing a natural language processing application for measuring the quality of colonoscopy proceduresJournal of the American Medical Informatics Association, 2011
- Data Resources in the Department of Veterans AffairsDiabetes Care, 2004