Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment
- 3 April 2018
- journal article
- research article
- Published by Georg Thieme Verlag KG in Endoscopy
- Vol. 50 (08), 770-778
- https://doi.org/10.1055/a-0576-6667
Abstract
Background Direct Observation of Procedural Skills (DOPS) is an established competence assessment tool in endoscopy. In July 2016, the DOPS scoring format changed from a performance-based scale to a supervision-based scale. We aimed to evaluate the impact of changes to the DOPS scale format on the distribution of scores in novice trainees and on competence assessment. Methods We performed a prospective, multicenter (n = 276), observational study of formative DOPS assessments in endoscopy trainees with ≤ 100 lifetime procedures. DOPS were submitted in the 6-months before July 2016 (old scale) and after (new scale) for gastroscopy (n = 2998), sigmoidoscopy (n = 1310), colonoscopy (n = 3280), and polypectomy (n = 631). Scores for old and new DOPS were aligned to a 4-point scale and compared. Results 8219 DOPS (43 % new and 57 % old) submitted for 1300 trainees were analyzed. Compared with old DOPS, the use of the new DOPS was associated with greater utilization of the lowest score (2.4 % vs. 0.9 %; P < 0.001), broader range of scores, and a reduction in competent scores (60.8 % vs. 86.9 %; P < 0.001). The reduction in competent scores was evident on subgroup analysis across all procedure types (P < 0.001) and for each quartile of endoscopy experience. The new DOPS was superior in characterizing the endoscopy learning curve by demonstrating progression of competent scores across quartiles of procedural experience. Conclusions Endoscopy assessors applied a greater range of scores using the new DOPS scale based on degree of supervision in two cohorts of trainees matched for experience. Our study provides construct validity evidence in support of the new scale format.Keywords
This publication has 11 references indexed in Scilit:
- Scale format effects on response option interpretation and useJournal of Business Research, 2016
- The learning curve to achieve satisfactory completion rates in upper GI endoscopy: an analysis of a national training databaseGut, 2016
- An analysis of the learning curve to achieve competency at colonoscopy using the JETS databaseGut, 2014
- The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopistsGastrointestinal Endoscopy, 2012
- Development and validation of a novel method for assessing competency in polypectomy: direct observation of polypectomy skillsGastrointestinal Endoscopy, 2011
- Good questions, good answers: construct alignment improves the performance of workplace-based assessment scalesMedical Education, 2011
- Development and roll out of the JETS e-portfolio: a web based electronic portfolio for endoscopistsFrontline Gastroenterology, 2010
- The Acute Care Assessment Tool: a new assessment in acute medicineThe Clinical Teacher, 2009
- Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and ApplicationAmerican Journal Of Medicine, 2006
- Scale Development: Theory and Applications.Contemporary Sociology: A Journal of Reviews, 1992