Crowd-Sourced Assessment of Technical Skills: Differentiating Animate Surgical Skill Through the Wisdom of Crowds
- 1 October 2015
- journal article
- research article
- Published by Mary Ann Liebert Inc in Journal of Endourology
- Vol. 29 (10), 1183-1188
- https://doi.org/10.1089/end.2015.0104
Abstract
Background: Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing. Methods: Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement. Results: Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. Conclusion: Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.Keywords
This publication has 12 references indexed in Scilit:
- Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performanceJournal of Surgical Research, 2013
- Surgical Skill and Complication Rates after Bariatric SurgeryNew England Journal of Medicine, 2013
- Future of Robotic SurgeryThe Cancer Journal, 2013
- Content and Construct Validation of a Robotic Surgery Curriculum Using an Electromagnetic Instrument TrackerJournal of Urology, 2012
- Global Evaluative Assessment of Robotic Skills: Validation of a Clinical Assessment Tool to Measure Robotic Surgical SkillsJournal of Urology, 2012
- Objective assessment of technical surgical skillsBritish Journal of Surgery, 2010
- The surgical efficiency score: a feasible, reliable, and valid method of skills assessmentThe American Journal of Surgery, 2006
- Analysis of surgical errors in closed malpractice claims at 4 liability insurersSurgery, 2006
- My Current Thoughts on Coefficient Alpha and Successor ProceduresEducational and Psychological Measurement, 2004
- Excess Length of Stay, Charges, and Mortality Attributable to Medical Injuries During HospitalizationJAMA, 2003