Crowd-Sourced Assessment of Technical Skill: A Valid Method for Discriminating Basic Robotic Surgery Skills
- 1 November 2015
- journal article
- Published by Mary Ann Liebert Inc in Journal of Endourology
- Vol. 29 (11), 1295-1301
- https://doi.org/10.1089/end.2015.0191
Abstract
A surgeon's skill in the operating room has been shown to correlate with a patient's clinical outcome. The prompt accurate assessment of surgical skill remains a challenge, in part, because expert faculty reviewers are often unavailable. By harnessing the power of large readily available crowds through the Internet, rapid, accurate, and low-cost assessments may be achieved. We hypothesized that assessments provided by crowd workers highly correlate with expert surgeons' assessments. A group of 49 surgeons from two hospitals performed two dry-laboratory robotic surgical skill assessment tasks. The performance of these tasks was video recorded and posted online for evaluation using Amazon Mechanical Turk. The surgical tasks in each video were graded by (n=30) varying crowd workers and (n=3) experts using a modified global evaluative assessment of Robotic Skills (GEARS) grading tool, and the mean scores were compared using Cronbach's alpha statistic. GEARS evaluations from the crowd were obtained for each video and task and compared with the GEARS ratings from the expert surgeons. The crowd-based performance scores agreed with the performance assessments by experts with a Cronbach's alpha of 0.84 and 0.92 for the two tasks, respectively. The assessment of surgical skill by crowd workers resulted in a high degree of agreement with the scores provided by expert surgeons in the evaluation of basic robotic surgical dry-laboratory tasks. Crowd responses cost less and were much faster to acquire. This study provides evidence that crowds may provide an adjunctive method for rapidly providing feedback of skills to training and practicing surgeons.Keywords
This publication has 18 references indexed in Scilit:
- Beyond task time: automated measurement augments fundamentals of laparoscopic skills methodologyJournal of Surgical Research, 2014
- Surgical Skill and Complication Rates after Bariatric SurgeryNew England Journal of Medicine, 2013
- Global Evaluative Assessment of Robotic Skills: Validation of a Clinical Assessment Tool to Measure Robotic Surgical SkillsJournal of Urology, 2012
- Review of methods for objective surgical skill evaluationSurgical Endoscopy, 2010
- Anterior Perineal Plane for Ultra-low Anterior Resection of the Rectum (The APPEAR Technique): A Prospective Clinical Trial of a New ProcedureAnnals of Surgery, 2009
- HMM assessment of quality of movement trajectory in laparoscopic surgeryComputer Aided Surgery, 2007
- Generalized Approach for Modeling Minimally Invasive Surgery as a Stochastic Process Using a Discrete Markov ModelIEEE Transactions on Biomedical Engineering, 2006
- Excess Length of Stay, Charges, and Mortality Attributable to Medical Injuries During HospitalizationJAMA, 2003
- Metrics for objective AssessmentSurgical Endoscopy, 2003
- Objective structured assessment of technical skill (OSATS) for surgical residentsBritish Journal of Surgery, 1997