A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery
Top Cited Papers
Open Access
- 4 January 2017
- journal article
- video audio-media
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biomedical Engineering
- Vol. 64 (9), 2025-2041
- https://doi.org/10.1109/tbme.2016.2647680
Abstract
Objective : State-of-the-art techniques for surgical data analysis report promising results for automated skill assessment and action recognition. The contributions of many of these techniques, however, are limited to study-specific data and validation metrics, making assessment of progress across the field extremely challenging. Methods : In this paper, we address two major problems for surgical data analysis: First, lack of uniform-shared datasets and benchmarks, and second, lack of consistent validation processes. We address the former by presenting the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), a public dataset that we have created to support comparative research benchmarking. JIGSAWS contains synchronized video and kinematic data from multiple performances of robotic surgical tasks by operators of varying skill. We address the latter by presenting a well-documented evaluation methodology and reporting results for six techniques for automated segmentation and classification of time-series data on JIGSAWS. These techniques comprise four temporal approaches for joint segmentation and classification: hidden Markov model, sparse hidden Markov model (HMM), Markov semi-Markov conditional random field, and skip-chain conditional random field; and two feature-based ones that aim to classify fixed segments: bag of spatiotemporal features and linear dynamical systems. Results : Most methods recognize gesture activities with approximately 80% overall accuracy under both leave-one-super-trial-out and leave-one-user-out cross-validation settings. Conclusion : Current methods show promising results on this shared dataset, but room for significant progress remains, particularly for consistent prediction of gesture activities across different surgeons. Significance : The results reported in this paper provide the first systematic and uniform evaluation of surgical activity recognition techniques on the benchmark database.Keywords
Funding Information
- NIH (1R01-DE025265)
- NSF (0534359, IIS-0748338, OIA 0941362, CSN 0931805)
- Sloan Foundation
- NSF Graduate Research Fellowship Program
- Talentia Fellowship Program of the Andalusian Regional Ministry of Economy, Innovation and Science
- NSF (DGE-1232825)
- European Research Council (VideoWorld)
This publication has 62 references indexed in Scilit:
- Cutting-plane training of structural SVMsMachine Learning, 2009
- $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse RepresentationIEEE Transactions on Signal Processing, 2006
- Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motionsComputer Aided Surgery, 2006
- Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motionsComputer Aided Surgery, 2006
- Task decomposition of laparoscopic surgery for objective evaluation of surgical residents' learning curve using hidden Markov ModelComputer Aided Surgery, 2002
- Task Decomposition of Laparoscopic Surgery for Objective Evaluation of Surgical Residents' Learning Curve Using Hidden Markov ModelComputer Aided Surgery, 2002
- Assessment of surgical competenceHeart, 2001
- A metric for ARMA processesIEEE Transactions on Signal Processing, 2000
- Objective structured assessment of technical skill (OSATS) for surgical residentsBritish Journal of Surgery, 1997
- A tutorial on hidden Markov models and selected applications in speech recognitionProceedings of the IEEE, 1989