Psychometric characteristics of simulation‐based assessment in anaesthesia and accuracy of self‐assessed scores*
- 14 February 2005
- journal article
- Published by Wiley in Anaesthesia
- Vol. 60 (3), 245-250
- https://doi.org/10.1111/j.1365-2044.2004.04073.x
Abstract
The purpose of this study was to define the psychometric properties of a simulation-based assessment of anaesthetists. Twenty-one anaesthetic trainees took part in three highly standardised simulations of anaesthetic emergencies. Scenarios were videotaped and rated independently by four judges. Trainees also assessed their own performance in the simulations. Results were analysed using generalisability theory to determine the influence of subject, case and judge on the variance in judges' scores and to determine the number of cases and judges required to produce a reliable result. Self-assessed scores were compared to the mean score of the judges. The results suggest that 12-15 cases are required to rank trainees reliably on their ability to manage simulated crises. Greater reliability is gained by increasing the number of cases than by increasing the number of judges. There was modest but significant correlation between self-assessed scores and external assessors' scores (rho = 0.321; p = 0.01). At the lower levels of performance, trainees consistently overrated their performance compared to those performing at higher levels (p = 0.0001).Keywords
This publication has 21 references indexed in Scilit:
- Reliability and Validity of a Simulation-based Acute Care Skills Assessment for Medical Students and ResidentsAnesthesiology, 2003
- Evaluation of high fidelity patient simulator in assessment of performance of anaesthetistsBritish Journal of Anaesthesia, 2003
- Generalisability: a key to unlock professional assessmentMedical Education, 2002
- The Validity of Performance Assessments Using SimulationAnesthesiology, 2001
- Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examinationAcademic Medicine, 1998
- Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral RatingsAnesthesiology, 1998
- Testing Internal Consistency and Construct Validity During Evaluation of Performance in a Patient SimulatorAnesthesia & Analgesia, 1998
- Testing the raters: inter-rater reliability of standardized anaesthesia simulator performanceCanadian Journal of Anesthesia/Journal canadien d'anesthésie, 1997
- Responses to simulated anaesthetic emergencies by anaesthetists with different durations of clinical experienceBritish Journal of Anaesthesia, 1997
- Psychometric characteristics of the objective structured clinical examinationMedical Education, 1988