Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?
- 7 September 2006
- journal article
- Published by Springer Science and Business Media LLC in Advances in Health Sciences Education
- Vol. 13 (1), 11-24
- https://doi.org/10.1007/s10459-006-9027-1
Abstract
Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers, respectively. Data collected has been analysed to investigate whether reliable standards can be achieved for small-scale, medical school-based assessments, to establish the minimum number of judges required and the effect of a discussion phase on reliability. The root mean squared error (RMSE) has been used as a measure of reliability and used to compute 95% confidence intervals for comparison to the examination statistics. The RMSE has been used to calculate the minimum number of judges required to obtain a predetermined minimum level of reliability, and the effect of the number of judges and number of items have been examined. Values of the RMSE obtained vary from 0.9 to 2.2%. Using average variances across each paper type, the minimum number of judges to obtain a RMSE of less than 2% is 10 or more judges before discussion or 6 or more judges after discussion. The results indicate that including a discussion phase improves the reliability and reduces the minimum number of judges required. Decision studies indicate that increasing the number of questions included in the assessments would not significantly improve the reliability of the standard setting.Keywords
This publication has 22 references indexed in Scilit:
- How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory StudyEducational and Psychological Measurement, 1999
- Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judgesMedical Education, 1999
- Standard setting in medical educationAcademic Medicine, 1996
- The passing score in the Objective Structured Clinical ExaminationMedical Education, 1996
- Standard Setting: The Next Generation (Where Few Psychometricians Have Gone Before!)Applied Measurement in Education, 1996
- Standard‐Setting GuidelinesEducational Measurement: Issues and Practice, 1996
- METHODOLOGICAL AND PSYCHOMETRIC ISSUES IN SETTING CUTOFF SCORES USING THE ANGOFF METHODPersonnel Psychology, 1991
- A Comparison of Three Variations on a Standard-Setting MethodJournal of Educational Measurement, 1987
- A Consumer’s Guide to Setting Performance Standards on Criterion-Referenced TestsReview of Educational Research, 1986
- A Framework for Viewing the Process of Standard SettingEvaluation & the Health Professions, 1983