Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions
Open Access
- 20 April 2020
- journal article
- research article
- Published by Korea Health Personnel Licensing Examination Institute in Journal of Educational Evaluation for Health Professions
- Vol. 17, 12
- https://doi.org/10.3352/jeehp.2020.17.12
Abstract
Purpose: It aimed to know the performance of the Ebel standard-setting method in in spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisted of multiple-choice questions. Specifically followings were searched: the inter-rater agreement; the correlation between Ebel scores and item facility indices; raters' knowledge of correct answers' impact on the Ebel score; and affection of rater's specialty on theinter-rater agreement and Ebel scores. Methods: Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. Ebel's method was applied to 203 MCQs by 49 raters. Facility indices came from 194 candidates. We computed Fleiss' kappa and the Pearson correlation between Ebel scores and item facility indices. We investigated differences in the Ebel score (correct answers provided or not) and differences between internists and other specialists with t-tests. Results: Kappa was below 0.15 for facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same, whether the correct answers were provided or not. Inter-rater agreement and Ebel scores was not differentbetween internists and other specialists. Conclusion: Inter-rater agreement and correlations between item Ebel scores and facility indices wee consistently low; furthermore, raters' knowledge of correct answer and rater specialty had no effect on Ebel scores in the present setting.Keywords
This publication has 11 references indexed in Scilit:
- Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and HofsteeJournal of Educational Evaluation for Health Professions, 2018
- Systematic review to establish absolute standards for technical performance in surgeryBritish Journal of Surgery, 2016
- Setting standards in knowledge assessments: Comparing Ebel and Cohen via RaschMedical Teacher, 2016
- Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionalityAssessment & Evaluation in Higher Education, 2012
- RESEARCH METHODOLOGY: Procedures for Establishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions EducationTeaching and Learning in Medicine, 2006
- Establishing Passing Standards for Classroom Achievement Tests in Medical Education: A Comparative Study of Four MethodsAcademic Medicine, 2003
- Setting standards on educational testsMedical Education, 2003
- Standard setting in medical educationAcademic Medicine, 1996
- Setting content-based standards for national board examsAcademic Medicine, 1990
- The Measurement of Observer Agreement for Categorical DataBiometrics, 1977