Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch
- 20 September 2016
- journal article
- research article
- Published by Taylor & Francis Ltd in Medical Teacher
- Vol. 38 (12), 1267-1277
- https://doi.org/10.1080/0142159x.2016.1230184
Abstract
Introduction: It is known that test-centered methods for setting standards in knowledge tests (e.g. Angoff or Ebel) are problematic, with expert judges not able to consistently predict the difficulty of individual items. A different approach is the Cohen method, which benchmarks the difficulty of the test based on the performance of the top candidates. Methods: This paper investigates the extent to which Ebel (and also Cohen) produces a consistent standard in a knowledge test when comparing between adjacent cohorts. The two tests are linked using common anchor items and Rasch analysis to put all items and all candidates on the same scale. Results: The two tests are of a similar standard, but the two cohorts are different in their average abilities. The Ebel method is entirely consistent across the two years, but the Cohen method looks less so, whilst the Rasch equating itself has complications – for example, with evidence of overall misfit to the Rasch model and change in difficulty for some anchor items. Conclusion: Based on our findings, we advocate a pluralistic and pragmatic approach to standard setting in such contexts, and recommend the use of multiple sources of information to inform the decision about the correct standard.Keywords
This publication has 36 references indexed in Scilit:
- The misinterpretation of the standard error of measurement in medical education: A primer on the problems, pitfalls and peculiarities of the three different standard errors of measurementMedical Teacher, 2012
- How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49Medical Teacher, 2010
- A standard setting method with the best performing students as point of reference: Practical and affordableMedical Teacher, 2010
- Judges' Use of Examinee Performance Data in an Angoff Standard‐Setting Exercise for a Medical Licensing Examination: An Experimental StudyJournal of Educational Measurement, 2009
- A plea for the proper use of criterion-referenced tests in medical assessmentMedical Education, 2009
- An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting ProcedureApplied Measurement in Education, 2008
- The practical value of the standard error of measurement in borderline passfail decisionsMedical Education, 2008
- The Rasch measurement model in rheumatology: What is it and why use it? When should it be applied, and what should one look for in a Rasch paper?Arthritis Care & Research, 2007
- Standard setting in medical educationAcademic Medicine, 1996
- COMPARABILITY OF METHODS FOR SETTING STANDARDSJournal of Educational Measurement, 1980