A standard setting method with the best performing students as point of reference: Practical and affordable
- 1 January 2010
- journal article
- research article
- Published by Taylor & Francis Ltd in Medical Teacher
- Vol. 32 (2), 154-160
- https://doi.org/10.3109/01421590903196979
Abstract
Background: Teachers involved in test development usually prefer criterion-referenced standard setting methods using panels. Since expert panels are costly, standards are often set by a pre-fixed percentage of questions answered correctly or norm-referenced methods aimed at ranking examinees. Aim: To discuss the (dis)advantages of commonly used criterion and norm-referenced methods and present a new compromise method: standards based on a fixed cut-off score using the best scoring students as reference point. Methods: Historical data from 54 Maastricht (norm-referenced) and 52 Groningen (criterion-referenced) tests were used to demonstrate huge discrepancies and variability in cut-off scores and failure rates. Subsequently, the compromise model – known as Cohen's method – was applied to the Groningen tests. Results: The Maastricht norm-referenced method led to a large variation in required cut-off scores (15–46%), but a stable failure rate (about 17%). The Groningen method with a conventional, pre-fixed standard of 60% led to a large variation in failure rates (17–97%). The compromise method reduced variation in required cut-off scores as well as failure rates. Conclusion: Both the criterion and norm-referenced standards, used in practice, have disadvantages. The proposed compromise model reduces the disadvantages of both methods and is considered more acceptable. Last but not least, compared to standard setting methods using panels, this method is affordable.Keywords
This publication has 18 references indexed in Scilit:
- Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37Medical Teacher, 2008
- Benchmarking by cross-institutional comparison of student achievement in a progress testMedical Education, 2007
- A Conceptual Framework for a Psychometric Theory for Standard Setting with Examples of Its Use for Evaluating the Functioning of Two Standard Setting MethodsEducational Measurement: Issues and Practice, 2006
- RESEARCH METHODOLOGY: Procedures for Establishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions EducationTeaching and Learning in Medicine, 2006
- The Effects of Violating Standard Item Writing Principles on Tests and Students: The Consequences of Using Flawed Test Items on Achievement Examinations in Medical EducationAdvances in Health Sciences Education, 2005
- Combining Tests and Setting StandardsPublished by Springer Science and Business Media LLC ,2002
- AMEE Guide No. 18: Standard setting in student assessmentMedical Teacher, 2000
- Student assessment and examination rulesMedical Teacher, 1999
- Relative or Absolute Standards in Assessing Medical Knowledge Using Progress Tests.Advances in Health Sciences Education, 1998
- The Credibility and Comparability of StandardsApplied Measurement in Education, 1997