The Impact of Judge Selection on Standard Setting for a Patient Survey of Physician Communication Skills
- 1 October 2008
- journal article
- Published by Ovid Technologies (Wolters Kluwer Health) in Academic Medicine
- Vol. 83 (Supplement), S17-S20
- https://doi.org/10.1097/acm.0b013e318183e7bd
Abstract
Patient surveys such as the Communication Assessment Tool (CAT) are recommended to assess physicians. Optimal methods to select judges and set a minimum passing standard (MPS) for a patient survey are unknown. Fifty-eight judges in five groups provided item-based (Angoff) and group-based (Hofstee) judgments for the CAT on two occasions. Judges were communication experts, program directors, trainees, and patients (two groups). Ratings were used to calculate MPSs for the CAT. Interrater reliability and test-retest reliability (stability) were estimated. MPSs were applied to data from a CAT pilot study. Judges produced different MPSs depending on their background. Patients and communication experts were most stringent in their MPS decisions; program directors and trainees were most lenient. Failure rates from the pilot study ranged from 0% to 47%. Judge expertise and background dramatically influenced MPSs for a patient survey of physician communication skills. Judge selection is a key decision when setting an MPS for a clinical skills evaluation.Keywords
This publication has 11 references indexed in Scilit:
- Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practiceJournal of Hospital Medicine, 2008
- Do Baseline Data Influence Standard Setting for a Clinical Skills Examination?Academic Medicine, 2007
- Measuring patient views of physician communication skills: Development and testing of the Communication Assessment ToolPatient Education and Counseling, 2007
- Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practiceJournal of General Internal Medicine, 2006
- RESEARCH METHODOLOGY: Procedures for Establishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions EducationTeaching and Learning in Medicine, 2006
- Comparison of Two Standard-setting Methods for Advanced Cardiac Life Support TrainingAcademic Medicine, 2005
- SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance RatingsTeaching and Learning in Medicine, 2003
- Setting standards on educational testsMedical Education, 2003
- The SEGUE Framework for teaching and assessing communication skillsPatient Education and Counseling, 2001
- Validating the Performance Standards Associated With Passing ScoresReview of Educational Research, 1994