The Utility of Letters of Recommendation in Predicting Resident Success: Can the ACGME Competencies Help?

Abstract

Background: The Accreditation Council for Graduate Medical Education (ACGME) core competencies are used to assess resident performance, and recently similar competencies have become an accepted framework for evaluating medical student achievements as well. However, the utility of incorporating the competencies into the resident application has not yet been assessed. Purpose: The objective of this study was to examine letters of recommendation (LORs) to identify ACGME competency–based themes that might help distinguish the least successful from the most successful residents. Methods: Residents entering a university-based residency program from 1994 to 2004 were retrospectively evaluated by faculty and ranked in 4 groups according to perceived level of success. Applications from residents in the highest and lowest groups were abstracted. LORs were qualitatively reviewed and analyzed for 9 themes (6 ACGME core competencies and 3 additional performance measures). The mean number of times each theme was mentioned was calculated for each student. Groups were compared using the χ2 test and the Student t test. Results: Seventy-five residents were eligible for analysis, and 29 residents were ranked in the highest and lowest groups. Baseline demographics and number of LORs did not differ between the two groups. Successful residents had statistically significantly more comments about excellence in the competency areas of patient care, medical knowledge, and interpersonal and communication skills. Conclusion: LORs can provide useful clues to differentiate between students who are likely to become the least versus the most successful residency program graduates. Greater usage of the ACGME core competencies within LORs may be beneficial.