EducationDo United States Medical Licensing Examination (USMLE) Scores Predict In-Training Test Performance for Emergency Medicine Residents?
Introduction
Objective forms of data are routinely relied upon to select emergency medicine (EM) residents from a large and talented applicant pool. United States Medical Licensing Examination (USMLE) scores are a frequently used objective criterion in the residency selection process. Although other specialties have demonstrated a correlation between USMLE scores and medical school grade point average with residency performance, there has been no such evidence in the field of EM (1, 2). Because USMLE scores may reflect medical student knowledge, it is useful to determine if they have any predictive value on residency knowledge.
With the progression of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project into phase III, there will be increased emphasis on the development of outcome measures in residency training for each of the core competencies: patient care, medical knowledge, professionalism, systems-based practice, practice-based learning and improvement, and interpersonal and communication skills (3). At present, American Board of Emergency Medicine (ABEM) certification is a widely acceptable medical knowledge competency outcome measure. Intuitively, it would seem that USMLE scores are measures of medical knowledge or standardized test-taking ability that should be highly correlated with ABEM in-training scores. Consequently, USMLE scores have been used by residency selection committees as a useful objective measure by which to compare residency candidates across a wide range of backgrounds. To date, no previous studies in the field of EM have attempted to validate this practice. The purpose of this study is to evaluate the relationship between USMLE scores and subsequent ABEM in-training examination scores.
Section snippets
Study Design
Data were collected for this retrospective cohort study for the EM graduating classes of 2002 through 2006 in a single ACGME-accredited EM residency program.
Study Setting
The study setting is an accredited EM residency program with a post-graduate year (PGY) 1-2-3 configuration. This EM program, with over 20 years of accreditation, is situated in an urban environment in the southeastern portion of the United States. The program is currently approved for 12 residents per year; however, at the beginning of the
Results
Data from 51 graduating residents were analyzed. Fifty-one USMLE Step 1 and 39 Step 2 scores were available for comparison with 153 ABEM in-training scores. The mean USMLE Step 1 and Step 2 scores were 228.9 (range 197–252) and 228.4 (range 168–259), respectively. The mean in-training percentile for the PGY 1, 2, and 3 years were 40.4 (95% confidence interval [CI] 33.9–46.9), 68.3 (95% CI 61.5–75.1), and 81.7 (95% CI 77.2–86.2), respectively. The R-squared values for the Step 1 USMLE scores
Discussion
USMLE is a licensing examination that was not designed to predict residency performance, yet its use has been modified to screen applicants for EM residency positions (5). The evidence to support this practice across medical specialties has been conflicting. One study has demonstrated that moderate emphasis is placed on these scores in the selection of EM residents (4). However, this emphasis ranks below that of interview, clinical grades, and letters of recommendation (4). In a recent study by
Conclusions
USMLE Step 1 scores are minimally correlated with ABEM in-training scores, whereas USMLE Step 2 scores are moderately correlated with in-training scores. Despite its use as an objective criterion for residency selection, USMLE Step 1 scores are minimally predictive of residency performance on ABEM in-training examination. However, USMLE Step 1 and 2 scores below 200 are associated with a significantly poorer performance, and USMLE Step 1 scores above 220 are associated with significantly better
References (17)
- et al.
USMLE Step 1 Scores as a significant predictor of future board passage in pediatrics
Ambul Pediatr
(2007) - et al.
The relationship between the National Board of Medical Examiners' prototype of the Step 2 clinical skills exam and interns' performance
Acad Med
(2005) - et al.
Orthopaedic in-training examination scores: a correlation with USMLE results
J Bone Joint Surg Am
(2006) ACGME Outcome Project
- et al.
Selection criteria for emergency medicine residency applicants
Acad Emerg Med
(2000) - et al.
Use of the USMLE to select residents
Acad Med
(1993) - et al.
What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident?
Acad Emerg Med
(2005) - et al.
Do US medical licensure Examination step 1 scores correlate with Council on Resident Education in Obstetrics and Gynecology in-training examination scores and American Board of Obstetrics and Gynecology written examination performance?
Mil Med
(2007)
Cited by (65)
Impacts of United States Medical Licensing Examination Step 1 Scoring Change on Emergency Medicine Applicant Screening
2023, Journal of Emergency MedicineThe Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director
2021, Journal of Surgical EducationCitation Excerpt :Surgical program directors (PDs) also use these scores to formulate rank lists13,14 and to differentiate between applicants.13,15,16 Scores not only reflect an applicant's basic medical knowledge, but have also been correlated with academic performance during residency across surgical and nonsurgical fields.5,17-24 As many US medical schools have transitioned to pass/fail preclinical grading systems, the numeric scoring of the USMLE Step 1 has been challenged.25
Selection criteria and match results for postgraduate residency programs: A cross-sectional model from a major academic center in Jordan
2020, Annals of Medicine and SurgeryThe USMLE Step 1 Pass/Fail Reporting Proposal: The APDR Position
2019, Academic RadiologyA Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States
2023, Journal of Graduate Medical Education