Education
Do United States Medical Licensing Examination (USMLE) Scores Predict In-Training Test Performance for Emergency Medicine Residents?

Abstract presented in poster format at Society for Academic Emergency Medicine Annual Meeting, Chicago, IL, May 2007.
https://doi.org/10.1016/j.jemermed.2008.04.010Get rights and content

Abstract

Background: Residency selection committees commonly utilize USMLE scores as criteria to screen residency applicants. Objectives: The purpose of this study is to evaluate the relationship between United States Medical Licensing Examination (USMLE) and American Board of Emergency Medicine (ABEM) in-training examination scores (ITEs). Methods: In an Accreditation Council for Graduate Medical Education-accredited emergency medicine residency program, data were collected for this retrospective cohort study for the classes of 2002–2006. USMLE Step 1 and 2 scores and the ABEM ITEs were recorded for each post-graduate year (PGY) within the aforementioned time frame. Step 1 and 2 scores were compared to consecutive PGY ABEM ITEs to evaluate for an association. Results: There were 51 USMLE Step 1 and 39 Step 2 scores available for comparison with 153 ABEM ITEs. The mean USMLE Step 1 and Step 2 scores were 228.9 (range 197–252) and 228.4 (range 168–259), respectively. The mean in-training percentiles for the PGY 1, 2, and 3 years were 40.4, 68.3, and 81.7, respectively. The R-squared values for the Step 1 scores compared to the PGY 1, 2, and 3 years' ITEs were 0.25, 0.18, and 0.16, respectively. The R-squared values for Step 2 scores as compared to the ABEM ITEs for the PGY 1, 2, and 3 years were 0.43, 0.44, and 0.38, respectively. Residents who scored below 200 on either USMLE Step 1 or Step 2 had significantly lower mean ABEM ITEs than residents who scored above 200 (p < 0.05) and were 10-fold more likely than residents who scored above 220 to score below the 70th percentile in their PGY3 ABEM ITE. Conclusions: USMLE Step 1 scores are mildly correlated and Step 2 scores are moderately correlated with ABEM ITEs. Scoring below 200 on either test is associated with significantly lower ABEM ITEs.

Introduction

Objective forms of data are routinely relied upon to select emergency medicine (EM) residents from a large and talented applicant pool. United States Medical Licensing Examination (USMLE) scores are a frequently used objective criterion in the residency selection process. Although other specialties have demonstrated a correlation between USMLE scores and medical school grade point average with residency performance, there has been no such evidence in the field of EM (1, 2). Because USMLE scores may reflect medical student knowledge, it is useful to determine if they have any predictive value on residency knowledge.

With the progression of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project into phase III, there will be increased emphasis on the development of outcome measures in residency training for each of the core competencies: patient care, medical knowledge, professionalism, systems-based practice, practice-based learning and improvement, and interpersonal and communication skills (3). At present, American Board of Emergency Medicine (ABEM) certification is a widely acceptable medical knowledge competency outcome measure. Intuitively, it would seem that USMLE scores are measures of medical knowledge or standardized test-taking ability that should be highly correlated with ABEM in-training scores. Consequently, USMLE scores have been used by residency selection committees as a useful objective measure by which to compare residency candidates across a wide range of backgrounds. To date, no previous studies in the field of EM have attempted to validate this practice. The purpose of this study is to evaluate the relationship between USMLE scores and subsequent ABEM in-training examination scores.

Section snippets

Study Design

Data were collected for this retrospective cohort study for the EM graduating classes of 2002 through 2006 in a single ACGME-accredited EM residency program.

Study Setting

The study setting is an accredited EM residency program with a post-graduate year (PGY) 1-2-3 configuration. This EM program, with over 20 years of accreditation, is situated in an urban environment in the southeastern portion of the United States. The program is currently approved for 12 residents per year; however, at the beginning of the

Results

Data from 51 graduating residents were analyzed. Fifty-one USMLE Step 1 and 39 Step 2 scores were available for comparison with 153 ABEM in-training scores. The mean USMLE Step 1 and Step 2 scores were 228.9 (range 197–252) and 228.4 (range 168–259), respectively. The mean in-training percentile for the PGY 1, 2, and 3 years were 40.4 (95% confidence interval [CI] 33.9–46.9), 68.3 (95% CI 61.5–75.1), and 81.7 (95% CI 77.2–86.2), respectively. The R-squared values for the Step 1 USMLE scores

Discussion

USMLE is a licensing examination that was not designed to predict residency performance, yet its use has been modified to screen applicants for EM residency positions (5). The evidence to support this practice across medical specialties has been conflicting. One study has demonstrated that moderate emphasis is placed on these scores in the selection of EM residents (4). However, this emphasis ranks below that of interview, clinical grades, and letters of recommendation (4). In a recent study by

Conclusions

USMLE Step 1 scores are minimally correlated with ABEM in-training scores, whereas USMLE Step 2 scores are moderately correlated with in-training scores. Despite its use as an objective criterion for residency selection, USMLE Step 1 scores are minimally predictive of residency performance on ABEM in-training examination. However, USMLE Step 1 and 2 scores below 200 are associated with a significantly poorer performance, and USMLE Step 1 scores above 220 are associated with significantly better

References (17)

  • Q.E. McCaskill et al.

    USMLE Step 1 Scores as a significant predictor of future board passage in pediatrics

    Ambul Pediatr

    (2007)
  • M.L. Taylor et al.

    The relationship between the National Board of Medical Examiners' prototype of the Step 2 clinical skills exam and interns' performance

    Acad Med

    (2005)
  • K.P. Black et al.

    Orthopaedic in-training examination scores: a correlation with USMLE results

    J Bone Joint Surg Am

    (2006)
  • ACGME Outcome Project

  • J.T. Crane et al.

    Selection criteria for emergency medicine residency applicants

    Acad Emerg Med

    (2000)
  • E.S. Berner et al.

    Use of the USMLE to select residents

    Acad Med

    (1993)
  • S.R. Hayden et al.

    What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident?

    Acad Emerg Med

    (2005)
  • A. Armstrong et al.

    Do US medical licensure Examination step 1 scores correlate with Council on Resident Education in Obstetrics and Gynecology in-training examination scores and American Board of Obstetrics and Gynecology written examination performance?

    Mil Med

    (2007)
There are more references available in the full text version of this article.

Cited by (65)

  • The Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director

    2021, Journal of Surgical Education
    Citation Excerpt :

    Surgical program directors (PDs) also use these scores to formulate rank lists13,14 and to differentiate between applicants.13,15,16 Scores not only reflect an applicant's basic medical knowledge, but have also been correlated with academic performance during residency across surgical and nonsurgical fields.5,17-24 As many US medical schools have transitioned to pass/fail preclinical grading systems, the numeric scoring of the USMLE Step 1 has been challenged.25

View all citing articles on Scopus
View full text