Impact of a structured review session on medical student psychiatry subject examination performance

Background

The National Board of Medical Examiners (NBME) subject examinations are widely used in North America as a means of assessing overall performance and potential need for remediation in required third-year medical student clerkships; their utility is rooted in the fact that they provide a standardized and objective measure of knowledge acquired during the clerkship¹. While the utility of NBME examinations for internal evaluation of students has been questioned², this notion is challenged by the findings that performance on these examinations is correlated with other measures of a medical student’s knowledge base^3,4, suggesting that higher scores are associated with improved overall educational outcomes. Furthermore, these scores are also correlated with a student’s eventual performance on the United States Medical Licensing Exam (USMLE) Step 2CK, which is a critical component of evaluation for residency selection⁵.

However, strategies for preparing students for NBME examinations remain inconsistent⁶. This process is particularly challenging for the psychiatry subject examination (PSE), in which performance has been found to be more strongly associated with interpersonal skills than with subjective faculty evaluations of a student’s medical knowledge and clinical skills⁷, although alternate measures of student performance (including faculty evaluations and standardized patient encounters) are still correlated with PSE scores⁸.

The impact of structured teaching on PSE scores has garnered some attention in the literature. Prior studies have demonstrated a significant improvement with a series of eight resident-led tutorials⁹ and with a complete curriculum overhaul with a goal of improving scores¹⁰. A single end-of-clerkship review session for the subject examination has also demonstrated an increase in scores, but this study was limited by a relatively small sample size, which limited the range of outcomes that could be effectively measured, and by potential selection bias, since attendance at the session was not mandatory¹¹. We investigated the impact of a single review session with a larger sample size and with mandatory attendance.

Methods

The study retrospectively investigated scores on the PSE after implementation of a review session covering a general overview of adult psychiatry with a focus on topics that are critical for medical students to understand. The review session was conducted less than 1 week before students were required to take the PSE. Students were required to attend, but were excused in the event of a conflict with their rotation schedules. Data were analyzed retrospectively based on de-identified scores provided by the NBME. This study was deemed exempt from review by the institutional review board at Washington University in St. Louis, which determined that consent from individual students was not required and students need not be notified because data were de-identified prior to retrospective analysis. The Associate Dean for Education at Washington University School of Medicine also approved the retrospective review of de-identified scores.

The review session was based on an interactive case-based discussion of evaluation and management of common psychiatric problems, with a focus on topics that are commonly misunderstood by medical students. The session was designed and conducted by a resident physician (SHS) with prior experience in developing study materials for various standardized examinations, including the PSE. Cases demonstrated hypothetical patients with mania, depression, psychosis, substance abuse, anxiety/panic, eating disorders, personality disorders, somatoform disorders, and psychotropic medication toxicity. Additional non-case-based discussions were included to differentiate the types of dementia and understand legal/ethical issues in psychiatry. Child psychiatry topics were not included because the clerkship already included a separate lecture on child psychiatry during the same week. Detailed psychopharmacology was also not included due to time constraints; instead, students were advised to independently review mechanisms, indications, and toxicity profiles of the different classes of psychotropic medications.

The experimental group consisted of nine groups of students completing their psychiatry clerkships between September 2014 and July 2015. The control group consisted of students completing the examinations during the corresponding time periods in the previous two academic years; the other months in previous years were not included to avoid confounding due to the tendency of scores to increase as the academic year progresses. No other changes were made to the students’ lecture schedules.

Statistical analyses were completed with R version 3.2.0 using individual de-identified scores that were provided by the NBME in paper form. Mean scores for the full September to July period were compared between the experimental group and the control group via two-tailed paired t-test. In order to evaluate the effects on students with weaker knowledge base, a paired t-test was also used to compare means for all students who achieved lowest two scores in each 4-week clerkship block between the experimental group and the control group. A one-tailed Z-test for proportions was used to compare the fraction of students scoring 99 (the maximum possible score) and the fraction of students scoring 70 or less (typically corresponding approximately to the 10th percentile in the national sample; our school considers this a failing exam score that must be remediated to earn credit for the psychiatry clerkship).

Results

Eighty-three students took the exam during the experimental period, while 175 took the exam during the control period. Statistically significant improvements were found in the mean score, the two lowest scores in each group, and the fraction of students scoring 70 or less. Improvement in fraction of students achieving the maximum possible score (99) did not reach significance (p = 0.06). These results are summarized in Table 1.

Discussion

Implementation of a mandatory end-of-clerkship review session was associated with improvements in mean scores on the PSE, particularly for students whose scores were in the lower range. While similar improvements have been suggested in the past¹¹, this study reproduces these findings with a larger sample size, thereby allowing analysis of performance in different scoring ranges. This study also demonstrated a significant effect of the intervention despite higher baseline scores in this sample (mean baseline scaled score 85.3, compared to 77.2 in the previous study). Furthermore, attendance at the review session in this study was mandatory, thereby controlling for the selection bias introduced by the possibility that students choosing to attend a voluntary review session may have been more motivated at baseline.

Due to the retrospective nature of the analysis and lack of randomization, this study is subject to several limitations. Performance was compared between different academic years, so inter-class differences unrelated to the intervention may have confounded the results. Furthermore, while the review sessions followed a standardized format, we do not know how reproducible they may be in other academic settings.

This study did not investigate whether the improvement in students’ PSE performance translated to improvements in clinical skills. However, a recent large meta-analysis showed that clerkship grades (which usually incorporate NBME subject examination scores¹) and USMLE Step 2CK scores (which are correlated with NBME subject examination scores) predict a resident’s performance on both objective and subjective evaluations¹². Further research is needed in order to determine whether an end-of-clerkship review session translates to improvements in other measures of a student’s clinical skills and knowledge.

Overall, these results provide support for the notion that a single end-of-clerkship review session improves scores on the NBME psychiatry subject examination, even when eliminating selection bias by making the review session mandatory. Future studies should be geared at reproducing these findings in other specialties and standardizing the course for improved generalizability.

Data availability

F1000Research: Dataset 1. Medical student scores on the psychiatry NBME subject examination before and after institution of a mandatory review session, 10.5256/f1000research.7091.d102704.