Wording in medical student evaluations differ by gender, minority status, revealing potential bias

In an analysis of narrative medical school evaluations, researchers at UC San Francisco and Brown University have found significant differences in how female and underrepresented minority medical students are described.

The researchers used natural language processing to analyze a large bicoastal sample of nearly 90,000 narrative evaluations of third-year clerkships from UCSF and Brown. The data spanned nine years at UCSF, from 2006 to 2015, and five at Brown, from 2011 to 2016.

These evaluations are supposed to focus on student behaviors, or competencies, that are directly relevant to medicine. But the analysis found that evaluators often used personal descriptors to describe a student's performance, and they used strikingly different words for men and for women.

The study also identified personal descriptors that were applied differently depending on whether students were members of groups that are underrepresented in medicine, or URMs. But nearly all of these personal descriptors were used more often to describe students who were not in those groups.

IMPACT

The evaluations form the basis of the grades that students get in their core clerkships, which are like medical apprenticeships, and are frequently quoted in letters of recommendation for residencies. Even small biases can snowball, with lasting effects on students' career prospects.

A fourth-year UCSF medical student, Alexandra Rojek, used natural language processing to find the words that were commonly used to describe students -- yet not so common that they were used to describe too many different things, and therefore could have a wide range of meanings -- and that also were used more or less often depending on a student's gender or URM status.

The analysis found 37 descriptive words that evaluators applied differently by gender and 53 descriptive words applied differently by URM status.

Nearly two-thirds (23 of 37) of the words used differently by gender described students' personal attributes, and just over half of these were more likely to be used to describe women.

These included "pleasant," which was associated with getting a passing grade; "energetic," "cheerful," and "lovely," which were not associated with any particular grade; and "wonderful" and "fabulous," which were associated with honors grades.

The words used more commonly for men included "respectful" and "considerate," which had no association with a grade, while "good" was associated with earning passing grades, and "humble" was more common among those who earned honors grades.

Descriptive words that varied by gender also included four competency-related words. "Efficient," "comprehensive," and "compassionate," which the raters deemed related to competency in patient care, predominated for women, and were associated with honors grades. "Relevant" was more common for men, and was also associated with honors grades.

WHAT ELSE YOU SHOULD KNOW

Of the 53 words that differed by URM status, nearly a third described personal attributes, and the overwhelming majority of those, 13 of 16, were used for non-URM students.

Evaluators were more likely to use words like "pleasant," "open," and "nice" to describe URM students, and these words were associated with passing grades. Descriptors like "enthusiastic," "sharp," and "bright" were used more commonly for non-URM students. These words were not associated with getting a particular grade, but "mature" and "sophisticated" were more frequently associated with honors grades.

Just over a quarter of the words, 15 of 53, were related to competencies, and all of them were used more often for non-URM students. These included, "outstanding," "impressive," and "advanced," which were associated with honors, while "superior," "conscientious," and "integral" were not associated with a grade.

THE TREND

Unfortunately, these gender disparities often seep into the actual workplace for many physicians.

One example: The 2019 Medscape Physician Compensation Report released this month found the gender disparity in physician compensation increased by 6 percentage points, with men earning 25 percent more than women, up from 18 percent more in 2018.

While the report found that women see patients an average of four hours less per week, and gravitate towards the lower paid specialties, those factors do not fully explain the pay differences.
 

Twitter: @JELagasse

Email the writer: jeff.lagasse@himssmedia.com