U.S. News revises 'Best Hospitals' methodology in wake of backlash

Photo: John Fidele/Getty Images

Following backlash, U.S. News and World Report has made a number of changes to its methodology for its upcoming Best Hospitals rankings, which are due for release on August 1.

Officials from U.S. News, including managing editor and Chief of Health Analysis Ben Harder and senior health data scientist Dr. Min Hee Seo, released 18 "refinements" to the methodology for the upcoming rankings. A handful of others that are under consideration.

The changes come on the heels of scrutiny from medical experts for alleged imprecise methodology and bias, as well as an inquiry from San Francisco City Attorney David Chiu into the legality of the rankings.

Last month, Chiu sent a letter to U.S. News seeking information on the rankings, and demanding that the publication disclose the payments it receives from the hospitals it endorses, as required by federal regulations.

"Consumers use these rankings to make consequential healthcare decisions, and yet there is little understanding that the rankings are fraught and that U.S. News has financial relationships with the hospitals it ranks," said Chiu.

Although U.S. News does not state how many people visit its online hospital rankings specifically, it claims that more than 40 million people visit its website every month, with others buying its annual "Best Hospitals Guidebook."

But Chiu claimed that the data and methodology behind the hospital rankings are unreliable or imprecise at best. According to his office, U.S. News uses data not intended or appropriate for assessment purposes to formulate the rankings. In some areas, the rankings are based on a small subset of patients and exclude data from treatment of lower-income patients, such as Medicaid patients.

Peer opinion surveys play an outsize role in rankings, he said, and in some specialties the rankings are based entirely on peer opinion surveys.

What that does, alleged Chiu, is create perverse incentives for hospitals to invest in areas that score more points in the rankings instead of investing in primary care, other specialties, or ways to reduce the costs of care.

Another allegation in the letter is that, in some instances, U.S. News weights treatment for conditions experienced primarily by white people over treatment for conditions experienced primarily by people of color. By way of example, Chiu said that in the children's hospital rankings, U.S. News gives disproportionate weight to cystic fibrosis treatment, experienced by 1 in 3,500 white Americans, over sickle cell disease, experienced by 1 in 365 Black newborns.

Additionally, U.S. News allegedly receives revenue it does not disclose from hospitals it ranks. By failing to disclose this information, Chiu said U.S. News appears to be violating FTC regulations requiring the disclosure of a material connection between an endorser and the subject of an endorsement.

U.S. News receives revenues from hospitals through licensing fees to use its "best hospitals" badge, subscriptions to access the granular data underpinning the rankings, and advertising on its website and in the Best Hospitals Guidebook, according to the city attorney's office. This funding is significant. One hospital in Kansas allegedly acknowledged that it paid U.S. News $42,000 to use the "best hospitals" badge for one year.

WHAT'S THE IMPACT

A number of changes will be implemented for this year's rankings, U.S. News has posted. Under the umbrella of health equity, measures that quantify racial disparities in outcomes will shift focus from readmission to giving patients time at home. The latter is an outcome measure that several working group members felt to be superior to readmission. Also, a marker of neighborhood socioeconomic disadvantage known as the Area Deprivation Index (ADI) has been used in lieu of Social Vulnerability Index in risk-adjusting this outcome measure.

New "Prevention of outpatient procedural complications" outcome measures were added in this year's orthopedics and urology rankings, and similar outcome measures may be added in other specialties in future editions of Best Hospitals.

Volume measures and volume-based eligibility rules used in the ear, nose and throat and obstetrics and gynecology rankings now include relevant outpatient procedural cases, U.S. News said. And as announced in February, the weight on outcome measures increased from 37.5% to 45% – and other objective measures increased from 35% to 40% – in each of 11 specialties. The weight on physician opinion was reduced from as much as 27.5% to either 12% or 15%, depending on the specialty.

All transparency measures were given a standard weight of 3% in specialties that utilize these measures, and a new transparency measure was added to the pulmonology and lung surgery specialty rankings and lung cancer surgery ratings, based on the Society of Thoracic Surgeons General Thoracic Surgery Database quality improvement program. That new measure, and three existing transparency measures in other specialties, each received a weight of 3%.

Using criteria from the Elixhauser Comorbidity Software Refined for ICD-10-CM, risk adjustment of all inpatient outcome measures in all specialties, procedures and conditions employed an expanded set of 38 comorbidities, compared to 29 comorbidities used in previous editions of Best Hospitals.

U.S. News is also introducing a leukemia, lymphoma and myeloma rating; trauma center verification; HCAHPS adjustment for orthopedic and surgical hospitals; and specialty-specific eligibility criteria.

Future changes U.S. News is considering include incorporating social risk into risk adjustment, employing a broader use of outpatient outcomes, a greater emphasis on patient outcomes and supplementing traditional Medicare data and reducing data latency.

THE LARGER TREND

Last week, Penn Medicine said it would stop participating in U.S. News' rankings, meaning that while it's still eligible to be ranked, it will no longer submit data to the American Hospital Association's Annual Survey, used by U.S. News for its rankings.

"Underscoring the need for greater transparency and access to more comprehensive quality data beyond the narrow information collected by U.S. News & World Report, the health system is committed to annual publication of consistent, objective performance metrics," the system posted on its website.

"The U.S. News & World Report 'Best Hospitals' rankings are based solely on inpatient hospital care of those insured by Medicare," the system said. "These patients are typically at least 65 years old or have long-term disabilities. The rankings also do not reflect the field's shift toward outpatient medicine. At Penn Medicine, for instance, the majority of care is now delivered outside of hospitals."

St. Luke's University Health Network, based in Pennsylvania and New Jersey, recently announced that the network would no longer participate in U.S. News rankings, citing similar concerns to those raised in the city attorney's letter.

The letter to U.S. News from St. Luke's says the rankings rely heavily on surveys of doctors, essentially making the rankings a "popularity contest," when they should instead be based on objective metrics.

St. Luke's alleged the rankings fail to consider the cost of care and don't appropriately weigh patients treated for chronic illnesses. The health system also said that mortality rates are given undue weight.
 

Twitter: @JELagasse
Email the writer: Jeff.Lagasse@himssmedia.com