FOR IMMEDIATE RELEASE
October 25, 2005
Contact: Richard Deutsch
Phone: 202-261-7573

Hopkins Researchers Admit Flaws in Study of Medicare Efforts to Improve Quality of Health Care

Washington, DC – In a letter to the editor published in JAMA today, Johns Hopkins School of Public Health researchers Dr. Gerard Anderson and Claire Snyder acknowledge that critics have raised “valid points” about the core methodology the researchers used to evaluate Medicare Quality Improvement Organizations (QIOs) earlier this year. Their statement is a response to a number of letters published in JAMA today by experts who identify critical flaws in the Snyder-Anderson methodology.

In the June 15 th edition of JAMA, Snyder and Anderson published a study contending that the Medicare-sponsored QIO program has not helped hospitals provide better care. The study is based on analysis of data from 5 states that Snyder and Anderson claim shows no significant link between hospital performance on quality indicators and QIO assistance. The study says that the data coincided with a three-year (1998-2001) cycle of QIO work under contract to Medicare.

However, letters to the editor of JAMA today point out that the data covered only about half that period—and therefore cannot accurately reflect the impact of QIO work during the three-year contract cycle.

“The quality indicator data we provided to Snyder and Anderson included care delivered only during the first 17 months of the 36 month QIO intervention period,” said Dr. Jonathan Sugarman in his letter to the JAMA editor. He and other experts say that efforts to improve quality accelerated in many hospitals as the QIO work cycle progressed.

“We believe the data set the authors analyzed could not be used to reach their principal conclusion,” said Sugarman. “The effectiveness of a three-year effort cannot be judged using information from only the first 17 months of that period.” Sugarman is the CEO of Qualis Health, the Seattle-based QIO, and president of the American Health Quality Association (AHQA), which represents the national network of QIOs.

The Snyder-Anderson study, Do Quality Improvement Organizations Improve the Quality of Hospital Care for Medicare Beneficiaries?, tried to evaluate QIOs by “comparing the improvement in the quality of care of patients in hospitals that actively participated with the QIOs versus hospitals that did not.” It concludes that “hospitals that participate with the QIO program are not more likely to show improvement on quality indicators than those that do not participate.”

This basis for comparison, experts said in letters to JAMA’s editor today, cannot be used for the three-year period covered by the study. They note that during 1998-2001, QIOs were required by Medicare to work with ALL hospitals.

“During the study period, QIOs had some contact with essentially all hospitals and often worked with an individual hospital on one topic but not another,” notes a letter to the editor from Dr. Stephen Jencks, a Medicare official who headed the QIO program during the period covered by the Snyder-Anderson study. “The QIO program has developed many of the measures and much of the infrastructure that the Centers for Medicare & Medicaid Services and the nation use in quality improvement and public reporting programs, so there are no real nonintervention facilities,” writes Jencks.

In a 2003 JAMA article, Jencks used Medicare data to show that hospitals broadly improved quality of care during 1998-2001. But, Jencks said in that article that the data “do not provide conclusive evidence” about the effectiveness of QIOs—one way or another.

In their study, Snyder and Anderson used the same data over the same time period to try to evaluate QIOs by assigning hospitals to control groups that did not exist at the time, an effort Jencks and other experts say is simply misguided.

“Generalizing from this underpowered study of a convenience sample of 5 QIOs to the entire program of 53 QIOs, especially given the differences the authors report among these 5 QIOs, is imprudent,” Jencks concluded in his letter published in JAMA today.

The American Health Quality Association represents the national network of Quality Improvement Organizations that work with hospitals, medical practices, health plans, long-term care facilities, home health agencies, and employers to encourage adoption of best practices and improve systems of care.