June 25, 2001 Thomas A. Scully Administrator Centers for Medicare and Medicaid Services Hubert Humphrey Building 200 Independence Avenue, SW, Room 443-G Washington, DC 20201 Attention: File Code CMS-3056-NC Re: Federal Register Volume 66, No. 81, Medicare Program: Evaluation Criteria and Standards for Peer Review Organization 6th Round Contract, Notice with Comment Period. Dear Mr. Scully: On behalf of the American Health Quality Association (AHQA), the membership organization of state-based quality improvement organizations (QIOs, known for their Medicare work as PROs), thank you for the opportunity to comment on the evaluation criteria and standards for the 6th PRO contract. Quality improvement organizations are private organizations that work for a variety of private and public customers. For purposes of this letter we will refer to them as PROs to acknowledge that the contract being referred to in this notice is the Medicare Peer Review Organization contract. Although the PRO contracts and previous correspondence on this subject refer to the Health Care Financing Administration (HCFA), this letter adopts the new convention of referring to the agency as the Centers for Medicare and Medicaid Services (CMS). Overview. AHQA acknowledges the difficulty of designing an evaluation methodology for individual PROs for their Health Care Quality Improvement Program and Payment Error Prevention Program work. Much of this work is new, motivating provider change is difficult, and measuring the results is fraught with complexity. The fact that improvement is not directly within the control of individual PROs adds to the challenge. Nonetheless, AHQA supports CMS's attempts to measure actual improvement and to factor into the evaluation documented PRO efforts to achieve results. Most of our concerns arise because this evaluation criteria will be used to rank PROs to determine who should face a review panel. Because of the measurement challenges discussed below, AHQA proposes several adjustments to the weight assigned to various factors, and recommends ways to improve the review panel process. We are also seeking clarification on several issues. Task 1: National Quality Improvement Projects. Performance Based Service Contracting. The use in the current contract of 22 quality indicators within six clinical topics to assess each PRO's performance is a first effort by CMS, not previously tested, to apply performance-based service contracting (PBSC) to health care quality improvement. During the first two years of this trial a number of flaws have been observed in this methodology. The baseline data needed to target quality improvement efforts were not available until about a year into the contract period. Remeasurement, upon which contract performance will be assessed, began in many States before improvement efforts could be completely implemented. The baseline data suggest that many of the major differences between states may be based on geographic location and socioeconomic factors that have nothing to do with PRO performance. While this pilot effort to use PBSC will be of great value to CMS in improving its contract processes, the flaws in this newly developed approach should not be allowed to penalize the PROs. Utility of the Combined Topic Weighted Average (CTWA) for evaluating individual PRO performance. While AHQA understands that the CTWA will be the method for measuring progress on Task 1, we believe its use should be given less weight in the 6th contract cycle. The strength of the relationship between the rank of an individual PRO based on their CTWA and their level of effort and effectiveness on Task 1 remains unproven. In addition, the PROs are not directly in control of the levers that move improvement, and the confidence intervals around the measures are large. In addition, the time-frame of the measurement cycle in which PROs must demonstrate change is so limited that it is very difficult to show any significant movement on measures. AHQA remains concerned that the short time period between baseline and re-measurement will result in low national improvement and lead inappropriately to the perception that the PRO program is ineffective. We are gratified that CMS and AHQA have been able to discuss and agree on an approach for the future that will allow more "real time" performance measurement and feedback to contractors and health care providers. We recommend that CMS continue efforts to find alternatives to the CTWA. The shortcomings of this methodology will only become more troublesome as PROs are asked to move to new clinical indicators at different points in time in the future, rendering the rankings based on the CTWA even less reliable for comparison of the PROs. We urge CMS to consider the factors above when making determinations about whose contracts should be competed and to give serious consideration of the efforts that PROs have undertaken to motivate change, whether improvement occurred or not. A PRO in a state where providers are very resistant to change may, in fact, have performed better on the PRO contract than a PRO in a state where motivation came from other sources. Individual PROs should not be penalized for forces outside their control. TQIP and Process Measures. One implication of relying less heavily on improvement measured on the CTWA is that CMS would pay more attention to process measures and in particular those known to be associated with improvement. However, the experience of the PROs and CMS with the tool currently being used to collect data on processes - TQIP - is that it collects and will provide very little useful information. It is being used as a research tool to determine which activities are associated with improvement, but the data fields do not tell the PROs or CMS what they need to know to judge whether improvement has occurred. Because terms were not clearly defined, PROs are entering data in different ways. Often, multiple interventions were employed by each PRO, making it difficult to ascribe results to a particular intervention. Additionally, entering the data is very resource intensive, perhaps detracting from other work the PROs could or should be doing to encourage improvement or track progress. AHQA and CMS have had many conversations about how to address the issues related to TQIP. At the root of the problem appears to be the fact that there is no single articulated purpose or a framework for how TQIP would achieve multiple purposes. We recommend that CMS work with the PRO community to articulate and define what type of data should be collected in order to more fully understand how it is the PROs do what they do. Clarity of the relationship between PRO efforts and provider improvement would aid in building support for the program, diffusion of best practices, and evaluation of individual PROs. Review Panel Process and Eligibility. Given the limitations of the methodology for determining whether and which PRO comes before the review panel and the lack of an objective, reliable process for measuring and comparing the effectiveness of PRO efforts, it is critical that the review panels allow as much interaction as feasible with the individual PRO. AHQA has several recommendations: CMS should specify the method for determining whether and which of the PROs who may fall in the bottom 25% will not be reviewed based on the fact that they are very close to the line. CMS should only require the true outliers to expend the resources necessary to demonstrate that they have adequately performed their contractual obligations. Policies governing the process for the review panel, including the criteria upon which the panel will base its decisions, should be established and given to the PROs ahead of time. For equity and consistency in management, these policies and procedures should not be left up to the review panels, but should be defined by the central office with regional office input. An outlier PRO that must go before a review panel should be allowed to be present for a significant portion of the deliberations to offer clarifications and answer questions. In addition, the PRO should be given a two-hour period for a presentation and discussion. Timing of Baseline Release. Under the task-specific standards section in Task 1 the notice states that a baseline was provided to the PROs "near" the start of the 6th round contract. In fact, the release of the baseline data occurred in the fall of 2000, one year into the first contracts and only a year before re-measurement occurs. PROs did not wait for this baseline release to occur before beginning work with providers, but this release, and the national and local press surrounding it, would have been useful in focusing provider and public attention on the need to improve. Moreover, having the baseline data available at the start of the contract would have made it easier to encourage provider participation and given the PROs a sense of which clinical areas to focus on. This should be taken into account when discussing an individual PRO's improvement rates, and addressed in the 7th contract. This is even more of an issue in Task 4, as the baseline dataset is not complete even now, when re-measurement is already beginning to occur. Time to Reperfusion and the CTWA. The time to reperfusion indicator should be removed from the list of measures used to calculate the CTWA. The number of patients eligible for these procedures is very low and, so far, no uniform method has been developed for calculating the measure. Task 2: Local Quality Improvement Projects. "Achieving Measurable Improvement." The notice includes a subtle shift in words from the contracts that could have a direct impact on the evaluation of PROs in Task 2. The current contract states that if the PRO is working in a new area where measures have not been developed that other criteria than "achieving measurable improvement" will be used to evaluate PRO performance. However, the notice says other criteria will only be used "if" measurable improvement is not achieved. This wording suggests CMS will attempt to gauge whether measurable improvement has occurred even if measures have a weak scientific basis. The contract has been used by PROs to develop projects that help develop the knowledge and expertise to approach new clinical topics and to design effective interventions with providers. These projects were not designed to achieve measurable improvement, but to lay the groundwork to create the ability to do so in the future. They should be evaluated as such. PROs should not have to demonstrate the reasons why improvement wasn't achieved in such projects. Task 3: Quality Improvement Projects in Conjunction with Medicare+Choice Plans. AHQA has two questions related to the provisions discussed in Task 3, and one comment. How will CMS evaluate PROs in states where managed care penetration has changed dramatically? We are unclear as to whether M+C plans have had to use the same indicators that the PROs have been using for their HCQIP work. If the M+C plans did not need to measure the same clinical areas or use the same indicators it would be impossible for the PROs to be evaluated in the same manner as they are under Task 1 , as stated in the notice. CMS may need to alter its evaluation strategy because the indicators or measures were not the same. The second paragraph under the Task 3 section implies that technical assistance is given to the M+C plans through "projects." Much of the assistance PROs give to M+C plans is more general or collaborative in nature and should not be considered as a part of a "project." We believe it is important to make the distinction because the term project implies a more discreet work product than most technical assistance may produce. Task 4: Payment Error Prevention. Comparatively evaluating and ranking the PROs under Task 4 is very problematic. Some PROs are nearly two years into the 6th Scope of Work and CMS has not yet provided any PROs with baseline data for their Payment Error Prevention Program (PEPP) work. CMS has offered no date certain that the PROs can expect to receive this data. To make matters worse, even though they have received no data on which to base their improvement strategies, CMS has already begun gathering data for remeasurement. Preliminary results show that state rates vary widely. This data may be useful on a national level, but it is unclear that the data will be in any way suitable for drawing comparisons between PROs as a measure of success. Task 5: Other Mandatory Activities. For more than two years, AHQA has been negotiating with CMS to correct the insufficient level of funding provided for case review in the 6th Scope of Work. CMS recently conducted a study sampling a number of PROs across the country and obtained and evaluated information regarding the varying case mix and volume of cases the PROs are required to review. CMS also collected information on the varying processes PROs used to handle these cases. The study confirmed that CMS' original funding assumptions for Task 5 were, in fact, incomplete, leading to insufficient funding of this activity. AHQA recommends that CMS take these findings into consideration in evaluating PROs. The discrepancy between budgeted funds and actual costs of conducting these mandatory reviews plays an important role in problems of optimally executing Task 5. Thank you for this opportunity to comment on the Federal Register notice and hope that our comments are useful in improving evaluation in the 6th PRO contract, and in identifying issues that need to be addressed in future PRO contract cycles. Sincerely, David Schulke Executive Vice President