January 22, 2002 Ms. Julie Brown Office of Information Services Security and Standards Group Division of CMS Enterprise Standards CMS-R-305, Room N2-14-26 Centers for Medicare and Medicaid Services 7500 Security Boulevard Baltimore, MD 21244-1850 Re: Federal Register Volume 66, No. 226, Proposed Collection Activities; Comment Request Dear Ms. Brown: The American Health Quality Association (AHQA) appreciates the opportunity to comment on the proposed information collection activity pertaining to external quality review of Medicaid MCOs. AHQA is the national membership organization representing the state-based Quality Improvement Organizations (QIOs). Our members develop programs in quality improvement and evaluation for Medicare and Medicaid. Most QIOs also perform external quality review (EQR) of Medicaid managed care organizations (MCOs). The following comments reflect our recommendations and concerns about the newly released external quality review organization (EQRO) protocols. The Necessity and Utility of the Proposed EQRO Protocols AHQA generally is satisfied that the published EQRO protocols establish a framework that will achieve the intended goals of assessing and improving the quality of care for Medicaid beneficiaries in managed care plans. When the Balanced Budget Act of 1997 granted States increased authority to contract with MCOs for the provision of Medicaid benefits, AHQA shared the concern of Congress that an enhanced investment in quality assurance and quality improvement should coincide with the increased authority. The external review of MCOs must contain core elements that are consistent from state to state, but it also must be flexible enough to allow EQROs to apply the protocols across managed care delivery systems that are very different (e.g., rural versus urban populations, staff model versus PPO.) The Accuracy of the Estimated Burden: Methodology for Cost Estimates of the Three Mandatory Protocols AHQA is concerned about the estimates contained in the cost analysis of the published background material. The costs forecast by CMS may be significantly lower than the actual costs of the projected work. The methodology used by CMS to generate cost estimates is flawed in two ways: 1) the sample size is too small; and 2) it does not accommodate for indirect costs. Sample Size is Small AHQA is concerned that the sample size used to estimate burden is too small to draw the conclusions published in the background material. In estimating the burden of the new protocols, CMS interviewed four EQROs about their current review tasks. It is unclear if the four EQROs selected were a representative sample of review organizations. We particularly are concerned about the detailed data analysis performed on this limited sample size. Without knowing the standard deviation of the data set, we are nonetheless troubled regarding the tremendous variation in reported data. Data variation is an important consideration to determine the validity of cost averaging. To illustrate our concern, the CMS generated raw data is listed below: It takes 25-138 hours at a cost of $2,000-$10,000 to validate a performance improvement project; It takes 12-202 hours at a cost of $1,200-$7,000 to validate a performance measure calculated by an MCO/PHIP; It takes 200-800 hours at a cost of $11,000-$49,000 to review MCO-PHIP compliance with structural and operational standards. The raw data was then subject to simple averaging calculations to make the following conclusions: It takes an average of 65, 53, and 361 hours, respectively, to conduct the mandatory EQR activities; The average total burden associated with this requirement is 479 hours x 496 entities (339 MCOs + 157 PHPs). Assuming wages of $63 per hour per professional, the cost is $14,967,792. AHQA contends that the raw data contains a data set range too large for simple cost averaging to prove useful. Not knowing the median or standard deviation of the data collected, the final cost estimates of 65, 53, and 361 hours may not be representative of the actual cost for completing the protocols for many organizations. Additionally, with only four EQROs providing data, the averages may be too low or too high to draw any practical conclusion. Indirect Costs are Not Considered The final cost of $14,967,792 is based upon salary and fringe expenses of $63 per hour, per professional. Indirect expenses such as rent, transportation, medical record photocopies, etc. are not accounted for in the estimate. A reasonable allowance for indirect expenses must be added to accurately predict the cost of the new protocols. Ways to Enhance the Quality, Utility, and Clarity of the Information AHQA is concerned that the protocols lack an evidence-based approach to quality improvement. As CMS develops its final notice regarding this collection request, we would ask that you provide clarification regarding the collection and validation of performance measures. There is no description of essential EQRO activities to ensure that the performance measures being used by the state are scientifically sound, meaningful, valid and reproducible. If the performance measures do not have these attributes, the information collected will not be useful even if the collection methodology and the validation methodology are consistent with the protocols. Recommendation: AHQA recommends that CMS take steps to ensure that EQROs use only evidence based performance measures. Throughout the background material provided with this proposal, CMS recognizes that limited information is available about the estimated burden of the new protocols, especially the optional protocols. AHQA offers the following recommendations, which are necessary to ensure that adequate funding is available to complete the activities described in the EQRO protocols: Recommendation: AHQA recommends that CMS allow EQROs an appropriate amount of flexibility to complete the activities described by each protocol. Flexibility is important to alleviate the cost, burden, and duplication of these EQR activities. For example, the proposed EQRO process and the HEDIS auditing process parallel each other in many respects. One Midwestern QIO/EQRO has implemented a process to allow MCOs that were accredited by a private accreditation organization to undergo a deeming process. This deeming process evaluated the standards and quality measures to ensure the MCO achieved and sustained full compliance status for a designated period of time. The deeming process decreased much of the duplicative onsite review functions that normally would be carried out by both the accreditors and the EQRO. Recommendation: AHQA recommends that CMS reconsider the personnel costs associated with review activities. The cost estimate of $63 per hour, per professional, only states the direct cost of protocol implementation. Necessary indirect costs, with provision for reasonable geographic variation, should be considered in addition to the direct cost. Thank you for this opportunity to comment on the EQRO protocols. Please contact me or Dr. Mark Boesen at (202) 261-7571 with any questions regarding this letter. Sincerely, David G. Schulke Executive Vice President