Detailed Information on the
Children's Hospitals Graduate Medical Education Payment Program Assessment

Assess feasibility and options for comprehensive, independent evaluation

Develop and apply a data collection instrument to meet new legislative requirements for teaching hospitals' accountability (P.L. 109-307).

Assess program's capacity and performance in training pediatricians. (First phase of multi-year effort to develop report to Congress in 2011)

Contingent upon the results of pilot studies, will verify 100% of hospitals' reported data on bed counts, case-mix index, and number of discharges by FY 2008.

The program is required by statute to pay hospitals on a bi-weekly basis. The Administration will examine whether the program can improve efficiency by paying hospitals on a quarterly basis.

Conduct 100 percent assessment of FTE resident counts including the "cap"

Make performance data available to the public.

Measure: Percent of hospitals with verified bed counts, case-mix index, and number of discharges. This measure is contingent upon the results of pilot studies to be completed in FY 2006.

Explanation:Studies underway to determine feasibility.

Measure: Percent of hospitals with verified FTE resident counts and caps

Explanation:

Measure: Percent of payments made on time

Explanation:The percentage of payments to hospitals made every 2 weeks. Monthly payments are made early in each fiscal year while final program allocations are determined.

Measure: Actions to assess the feasibility and cost effectiveness of verifying hospitals bed counts, case-mix indices, and number of discharges.

Explanation:

Is the program purpose clear?

Explanation: The purpose of the Children's Hospitals Graduate Medical Education Payment Program (CHGME PP) is to provide funds to free-standing children's hospitals. The program does not explicitly support teaching activities because the children's hospitals can utilize the subsidy for any purpose

Evidence: Section 340E of the Public Health Service Act provides the formula for determining payments to children's hospitals, similar to how Medicare reimburses teaching hospitals. Payments are allocated among the participating children's hospitals according to the number of residents at each participating hospital, a hospital's case mix, average length of stay, and the number of beds. The number of residents a hospital is allowed to claim is capped at 1996 levels. The authorizing statute and regulations do not stipulate what activities hospitals may use CHGME funds for. In FY2002, 59 children's hospitals received payments totaling $276 million.

Does the program address a specific and existing problem, interest, or need?

Explanation: Children's hospitals receive GME funding from a number of sources besides the CHGME PP. Federal and state Medicaid funds, private insurance, and charity donations pay for GME in children's hospitals. Medicaid is budgeted to pay $2.1 billion in direct Federal GME payments in FY2003. Children's hospitals receive limited Medicare GME funds because very few of their patients are enrolled in Medicare. Medicare reimburses hospitals for GME because Medicare pays for services used by its beneficiaries, including GME costs. CHGME PP is not purchasing services for enrollees in a health plan'it is providing a general subsidy to children's hospitals. Children's hospitals are more likely to have positive margins than other hospitals, including teaching hospitals. In 1999, 25% of CHGME PP eligible children's hospitals had negative margins. In 1999, 34% of all hospitals and 43% of major teaching hospitals had negative margins. In 2000, 26% of children's hospitals had negative margins and 33% of all hospitals and 41% of major teaching hospitals had negative total margins.

Evidence: According to a 1998 survey conducted by the National Conference of State Legislatures, nearly all states in which medical schools are located make some level of special payments to teaching hospitals under the Medicaid program. GPRA reports provided children's hospital margins data and MedPac's "Annual Report to Congress: Medicare Payment Policy" provided hospital margins data. In 2001, 21% of children's hospitals had negative margins. We do not have reliable margins data on hospitals other than children's hospitals for 2001.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: Children's hospitals receive GME funding from sources besides the CHGME PP, including Medicaid, private insurers, and charitable donations. Children's hospitals receive roughly 45% of their patient care revenue from Medicaid. Medicaid will spend $2.1 billion in direct federal GME payments in FY2003. These payments do not account for special payment rates to children's hospitals or GME payments not explicitly formulated. In addition, HRSA's Training in Primary Care and Medicine and Dentistry grants provide funding for pediatric residents training. In FY2002, the program awarded $11.6 million in grants for General Pediatrics and Pediatric Dentistry. As of June 2003, the program had awarded $10.0 million in FY2003 grants for General Pediatrics and Pediatric Dentistry. This program has no budgetary request for FY2004, but currently constitutes a revenue stream for training pediatric residents.

Evidence: In 2001, children's hospitals received 45% of their gross revenue from patient care attributed to Medicaid, Medicare, and uninsured patients. Medicaid constituted the bulk of this revenue since payments from Medicare and uninsured patients is limited in children's hospitals. According to a 1998 survey conducted by the National Conference of State Legislatures, nearly all states in which medical schools are located make some level of special payments to teaching hospitals under the Medicaid program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program pays children's hospitals CHGME funds in a timely and accurate manner. However, by statute, the program pays children's hospitals on a bi-weekly basis. The program could improve efficiency by paying hospitals on a quarterly basis.

Evidence: Public Health Service Act Section 340E requires that eligible hospitals receive bi-weekly payments.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The intended beneficiaries of this program are children's hospitals. The formula and program processes require that the eligible hospitals receive the correct payment on a bi-weekly basis. The authorizing legislation lists eligibility requirements and the program reevaluates eligibility each year. Program data indicates that currently all eligible children's teaching hospitals that have applied are receiving CHGME PP funding.

Evidence: Public Health Service Act Section 340E stipulates the payment formula. The March 1, 2001 Federal Register notice outlines the implementation of the payment formula. A press release detailing the funding level for each hospital is released at the end of the fiscal year.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program adopted new long-term goals during the assessment process. The long-term measures focus on improving the accuracy of data used to compute payments to hospitals. CHGME will verify FTE resident counts and caps, and will verify bed counts, case-mix indices, and number of discharges reported by hospitals, contingent on the results of a pilot study to be implemented in FY 2006. The program is currently working to improve the accuracy of a key payment formula data element: full-time equivalent (FTE) resident counts. In FY2003, the program, under a contract with Blue Cross Blue Shield Association, assessed the FTE resident cap reported by each of the hospitals applying for funds as well as the weighted and unweighted FTE resident counts for each of the three Medicare Cost Report years used to determine the weighted and unweighted rolling averages. The weighted rolling average is used to determine DME payments and the unweighted rolling average is used to determine the IME payments.

Evidence: The program has two long-term measures: 1) Verify all hospitals' bed counts, case-mix indices, and number of discharges contingent on the results of pilot studies to be implemented in 2006; 2) Verify all hospitals' FTE resident counts and caps.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has adopted ambitious targets for its long-term goals. The program allocates funds to individual hospitals on a proportionate basis. A reporting error in one hospital may affect the size of allocations to all hospitals. Therefore, it is important to verify data provided by all hospitals. The program's annual goals will allow the program to achieve the long-term targets.

Evidence: The program has targets for each of its long-term goals: 1) Contingent upon the results of pilot studies, verify 100% of hospitals' reported data on bed counts, case-mix index, and number of discharges in FY2008; 2) Beginning with FY 2003, verify 100% of hospital FTE resident counts and caps.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: During the assessment process, the program adopted new annual performance measures that demonstrate progress towards long-term goals. These goals are to ensure all payments are made on time and to verify the accuracy of data used to compute payments.

Evidence: CHGME PP annual goals measure: 1) The percentage of payments to hospitals made every 2 weeks or 1month, as appropriate, throughout the fiscal year, subject to availability of funds and factors outside of programmatic control. Monthly payments are made early in each fiscal year during the period when final program allocations are being determined (This includes any continuing resolution); 2) Verification of all hospitals' FTE resident counts and caps; 3) Actions to assess the feasibility and cost effectiveness of verification of all hospitals' bed counts, case-mix indices, and number of discharges used in the final determination of payments. The program is not currently auditing each hospital's bed counts, case-mix indices and discharges. Achieving this goal will require intermediate steps before program-wide changes can be implemented, including: 1) Develop methodologies for verifying case-mix indices, bed counts, and number of discharges, and estimate costs of verification; 2) Pilot test the methodologies to ensure their feasibility and cost effectiveness, and 3) Contingent upon the results of pilot studies, develop a Federal Register notice and analyze comments; and 4) Contingent upon the results of pilot studies and responses to the Federal Register notice, implement additional verification procedures for case-mix index, bed counts, and number of discharges.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: During the assessment process, the program provided baselines and adopted targets for its new annual output measures.

Evidence: The baseline for all payments processed on time was 100% in FY2002. The target for FY2003-FY2006 is 100% of all payments made on time. The baseline percentage of hospitals whose FTE resident counts were verified in FY2003 is 100%. The targets are 100% for FY2004 and beyond. The baseline percentage of hospitals whose FTE caps were verified in FY2003 is 100%. The targets are to verify 100% in FY2004 and beyond. The baseline percentage of hospitals whose case-mix index, bed counts, and number of discharges were verified in FY2003 is 0%. The targets are to: 1) Develop methodologies for verifying case-mix indices, bed counts, and discharges, and estimate costs of verification in FY2005; 2) Pilot test the methodologies and determine feasibility/cost effectiveness in FY2006; and 3) Contingent upon the results of pilot studies, develop a Federal Register to solicit comments on any proposed changes in FY2007; 4) Contingent upon the results of pilot studies and comments received in response to the Federal Register notice, implement additional verification procedures for all hospitals' case-mix indices, bed counts, and number of discharges in FY2008.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program has expressed commitment to work towards the long-term and annual goals. The program's long-term and annual goals call for the program to seek input from program partners in determining the feasibility and cost effectiveness of verifying case-mix indices, bed counts, and discharges.

Evidence: Questions 2.1, 2.2, 2.3, and 2.4.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Children's Hospital Graduate Medical Education Program does not have regularly scheduled objective, independent evaluations that examine how well the program is meeting its long-term goals and recommend how to improve the program's performance.

Evidence: Moody's, a bond rating firm, publishes regular bond rating reports on children's hospitals. However, these bond reports are designed to evaluate the credit characteristics of children's hospitals. They comment favorably on CHGME, but do not evaluate the program or examine how well the program is accomplishing its purpose.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program allocation formula is specified by Congress in the authorizing legislation and annual requests appropriations are not based on a determination of resources needed to meet specific quantifiable goals.

Evidence: Section 340E of the Public Health Service Act, HRSA Congressional Justification

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Bureau of Health Professions (BHPr), the Bureau within HRSA that oversees CHGME PP, revised its strategic plan to address planning deficiencies noted during FY2004 PART reviews. The Bureau is also systematically reviewing all of its programs, including CHGME PP, using a logic model approach to articulate program missions, develop meaningful and measurable outcomes, and improve coordination among programs. The Bureau also plans to improve their data system to meet the data requirements of the new performance measures and publish standardized reports on BHPr programs on HRSA website. This process is in the early stages of implementation and is expected to take about two years.

Evidence: Strategic plan, performance measurement workgroup meetings, and program logic models.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: CHGME PP receives regular feedback from CMS, eligible children's hospitals, fiscal intermediaries, and the trade association on how to manage the program and improve performance.

Evidence: During the first cycle of the CHGME PP applications, freestanding children's hospitals were not sufficiently versed in the laws and regulations governing GME payments. In response, the program created a comprehensive Technical Assistance Program designed to teach representatives of these hospitals how to complete the CHGME PP applications and error rates were reduced. Eligible children's hospitals did not know how to establish an Medicare GME affiliation agreement with other hospitals. CHGME PP invited CMS policy analysts to provide a detailed explanation to eligible hospitals on how to establish affiliation agreements. On a Technical Assistance Conference call in October 2002, about 80 participants participated in a tutorial on affiliation agreements. After the conference call, the number of queries regarding affiliation agreements decreased significantly. The program also contracted with Medicare FIs to make CHGME FTE assessments a higher priority to allow hospitals to finalize their FTE resident counts within the CHGME PP time frame.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: The agency's senior management is held responsible for the operations of their programs, including performance results. HRSA reports that all of its SES personnel have performance contracts with goals, states and outcomes that are results oriented. In addition, there are four Federal Regional Managers who each take responsibility for approximately fifteen CHGME hospitals. The role of these managers is to ensure that the hospital understands and successfully complies with the law and the timelines of the CHGME PP. The hospitals are held accountable under federal law for reporting their data correctly.

Evidence: Each supervisor is rated yearly on their Performance Evaluation Plan (PEP) that includes rating for: (1) individual work management, (2) technical competency, (3) innovation, and (4) customer service. All information filed by the hospitals is subject to audit by the Department and the General Accounting Office. No audits have been conducted to date. However, the program has adopted goals to ensure the accuracy of hospital data.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: To date, all CHGME PP funds have been obligated and disbursed in a timely manner. By statute, payments are made on a bi-weekly basis and the program withholds 25 percent of the funds until the final determination of each hospital's payment amount is made in the spring of each year. All CHGME PP payments are disbursed by the end of each FY. In order to receive their proportionate share of CHGME PP funds, children's teaching hospitals complete an 'initial' and a 'reconciliation' application. CHGME PP has no oversight over how the hospitals utilize the funds.

Evidence: Section 340E of the Public Health Service Act outlines the formula, but does not give CHGME authority to oversee how the hospitals use the funds. On March 1, 2001, CHGME PP published a Federal Register notice detailing eligibility and payment methodology. On July 20, 2001, HRSA published an additional Federal Register notice detailing the methodology for determining FTE counts and the calculation of Indirect Medical Education (IME) payments. At the end of each fiscal year, the CHGME PP publishes a press release listing the total amount received by each of the children's teaching hospitals that applied for and received program funds.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The CHGME PP has efficiency targets related to: 1) processing applications; 2) estimating payments; and 3) distributing payments. To date, the CHGME PP has been able to make payment calculations and process award letters and vouchers within one week of receiving a budget for disbursement. The program has contracted with fiscal intermediaries (FIs) to perform reviews of FTE resident counts for those hospitals that file full Medicare Cost Reports, as well as for those that file low or no utilization Medicare Cost Reports (MCRs). The FIs submit an assessment of FTE resident counts for each reconciliation application to ensure that the hospitals' counts were made in accordance with program rules and regulations.

Evidence: In FY 2001, the CHGME PP developed streamlined application materials and obtained OMB approval to implement them FY 2002. Major improvements included simplification of the application form and enhancement of the guidance material to include an explanation of the legislative requirements, along with identification of references and sources that allow applicants to gain a deeper understanding of the issues. The CHGME PP application and associated guidance are available electronically on the CHGME PP web site. Because of the need for certification and assurances by the hospitals, the program also requires a hard copy with original signatures. The financial database used to calculate payments has been improved to facilitate the reallocation of funds overpaid prior to reconciliation, based on the final determination of FTE resident counts. An expanded program of technical assistance has reduced confusion related to Medicare GME rules, and decreased the number and types of errors that hospitals make on their applications.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Since CHGME PP is based in large part on Medicare rules and policies, CHGME PP has implemented several procedures to avoid overlap with CMS procedures, including verification of a children's hospital's FTE resident count. CHGME PP is currently working with CMS on the development of an alternative case-mix index for children. The trade association, the National Association of Children's Hospitals (NACH), computes the case-mix index for two thirds of the eligible hospitals. The program obtains aggregate data from NACH.

Evidence: HCFA Transmittal A-01-75 HCFA Transmittal AB-02-007

Does the program use strong financial management practices?

Explanation: The September 30, 2002 and 2001 independent auditor's report for HRSA identifies five reportable conditions. 1) Preparation and analysis of financial statements - HRSA's process for preparing financial statements is manually intensive and consumes resources that could be spent on analysis and research of unusual accounting. 2) Health Education Assistance Loan (HEAL) program allowance for uncollectible accounts ' HRSA's financial statements indicate limited success in collecting delinquent HEAL loans. 3) Federal Tort Claims Liability ' HRSA is unable to estimate its malpractice liability under the Health Centers program. 4) Accounting for interagency grant funding agreements ' HRSA's interagency grant funding agreement transactions are recorded manually and are inconsistent with other agencies' procedures. 5) Electronic data processing controls ' HRSA has not developed a disaster recovery and security plan for its data centers. Although HRSA's CHGME PP have not been cited specifically by auditors for material weaknesses, the above reportable conditions constitute weaknesses within HRSA and its Office of Financial Integrity. The Office reports directly to the Administrator and is intended to ensure procedures are in place to provide oversight of all of HRSA's financial resources.

Evidence: The audit assessment is based on the independent auditor's reports for 2001-2002.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: HRSA developed a corrective action plan to address the reportable conditions identified in the September 30, 2002 and 2001 independent auditor's report. For each aspect of the five reportable conditions, HRSA assigned an office responsibility. The plan also outlines milestones and target completion dates. During the PART process, HRSA adopted goals to explore the feasibility of verifying the case-mix indexes, discharges, and number of inpatients days reported by each hospital.

Evidence: Questions 2.1, 2.2, 2.3, 2.4 HRSA Corrective Action Plan for FY2002 Financial Statement Audits as of 4/30/2003.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: By law, the program is required to make a final determination of FTE residents counts. CHGME PP fiscal intermediaries verify the FTE counts and caps for each hospital. However, the program does not verify the case-mix indexes, discharges, and number of inpatients days used in the IME payment calculation. The program has adopted goals to explore the feasibility of verifying this hospital-reported data.

Evidence: Public Health Service Act Section 340E Question 2.1, 2.2, 2.3, and 2.4

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program does not provide hospital-specific data in an accessible format. The GPRA report provides aggregated data on the number of FTE residents trained in eligible hospitals, but does not provide hospital specific data. The program does not make publicly available aggregated or hospital specific data on bed counts, case-mix indexes, and discharges. The GPRA report also provides aggregate data on the proportion of all eligible hospital's gross revenue from patient care attributed to public insurance and uninsured patients and the percentage of hospitals funded by the program with negative total margins. The program publishes aggregate and hospital-specific funding levels. At the end of each fiscal year, the program publishes a press release detailing the total payment for each hospital.

Evidence: FY2004 GPRA Plan FY2002 HSRA press release on annual payments to eligible hospitals

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: The program has taken action to verify the hospital's FTE counts including comparing data with Medicare FIs and pervious years' data. In addition, the program commissioned with Blue Cross Blue Shield Association to assess the FTE resident caps and the weighted and unweighted FTE resident counts. The program has adopted a new long-term measure to verify all hospitals' bed counts, case-mix indices, and number of discharges contingent on the results of pilot studies. However, no actions have been taken to date to assess the feasibility and cost-effectiveness of additional verification for bed counts, case-mix indexes, and discharges in each hospital.

Evidence: The baseline year for these goals is 2003 and progress towards one of the goals has been started. The target year for verification of FTE caps and counts is FY2003. The target year for verification of case-mix indices, bed counts, and discharges, contingent upon the results of pilot studies comments received in response to the Federal Register notice, is FY2008.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program currently meets its goal of processing payments on time and 100% of hospitals' FTE residents caps and counts will be verified in FY2003. However, no actions have been taken to assess the feasibility and cost-effectiveness of additional verification for bed counts, case-mix indexes, and discharges in each hospital.

Evidence: Questions 2.1, 2.2, 2.3, and 2.4.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The program met the standards for a Yes in Question 4 of Section III due to steps taken to improve the efficiency. The program has implemented several technological improvements including placing the application on the web and documenting email correspondence with hospitals. There is no evidence of improved efficiency per Federal dollar at the actual program level, since any savings in administrative costs are transferred to the eligible children's hospitals or held until the next fiscal year.

Evidence: Question 3.4

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: Medicare and Medicaid Graduate Medical Education payments, Health Professions, and National Health Service Corps (NHSC) also support hospitals and other institutions that train health professionals. However, a unit cost comparison between these programs is inherently difficult due to the relative size of the programs and different outcome measures.

Evidence: NHSC tracks the number of patients served by the placement and retention of a NHSC clinician and the average Health Professional Shortage Area (HPSA) score of areas receiving a NHSC clinician. Health Professions tracks the proportion of persons who have a specific reliable source of continuing health care, the proportion of grantees completing funding program that are serving in medically underserved communities, and the proportion of grant recipients of an underrepresented minority or disadvantaged background. Medicare and Medicaid GME reimburse hospitals for services used by their beneficiaries.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: No comprehensive independent evaluations of CHGME PP have been conducted.

Evidence: Question 2.6