Detailed Information on the
Emergency Medical Services for Children Assessment

Developing methodogy and arranging with AHRQ to obtain and analyze data for newly approved long-term and annual measures.

Administration will propose eliminating funding of categorical grants for EMSC due to the program's failure to demonstrate results

Program to conduct independent evaluation via contract.

Develop and propose a long-term health outcome measure for the program to more effectively guage program performance.

Measure: Decrease the applicaton and reporting time burden of grantees by 5% per year for 4 years, thereby collecting more accurate and timely data.

Explanation:Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Is the program purpose clear?

Explanation: By statute, the purpose of the Health Resources and Services Administration's (HRSA) Emergency Medical Services for Children (EMSC) program is clear. The statute indicates that the Secretary "may make grants to States or accredited schools of medicine in States to support a program of demonstration projects for the expansion and improvement of emergency medical services for children who need treatment for trauma or critical care". EMSC is a joint partnership with the Department of Transportation's National Highway Traffic Safety Administration. The program articulates its ultimate goal it to reduce child and youth morbidity and mortality resulting from severe illness or trauma by supporting injury prevention programs and improvements in the quality of medical care children receive. The focus is all levels of the EMS system, from paramedics to emergency departments. State offices of EMS are responsible for ensuring that State-wide guidelines exist for individual public and private EMS companies so that all residents have access to EMS that meets minimal requirements.

Evidence: Evidence 1. EMSC is authorized under Section 1910 of the Public Health Service Act, as amended (42 USC 300w-9) 2. Project HOPE - Federal Funding for Emergency Medical Services: Final Report (July 1, 2002) Background State EMS systems began in the 1950s and initially were designed to provide rapid intervention for heart attacks in adults and rapid transport for motor vehicle crash victims, with no specific focus on children. (Many injuries were caused as a result of automobile accidents.) EMSC was established in 1984, to address children's needs. Infants and smaller children often require smaller sized equipment. It is difficult to start an IV in an infant and infants and young children cannot talk and explain symptoms.

Does the program address a specific and existing problem, interest or need?

Explanation: When EMSC was established States did not have pediatric protocols integrated into their EMS systems. To-date, considerable progress has been made and 44 states have implemented state-wide pediatric protocols for medical direction; however, there is a small number of States that have not incorporated state-wide pediatric protocols. In addition, all but 3 states require all EMSC-recommended pediatric equipment is onboard Advanced Life Support ambulances. It is the case that children are injured each year; however, many States' are now better equipped to handle occurrences of critical or traumatic injury. This progress is consistent with the purpose/intent of the program.

Evidence: Evidence HRSA Annual GPRA Plan (FY 2005) Background The components of EMSC are: 1) State Partnership Grants to institutionalize pediatric EMS improvements; 2) Targeted Issue Grants to demonstrate the effectiveness of a model system that may be helpful to the field; 3) Network Development Demos with academic institutions for infrastructure development and personnel costs, while the network competes for outside research funding to investigate the efficacy of treatments, transport and care; 4) Natl Data Analysis Resource Ctr to collect and analyze data and communicate findings, develop research designs, provide TA to grantees; 5) Natl Resource Ctr, five-year contract with Children's Hospital in Washington, DC, to establish a national internet-based clearinghouse to identify resources available for EMSC activities, and provide TA to program staff; and 6) Regl Symposium grants to support coordinating, exchanging and dissemination knowledge that leads to reducing child and youth disability and death due to severe illness and injury.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: EMSC is complementary of other Federal programs. It is the only program that has improved services for children as its target. Since 1985, more than $190 million (M) was appropriated to EMSC. In FY 2004, it will award grants to States and medical schools ($14M); contracts for technical assistance (TA) and studies ($5M); and cooperative and interagency agreements ($1M). EMSC priorities include: 1) education & training, 2) equipment & supplies, and 3) evaluation & research.Since the early 1980s, the EMS Division of the Department of Transportation's National Highway Traffic Safety Administration (NHTSA) has partnered to improve EMS systems. In FY 2004, NHTSA will support the development/enhancement of comprehensive EMS systems to reduce deaths and injuries on highways ($2M), for such things as: 1) training, 2) research, 3) planning, and 4) demonstrations for integrated pre-hospital/hospital trauma systems.Since 1992, HRSA's Trauma/EMS program received $27M. It will contribute $4M in FY 2004 to: 1) conduct & support training, evaluations, and demos, 2) foster comprehensive state-wide systems, 3) collect & disseminate information, and 4) provide TA.

Evidence: Evidence1. Project HOPE - Federal Funding for Emergency Medical Services: Final Report (July 1, 2002)2. Academy of Emergency Medicine - Pediatric Emergency care Applied Research Network (PECARN): Rationale Development, and First Steps, Vol. 10, No.6 (June 2003) 3. www.nhtsa.dot.gov/people/injury/ems/4. www.hrsa.gov/grants/preview/guidancedot/hrsa04080.htm

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The current program design is not free of major flaws that may limit the program's effectiveness and efficiency. When EMSC was first authorized, States' EMS systems focused primarily on adult emergency needs. Since 1984, more than $190 million has been dedicated by EMSC to address children's needs and 44 states have developed state-wide pediatric protocols. The program focused on ensuring that States have infrastructure that includes EMSC components. Only a small number of States have not established state-wide pediatric protocols. The current structure of the program does not allow for the targeting of resources to those States that have been unable to make infrastructure and other changes on scale with other States.

Evidence:  

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Although the program's design could be more efficient and effective, the EMSC program does address its purpose and intended beneficiary population, as 44 states have developed state-wide pediatric protocols since the inception of the program.

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: No long-term health outcomes measures exist for the program or were developed for the FY 2006 PART. The program believes the purpose of EMSC is focused on system preparedness and thus has and continues to measure input into and outputs from systems, rather than how progressively prepared systems impact health outcomes. In March 2004, the program worked with a 25-member group of grantees, resource center personnel, Federal partners, and emergency care professionals to establish output measures that: 90 percent of States will have the operational capacity to provide pediatric emergency care and 100 percent of States will have adopted requirements for pediatric emergency education for recertification or paramedics. Other pre-existing long-term measures address the number of States that require all EMSC-recommended pediatric equipment on Advance Life Support ambulances and have implemented State-wide pediatric protocols for medical direction.

Evidence: Evidence HRSA FY 2005 GPRA Report

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has not established long-term health outcomes measures; therefore, associated ambitious targets with clear time frames have not been developed.

Evidence:  

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program lacks a long-term health outcome goal. Therefore, the program does not have annual performance measures that directly support a long-term outcome goal. The program has a developmental efficiency measure. HRSA's Maternal and Child Health Bureau anticipates implementing a new web-based grant application system by the end of FY 2004 to streamline the grant application process.

Evidence:  

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has not established long-term health outcomes measures; therefore, associated ambitious targets with clear time frames have not been developed. The program's developmental efficiency measure does not have a baseline. HRSA's Maternal Child Health (MCH) Bureau plans to implement a new web-based discretionary grant application system by the end of FY 2004 to streamline the grant application process. HRSA's MCH Bureau expects that the web-based application system will reduce the time needed to complete an application by 5% per year for the next four years. Once the system is in place, the program will be able to establish baselines and targets.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Because long-term health outcome measures have not been developed for the FY 2006 PART, partners and grantees do not commit to and work toward the annual and/or long-term goals of the program. In addition, the program's major partner is States. Not all States have made implementing pediatric protocols and other related goals of the program their own priorities. NHTSA works in partnership with EMSC to develop the goals and measures, but is not clearly held responsible for its progress in helping to achieve the goals.

Evidence: Evidence 1. Interagency Agreement between HRSA's Maternal and Child Health Bureau and the Department of Transportation's National Highway Traffic and Safety Administration (FY 2004) 2. Emergency Medical Services for Children State Partnership and Targeted Issue Grants Guidance (FY 2004) 3. Emergency Medical Services for Children Cooperative Agreement Application Guidance for National EMSC Data Analysis Resource Center Demonstration Grant (FY 2004) 4. National Emergency Medical Services for Children Resource Center Contract with Children's Hospital in Washington, DC (FY 2004)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Objective, high quality, independent evaluations are conducted but not a regular basis. Only two independent evaluations have occurred since its inception in 1984. However, a new assessment of its impact over the past 20 years has been initiated this year, which will look at the overall EMS system and will address the impact/effectiveness of Federal efforts in EMSC and the need/appropriateness of Federal resources in the context of the overall EMS system. The Institute of Medicine (IOM) report in 1993 was requested by the Senate in appropriations report language. The IOM undertook 'a study of pediatric emergency medical services to look at the issues more broadly than individuals demonstration projects could.' Thus the study is primarily a general assessment of States' EMS systems and EMS for children broadly, not a specific assessment of the EMSC program within HRSA. A 1996 seven state evaluation was completed by George Washington University; it assessed States' ability to sustain EMS for children.

Evidence: Evidence 1. IOM - Emergency Medical Services for Children (1993) 2. George Washington University - EMSC, An Evaluation of Sustainability in Seven States (1996)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: EMSC does not provide a presentation that makes clear the impact of funding, policy or legislative decisions on expected performance nor does it explain why a particular funding level/performance result is the most appropriate.

Evidence: Evidence DHHS Federal Fiscal Year Justification of Estimates for Appropriations Committees

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The majority of deficiencies highlighted in questions 2.1 through 2.7 have not been addressed. The program does not have any plans to develop health outcome goals. The program is, however, developing an efficiency measure that would apply to all HRSA Maternal and Child Health Bureau programs in the reporting of financial and program performance data. It is anticipated that a new, on-line, web-based system will be implemented by the end of FY 2004. The program also anticipates that this system will greatly reduce the application and reporting burden for grantees. Baseline data are not yet available, but are expected by the end of FY 2004. The program is aiming to reduce the amount of time it takes to complete applications by at least 5 percent per year for the next 4 years. Also during FY 2004, EMSC contracted with the Lewin Group, a national health care and human services consulting firm, to develop performance measures to assess grantee performance. This contract has lead to the long-term and annual output performance measures referenced in 2.1 and 2.3.

Evidence: Evidence HRSA contract with the Lewin Group

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program requires all grantees to complete semi-annual reports that document performance towards stated goals and objectives. Information from these reports and the grantee survey are made available to the public through a variety of mechanisms. EMSC also produces fiscal year highlights that enumerate progress for that year. This information is made available to grantees and the public on the EMSC website. In addition, in response to the 1993 IOM report, the program developed a five year strategy composed of program objectives. A new five-year strategic plan was published in 2000 with baseline data for each objective. Midcourse reviews of the plans were also completed.

Evidence: Evidence 1. Emergency Medical Services for Children 5-Year Plan (1995-2000) 2. Emergency Medical Services for Children 5-Year Plan, Midcourse Review (1995-2000) 3. Emergency Medical Services for Children 5-Year Plan (2001-2005) 4. Emergency Medical Services for Children 5-Year Plan, Midcourse Review (2001-2005)

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers for the EMSC program are officers in the Public Health Service Commission Corps. Commission Corps members receive a standard annual performance evaluation. While the performance of the EMSC Program can be considered in the evaluation of the Program Director and supervising Division Director, evaluations do not explicitly consider the management oversight of the program's performance, costs, and schedule. The program's GPRA goals are not required to be considered as part of the Federal managers' formal performance assessment. However, all grantees are held to fulfilling any conditions placed on their grants. Progress toward meeting grant conditions is monitored by both program and grants management staff. Changes in the objectives of the grant project must be submitted for approval by the Program Director. Contractors are closely monitored and include specific and measurable deliverables.

Evidence: Evidence 1. Commission Corps Annual Performance Assessment 2. Emergency Medical Services for Children State Partnership and Targeted Issue Grants Guidance (FY 2004) 3. Emergency Medical Services for Children Cooperative Agreement Application Guidance for National EMSC Data Analysis Resource Center Demonstration Grant (FY 2004) 4. National Emergency Medical Services for Children Resource Center Contract with Children's Hospital in Washington, DC (FY 2004)

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All funds are obligated to grantees in a timely manner. Notice of grant awards are sent to grantees within 30 days of the grant start date. Grants Management Specialists within HRSA monitor budget expenditures and inform grantees if funds are not being expended on schedule. Grantees can then modify their expenditure timeline to meet the agreed upon schedule for expending funds. Also, at the end of the prior year, grantees submit Financial Status Reports which indicate whether they funded what agreed to fund. During the fourth quarter, grantees must notify HRSA of any unobligated balances and must submit a request to use these funds.

Evidence:  

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program is developing procedures to measure and achieve efficiencies and cost effectiveness in program execution. EMSC grant applications are currently paper-based. HRSA's Maternal and Child Health (MCH) Bureau is in the process of implementing a web-based grant application system, which will be completed by the end of FY 2004. In addition, EMSC out sources technical assistance through a competitive contract with Children's Hospital in Washington, DC to serve as a National EMSC Resources Center.

Evidence: Evidence Beginning in September 2004, all MCH Bureau applications will be web-based.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program effectively collaborates and coordinates with multiple programs that have related purposes. EMSC activities complement the activities of other Federal agencies. EMSC coordinates with Federal and nonfederal entities. The Department of Transportation's National Highway Traffic Safety Administration has partnered with EMSC on topics ranging from EMS provider education to public information and education, to research and evaluation. HRSA's Trauma/EMS program focuses on States' EMS infrastructure, which supports the EMSC component. EMSC also works closely with national organizations involved with EMS, medicine, nursing and public health. These groups represent researchers, educators, physicians, nurses, emergency medical technicians, and allied health providers.

Evidence: Evidence Interagency Agreement between HRSA's Maternal and Child Health Bureau and the Department of Transportation's National Highway Traffic and Safety Administration (FY 2004)

Does the program use strong financial management practices?

Explanation: In FY 2003, HHS OIG conducted an HHS financial statement audit. The audit reported that the Department had serious internal control weaknesses in its financial systems and processes for producing financial statements. OIG considered this weakness to be material. The audit recommended that HHS improve their reconciliations, financial analysis, and other key controls. The September 30, 2002 HRSA independent auditor's report found that the preparation and analysis of financial statements was manually intensive and consumed resources that could be spent on analysis and research of unusual accounting. The audit also found that HRSA's interagency grant funding agreement transactions were recorded manually and were inconsistent with other agencies' procedures. Finally, the audit found that HRSA had not developed a disaster recovery and security plan for its data centers.

Evidence: Evidence 1. HRSA - Annual Report (FY 2002) 2. HHS Performance and Accountability Report (FY 2003)

Has the program taken meaningful steps to address its management deficiencies?

Explanation: HHS' long-term strategic plan is to resolve the internal control weaknesses is to replace existing accounting systems and other financial systems within HHS with the Unified Financial Management System (UFMS). HHS plans to fully implement the UFMS Department-wide by 2007. HRSA developed a corrective action plan to address the reportable conditions identified in the September 30, 2002 independent auditor's report. For each aspect of the five reportable conditions, HRSA assigned an office responsibility. The plan also outlines milestones and target completion dates.The program is developing a new efficiency measure during the PART process. HRSA's Maternal and Child Health Bureau anticipates implementing a new, on-line, web-based system for all discretionary grant programs (non-block grant) before the end of FY 2004. This system will be used by all discretionary grantees in submission of their applications and in the reporting of financial and program performance data. The program anticipates that this system will greatly reduce the application and reporting burden for grantees. In addition, the program is working with the National Resource Center, who maintains the current site (www.ems-c.org) to change the domain from .org to .gov. Also, the program made a link to its National EMSC Data Analysis Resource Center more prominent on the ems-c.org site.

Evidence: Evidence1. www.ems-c.org2. Beginning in September 2004, all MCH Bureau applications will be web-based.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: EMSC has awarded new and continuing grants through a clear competitive process that includes an assessment of merit. All new EMSC grants are peer-reviewed through HRSA's Division of Independent Review. Reviewers are selected based upon a careful review of their area of expertise and the focus area of submitted proposals as stated from the applicant Letter of Intent. All continuation grants are evaluated for successful progress on completing approved grant objectives. Contracts include specific and measurable deliverables.

Evidence: Evidence 1. HRSA contract with the National Resource Center 2. Emergency Medical Services for Children State Partnership and Targeted Issue Grants Guidance (FY 2004) 3. Emergency Medical Services for Children Cooperative Agreement Application Guidance for National EMSC Data Analysis Resource Center Demonstration Grant (FY 2004) 4. National Emergency Medical Services for Children Resource Center Contract with Children's Hospital in Washington, DC (FY 2004)

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: EMSC collects information on contracts, grant objectives, accomplishments and products produced. The program uses HRSA's Grants Electronic Management System to track grantee financial status and to maintain follow-up on grant conditions and recommendations. The EMSC program has 1 FTE associated with it. To maximize resources, EMSC contracts with the Children's Hospital in Washington, DC; it serves as a National EMSC Resource Center (NRC). NRC provides technical assistance and makes the Program Director aware of all activities on a regular basis. There is daily communication between the program and NRC. EMSC has an active indirect oversight role.

Evidence: Evidence HRSA contract with the National Resource Center

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects grantee performance data on an annual basis, the information is available to the public in a transparent and meaningful manner. The link to the National EMSC Data Resource Center web-page that displays the annual assessments is easily located on the www.ems-c.org web-site. Grantees complete semi-annual reports that document performance towards stated goals and objectives. Information from these reports and the grantee survey are made available to the public through a variety of mechanisms. EMSC also produces fiscal year highlights that enumerate the progress of the program for that year. This information is made available to grantees and the public on the EMSC website.

Evidence: Evidence 1. www.ems-c.org2. nedarc.med.utah.edu or www.nedarc.org

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has not adopted long-term health outcome goals. The outcome of the program should be to improve the health and/or well-being of traumatically injured children who access the EMS system.

Evidence: See Questions 2.1-2.2

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has not developed long-term health outcome measure associated annual goals.

Evidence: See Questions 2.3-2.4

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: During the PART process, the program developed an efficiency measure. HRSA's Maternal Child Health Bureau anticipates implementing a new web-based grant application system by the end of FY 2004 to streamline the grant application process. Once the system is in place, the program will be able to track progress towards the new efficiency measure.

Evidence: See Question 2.8

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Other programs focus mainly on developing EMS systems; EMSC focuses on ensuring that States' EMS systems include services for children.

Evidence: See Question 1.3

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Both evaluations conclude that prior to the EMSC program States' systems were not highly developed, and States' commitment and organizational support varied. Training and equipment were minimal. As a result of both evaluations, the program began implementing changes. George Washington University's seven state evaluation found that EMSC was "highly successful in achieving some of its goals, particularly in the area of training and education, systems capacity development, coalition building, product development, and knowledge transfer". Also, "State legislatures, state and local agency officials, providers and communities were educated to the special needs of ill and injured children." In addition, there were barriers with grantees. "Staff turnover, poor project management, lack of coordination with [the Maternal and Child Health Bureau], and changes in grant guidance posed problems .... " The Institute of Medicine report found that 'EMS-C must establish three important linkages. First, the separate components of EMS-C must be connected to form a system. Second, EMS-C must be integrated into the larger EMS system. Third, EMS-C must develop strong ties to the broader elements of child health care.'

Evidence: Evidence 1. George Washington University - EMSC, An Evaluation of Sustainability in Seven States (1996) 2. IOM - Emergency Medical Services for Children (1993)