Detailed Information on the
Superfund Removal Assessment

Investigate the feasibility of outcome-oriented measures that test the linkage between program activities and impacts on human health and the environment.

Modernize the program's data repository (CERCLIS) to ensure accurate and complete information on program performance and financial management.

Develop a plan for regular, comprehensive and independent assessments of program performance.

Measure: Total Superfund-lead removal actions completed

Explanation:Superfund-lead removals are those actions entirely funded and completed by EPA. These actions reduce, control or eliminate exposure risks to human health and the environment.

Measure: Total voluntary removal actions overseen by EPA completed

Explanation:Voluntary removal actions are completed by potentially responsible parties without enforcement action and are overseen by EPA.

Measure: Superfund-lead removal actions completed annually

Explanation:Superfund-lead removals are those actions entirely funded and completed by EPA. These actions reduce, control or eliminate exposure risks to human health and the environment.

Measure: Voluntary removal actions, overseen by EPA and completed annually

Explanation:Voluntary removal actions are completed by potentially responsible parties without enforcement action and are overseen by EPA.

Measure: Superfund-lead removal actions completed annually per million dollars

Explanation:The measure is calculated by dividing the number of removals completed in a given year by the sum of total expenditures for each of those removal actions (including intramural and extramural program costs).

Is the program purpose clear?

Explanation: The CERCLA statute, as implemented through executive order, authorizes EPA to remove, or arrange for removal of released hazardous substances, pollutants or contaminants, consistent with the National Contingency Plan (NCP) to protect public health and welfare and the environment.

Evidence: The authorizing statute is the Comprehensive Environmental Response, Compensation and Liability Act (CERCLA) 42 USC 9604(a)(1). It specifies that primary attention is to be given to those releases that may present a public health threat. Executive Order 12580 (as amended) delegates the CERCLA authority to EPA. 42 USC 9601(14) and (33) define the petroleum exclusion.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses actual and possible releases of hazardous substance into the environment that present a threat to public health or welfare. The National Response Center (NRC), established through the NCP, receives notifications of releases or threats of releases of hazardous substances or oil, and other incidents. EPA has primary responsibility for responding to those releases or incidents that occur in the inland zone.

Evidence: National Response Center statistics show that more than half of the over 30,000 incidents reported in 2004 were referred to EPA's Removal program.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: States, territories, responsible parties and other federal agencies conduct emergency response and removal operations for hazardous substance releases. EPA's Removal program acts as a "safety net" because even those with advanced response programs do not have the capacity to address all removal actions. EPA's Removal program also works with other federal programs to mitigate the effects of large-scale natural disasters and terrorist events. The responsibilities of each entity are clearly defined in law and/or through memoranda of understanding (MOUs).

Evidence: The NCP clearly identifies the distinct roles and responsibilities of federal and state entities for response to releases of hazardous substances, pollutants and contaminants. EPA's unique role as the responder of last resort is evident in NCP requirements and MOU delineations of the roles and responsibilities of each party. The National Response Plan (NRP, Emergency Support Function #10) defines the roles and responsibilities of federal agencies for response and recovery following an Incident of National Significance. EPA is the lead federal agency for hazardous materials response.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: By design, the program has the ability to optimize its effectiveness by adjusting incident response according to need and field conditions. There is no evidence that another mechanism would be more efficient or effective to achieve the intended purpose.

Evidence: Regulatory and prevention programs might reduce the overall number of incidents, especially where responsible parties can be identified. However, these programs will not eliminate the need for federal emergency services, especially for Incidents of National Significance as defined in the NRP.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program acts as a "safety net" for removals beyond the abilities of other government entities or private parties. The program does not take action unless it is beyond the scope of other agencies or responders, and then works closely with all applicable agencies and organizations in conducting the response action. The NRC's notification system ensures that each incident is directed to the Agency with the proper response apparatus.

Evidence: The National Response Center Analysis Report (June 2005) describes the NRC's process of referring notifications to response agencies including EPA. The MOU between EPA and USCG for oil spill responses includes a process to determine when an oil spill response is appropriate. The NCP and NRP identify the process used by On-Scene Coordinators (OSCs) to determine if a CERCLA response is appropriate.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has two long-term output measures that focus on the aggregate completions over time of two types of removal actions; Superfund-lead and voluntary. Completion of Superfund-lead removals is assumed to accelerate and improve the overall performance of the Superfund program. The program is investigating ways to test this assumption. Completion of voluntary removals reduces the need for EPA intervention, thereby conserving EPA resources. The program is investigating outcome measures that focus on risk reduction due to removal actions, however, OMB and the Agency agree that these measures are currently not feasible due to the difficulties in accurately estimating exposure and population variables.

Evidence: The Agency has committed to including these measures in forthcoming GPRA documents, including the FY 2006 Performance and Accountability Report. The program has provided to OMB an acceptable description of each measure and a methodology for implementation including baseline and targets.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has ambitious targets and has defined baseline for both of its long-term measures. The program defines its targets based on the range of removals completed in past years and taking into account the difficulty of predicting the number and size of future emergencies. The long-term targets are an aggregation of the annual targets.

Evidence: For the past three years, the program has completed between 184 and 221 Superfund-lead removals annually. For the past three years, the program has overseen completion of between 109 and 114 voluntary removals annually.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has two annual output measures that support long-term measures by tracking annual completion of the two types of removal actions; Superfund-lead and voluntary. OMB and the Agency agree that annual outcome measures are currently not feasible due to data collection obstacles and burdens.

Evidence: The Agency has committed to including these measures in forthcoming GPRA documents, including the FY 2006 Performance and Accountability Report. The program has provided to OMB an acceptable description of each measure and a methodology for implementation including baseline and targets.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has ambitious targets and has defined baseline for both of its annual measures. The program defines its targets based on the range of removals completed in past years and taking into account the difficulty of predicting the number and size of future emergencies.

Evidence: For the past three years, the program has completed between 184 and 221 Superfund-lead removals annually. For the past three years, the program has overseen completion of between 109 and 114 voluntary removals annually.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Federal partners are identified in the NCP and NRP. Their commitments to the program's readiness and response goals are documented in interagency MOUs. Contractors provide only site-specific assistance and are not relied upon for performance information. PRP (Potentially Responsible Party) commitments to achieve the goal of voluntary removals are evident in EPA pollution reports that document the results of voluntary actions.

Evidence: Example pollution reports and several example MOUs between EPA and the Coast Guard were provided by the program.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There are no regularly scheduled independent evaluations of this program. There have been evaluations of the overall Superfund program, but these are not sufficiently rigorous to meet the quality standards required by the PART.

Evidence: Evaluations have been conducted externally by the General Accounting Office and some nongovernmental organizations (Resources for the Future) and internally by EPA's IG and others, but they do not adequately address program effectiveness.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: There is strong evidence that the program monitors and can define the relationship between changes in policy/legislation/funding and achievement of annual or long-term goals. Additionally, internal program budget presentations to management demonstrate budget-performance integration. The Agency budgets for the full cost of the Superfund Removal program and presents both direct and indirect costs in it's submission to OMB, but does not present full cost in its Congressional Justification (CJ). The program does not tie funding to performance levels in the CJ or any other Agency budget request document.

Evidence: EPA's CJ for the FY 2006 President's Budget includes a description of the Superfund Removal program that presents direct costs only and describes the work products for FY 2006 in only general terms. Budget Automation System (BAS) reports submitted to OMB with the Agency's FY 2006 Budget request provided a program-specific breakout of direct and indirect costs. The program provided materials used for presentations to the Division Director that reported integrated funding and performance information for FY 2004. A separate Director's presentation used integrated budget-performance information as part of strategic planning for emergency response.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: EPA's newly formed Office of Emergency Management (OEM) was established to integrate components of emergency planning, accident and spill prevention, and emergency response that were previously distributed throughout EPA's Office of Solid Waste and Emergency Response. The reorganization improves internal coordination and consolidates some functions for more effective program support. The Evaluation and Communication Division in the new OEM will focus on strategic planning and program performance evaluations, including the Removal program. The Superfund Removal program is spearheading implementation of EPA's initiatives to support the National Approach to Response (issued July 2003)

Evidence: Recent strategic planning initiatives that involve the program include: (1) reorganization (initiated September 2004) that defined goals and objectives for the new OEM that includes the Removal Program; (2) identification of priority needs to address EPA's role in the National Approach to Response and establishment of work groups; and (3) the Administrator's 120-Day Study of Superfund that included specific recommendations to improve performance and management of the Removal Program.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: EPA collects information on Removal Actions quarterly and end-of-year. Quarterly data are used to make mid-year funding adjustments based on highest priority national needs. Data related to protection of human health and the environment are used to help inform ongoing response decisions, strategic planning, and program management. Core Emergency Response evaluations are used to identify highest priority program needs and to take action to improve readiness. Regional Response Teams (including states and local governments, and other federal agencies) and the National Response Team produce after-action and lessons-learned reports to improve program performance and readiness. Managers are regularly provided with performance data from the Agency's CERCLIS database to identify program trends and help clarify priorities. The data extracted from CERCLIS periodically undergo additional verification by regional program managers. A large effort is underway to modernize CERCLIS and improve data accuracy and completeness.

Evidence: Action Memos and Pollution reports provide site-specific data that are used to make funding and/or schedule changes. Core Emergency Response Evaluation reports document the results of readiness evaluations in each regional office. CERCLIS database stores removal action data including removal quantities, removal costs, time to complete, etc., that are necessary to measure program performance.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The federal managers with responsibility for program performance at the national level are held accountable through performance agreements specified in their annual appraisal form. OEM's organization chart identifies the managers responsible for Removal program performance and results. Additionally, the program has Removal Managers in each Regional office that are responsible for day-to-day operations. Contractors are held accountable for meeting costs and schedules through regular progress reports. The program's major contract vehicles specify a performance standard. Award of subsequent task orders is based on compliance with the standard. Additionally, the contractor's performance is evaluated at the end of each twelve months of contract performance in accordance with the requirements of EPA's Office of Acquisition Management.

Evidence: Example appraisal form for OEM Director specifies responsibility for monitoring and managing to meet performance goals. Every contract (since 2002) includes EPAAR clause 1552.209-76, which specifies the frequency, scope and process for contractor performance evaluations. The evaluation includes an assessment of timeliness, cost control and quality of service. The main Removal program contracts are: Emergency Rapid Response Services (ERRS) and Superfund Technical Assistance and Response Team (START). Both include compliance with the performance standard as a criterion for award of multiple or subsequent task orders.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Removal advice of allowance (AOA) funds are distributed to EPA Regions quarterly and fully obligated each year for taking removal actions and maintaining EPA's response readiness. Reserve and deobligated funds that become available during the year are distributed to regional offices in accordance with a national prioritization of projects. The program's financial data (including commitments, obligations and payments) are transferred from funding documents to the Agency's Integrated Financial Management System (IFMS). Removal program managers can identify improper payments by comparing CERCLIS/WasteLAN data to the original funding documents.

Evidence: Budget execution reports for 2004 show that program resources were fully obligated. The program provided a tabulation of unobligated balances for the past three years showing that 97%-99% of funds are obligated each year. The Superfund Program Implementation Manual (SPIM) describes the financial management systems, accessible by the program, that are used to record and track obligations and to verify payments. IFMS data are downloaded nightly to CERCLIS/WasteLAN, the program's in-house database that tracks site-specific, as well as non-site-specific financial and performance data. The program has procedures in place for entering, checking and correcting financial information in the Agency's IFMS.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has an annual efficiency measure that focuses on cost efficiency for Superfund-lead removal actions (i.e., removals completed per million dollars). The program has established a baseline and targets for this measure. National managers meetings and workgroups, including the OSC Readiness Task Force, provide regular forums to identify and develop best management practices. The program anticipates efficiency gains from best practices that improve coordination and consistency. The same applies to the recently formed Emergency Management IT Forum which is charged to provide a nationally consistent, comprehensive and cohesive vision for the coordination of emergency management data, information, and technology across Regions and EPA Headquarters.

Evidence: The program has provided to OMB an acceptable description of the efficiency measure and a methodology for implementation including baseline and targets. The Emergency Management IT Forum was formed in November 2004. Among the expected outcomes listed in the charter, two focus on efficiency: (1) defining streamlined information management processes and (2) identifying tools to streamline and enhance the operations to satisfy business needs.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The National Response System is designed and implemented to ensure close coordination and collaboration among all federal and state emergency response programs. The Removal program participates fully in national training and emergency drills. Regional Response Teams (including states and local governments) and the National Response Team produce after-action and lessons-learned reports to improve program performance and readiness.

Evidence: Several examples of post-event evaluations document successful collaboration during the incident and record collaborative learning activities after the event. These include "Lessons Learned" reports on the Columbia Shuttle disaster, hurricane support, train derailment response and the TOPOFF exercises conducted nationally for homeland security.

Does the program use strong financial management practices?

Explanation: Auditor reports indicate that the program has no material weakness or other financial non-compliance issues. The program is served by three Funds Control Officers (FCOs) that oversee all accounts in OEM. These FCOs have documented experience and/or training in EPA's budget execution and financial management systems. Interfacing with the Agency's financial system is the program's CERCLIS/WasteLAN database, an integrated financial and performance data system. The program uses it for some aspects of day-to-day management, including review for improper payments. In addition to the Agency's Funds Control Manual, OEM publishes division-specific financial management guidelines that address payroll accounts, spending authority, packaging and processing agreements, travel and procurement.

Evidence: "Guidelines for Allocating and Managing Program Resources" (May 2005) and example SOPs provided by the program provide OEM-specific procedures for managing resources. OSWER Directive 9200.3-14-1G-R describes the financial data management systems and tools used by the program. No material weaknesses were reported under Federal Managers' Financial Integrity Act (FMFIA) or in the Agency's 2004 Performance and Accountability Report.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The EPA Administrator's 120-day study of the entire Superfund program included 108 recommendations including several that focused on management of the Removal program. By February 2005, the program had completed action on five of the ten recommendations for which the Removal program was responsible. Significant progress was made on an additional three recommendations by June 2005 and the remainder is underway.

Evidence: 120-Day Study Action Plan (February 2005) identifies the EPA offices responsible to address each recommendation and describes the approach including work underway or completed.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Results show that the program met its long-term goals in 2004 and annual results in 2005 show that the program is on track to meet 2007 targets. Prior to 2004, the program tracked only removal project starts, not completions which are the focus of the PART measures.

Evidence: Results are reported under the Performance Measures Section of the PART. The Agency's CERCLIS database tracks removal action status including the date of completion.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Results show that the program met its annual goals in 2004. Prior to 2004, the program tracked only removal project starts, not completions which are the focus of the PART measures. Results for 2005 show that the program exceeded its target for one measure (voluntary removals), but did not meet its target for Superfund-lead removals. Both results reflect the difficulty in predicting emergency response demands. In 2005, fewer Superfund-lead removals were completed because an unusually large number of the program's people and resources were diverted to disaster response (i.e., hurricane recovery).

Evidence: Results are reported under the Performance Measures Section of the PART. The Agency's CERCLIS database tracks removal action status including the date of completion.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has not provided evidence that its efficiency-promoting practices and initiatives have yielded cost effective innovations or actual efficiency gains. In 2005, the program exceeded its target for the new efficiency measure, reporting over 1.5 Superfund-lead removals per million dollars, up from 0.9 in 2004.

Evidence: Results are reported under the Performance Measures Section of the PART.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No report or other acceptable documentation was found that compared EPA's Removal program with other emergency response or cleanup programs.

Evidence: None.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Existing evaluations are not independent and/or sufficiently comprehensive to draw conclusions about the effectiveness of the program. An older GAO report (1996) concluded that certain types of removal actions have expedited cleanup at Superfund sites, making cleanup more cost effective. A more recent study (2001) by the independent research group Resources for the Future (RFF) also supported this contention.

Evidence: GAO/RCED-96-134R (April 15, 1996) evaluated the benefits and disadvantages of non-time-critical removals in Superfund cleanup. A report to Congress (RFF, 2001, "Superfund's Future: What Will it Cost?") evaluated the use and cost of the Removal program in Superfund.