Detailed Information on the
Drinking Water Research Assessment

Develop baselines and targets for all long term and annual performance measures. These will allow the program to set quantitative goals and assess progress through time.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

Develop a performance measure which tracks the efficiency with which the program delivers its services to its primary client, the EPA Office of Water.

Improve oversight of non-grant partners and require non-grant partners to work towards the annual and long term goals of the program.

Measure: Usefulness of ORD's characterization methodologies, data, and tools by EPA's Office of Water and other key stakeholders in developing health risk assessments, producing regulatory decisions, implementing new and revised rules, and achieving simultaneous compliance under the Safe Drinking Water Act.

Explanation:This measure captures the assessment by an independent expert review panel of the relevance, quality, and use of the program's research under Long-Term Goal 1. Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews. Targets for this measure are set using the previous BOSC rating??and BOSC recommendations??as a guide. The program outlines an action plan in response to BOSC recommendations; completion of the actions in this plan demonstrates progress from the baseline. The BOSC's 2007 report can be found at http://www.epa.gov/OSP/bosc/pdf/dwmc0708rpt.pdf. The program's formal action plan can be found at http://www.epa.gov/OSP/bosc/pdf/dwmc0801resp.pdf. Work under DWRP's LTG 1 primarily supports the Office of Water in decisions relating to: Unregulated Contaminant Monitoring Rule (UCMR), regulating/not regulating contaminants on the Contaminant Candidate List (CCL), the six year review, and the Underground Injection Control (UIC) program. ORD's work under this goal also supports regions and key stakeholders in meeting simultaneous compliance requirements while also aiding risk assessors in developing risk assessments that inform regulatory decisions.

Measure: Usefulness of ORD's risk management research products for enabling EPA's Office of Water, Regions, water utilities, and other key stakeholders to manage public health risks associated with exposure to drinking water, implement effective safeguards on the quality and availability of surface and underground sources of drinking water, improve the water infrastructure, and establish health-based measures of program effectiveness.

Explanation:This measure captures the assessment by an independent expert review panel of the relevance, quality, and use of the program's research under Long-Term Goal 2. Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews. Targets for this measure are set using the previous BOSC rating??and BOSC recommendations??as a guide. The program outlines an action plan in response to BOSC recommendations; completion of the actions in this plan demonstrates progress from the baseline. The BOSC's 2007 report can be found at http://www.epa.gov/OSP/bosc/pdf/dwmc0708rpt.pdf. The program's formal action plan can be found at http://www.epa.gov/OSP/bosc/pdf/dwmc0801resp.pdf. Work under DWRP's LTG 2 supports the Office of Water (OW) in rule implementation, simultaneous compliance, and evaluating the effectiveness of risk management decisions. ORD's work under this goal also supports OW, regions, states, utilities, and key stakeholders in protecting sources of drinking water, managing water availability, improving water infrastructure sustainability, increasing water and energy use efficiency, and responding to short and long-term water resource impacts of environmental stressors such as climate change, population growth and land use changes.

Measure: Percentage of research products used by the Office of Water as the basis of or in support of Six Year Review Decisions

Explanation:"Products" can include 1. EPA Peer-reviewed reports funded by the Drinking Water Research Program (DWRP), 2. DWRP funded research articles published in peer-reviewed scientific journals, or 3. Citable tools delivered to the Office of Water including monitoring methods, models, treatment guidance, and databases funded by the DWRP. "Use" can include citations in OW published decisions or documents that support decisions including risk assessments, rule-making support documents, proposed rules, or final rules.

Measure: Percentage of research products used by the Office of Water as the basis of or in support of Contaminant Candidate List Decisions

Explanation:"Products" can include 1. EPA Peer-reviewed reports funded by the Drinking Water Research Program (DWRP), 2. DWRP funded research articles published in peer-reviewed scientific journals, or 3. Citable tools delivered to the Office of Water including monitoring methods, models, treatment guidance, and databases funded by the DWRP. "Use" can include citations in OW published decisions or documents that support decisions including risk assessments, rule-making support documents, proposed rules, or final rules.

Measure: Percentage of planned methodologies, data, and tools delivered in support of EPA's Office of Water and other key stakeholders needs for developing health risk assessments, producing regulatory decisions, implementing new and revised rules, and achieving simultaneous compliance under the Safe Drinking Water Act.

Explanation:At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs (detailed in the program's Multi-Year Plan). The program strives to complete 100% of its planned outputs each year so that it can best meet EPA and other partners' needs. To ensure the ambitiousness of its annual output measures, ORD has better formalized the process for developing and modifying program outputs, including requiring that ORD programs engage partners when making modifications. Involving partners in this process helps to ensure the ambitiousness of outputs on the basis of partner utility. In addition, EPA's Board of Scientific Counselors (BOSC) periodically reviews programs' goals and outputs and determines whether they are appropriate and ambitious. The outputs tracked by this measure demonstrate progress towards completing DWRP's long term goal 1, which primarily supports the Office of Water in decisions relating to: Unregulated Contaminant Monitoring Rule (UCMR), regulating/not regulating contaminants on the Contaminant Candidate List (CCL), the six year review, and the Underground Injection Control (UIC) program. ORD's work under this goal also supports regions and key stakeholders in meeting simultaneous compliance requirements while also aiding risk assessors in developing risk assessments that inform regulatory decisions.

Measure: Percentage of planned risk management research products delivered to support EPA's Office of Water, Regions, water utilities, and other key stakeholders to manage public health risks associated with exposure to drinking water, implement effective safeguards on the quality and availability of surface and underground sources of drinking water, improve the water infrastructure, and establish health-based measures of program effectiveness.

Explanation:At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs (detailed in the program's Multi-Year Plan). The program strives to complete 100% of its planned outputs each year so that it can best meet EPA and other partners' needs. To ensure the ambitiousness of its annual output measures, ORD has better formalized the process for developing and modifying program outputs, including requiring that ORD programs engage partners when making modifications. Involving partners in this process helps to ensure the ambitiousness of outputs on the basis of partner utility. In addition, EPA's Board of Scientific Counselors (BOSC) periodically reviews programs' goals and outputs and determines whether they are appropriate and ambitious. The outputs tracked by this measure demonstrate progress towards completing DWRP's long term goal 1, which supports the Office of Water (OW) in rule implementation, simultaneous compliance, and evaluating the effectiveness of risk management decisions. ORD's work under this goal also supports OW, regions, states, utilities, and key stakeholders in protecting sources of drinking water, managing water availability, improving water infrastructure sustainability, increasing water and energy use efficiency, and responding to short and long-term water resource impacts of environmental stressors such as climate change, population growth and land use changes.

Measure: Percent variance from planned cost and schedule (New measure, added August 2007)

Explanation:This measure captures the ability of the program to increase cost effectiveness based on the extent to which it delivers annual research outputs relative to the amount of funds spent. Using an approach similar to Earned Value Management, the data are calculated by: 1) determining the difference between planned and actual performance and cost for each Long-Term Goal, 2) adding these data together to generate program totals, and 3) dividing the Earned Value of all work completed by the Actual Cost of all program activities."

Is the program purpose clear?

Explanation: The purpose of the Drinking Water Research Program (DWRP) is to provide timely, leading-edge research products to support sound scientific decisions by the Office of Water. The Safe Drinking Water Act (SDWA) authorizes EPA to conduct research to support the regulatory decisions required by SDWA including specific provisions for research. The DWRP's secondary purpose is to provide research products to state and local water authorities and to the drinking water research community.

Evidence: Research Provisions of the Safe Drinking Water Act (42 U.S.C. Section 300j-1 and 300j-18); Drinking Water Research Program Design Model; Drinking Water Research Program Multi-Year Plan (2003)

Does the program address a specific and existing problem, interest, or need?

Explanation: The Drinking Water Research Program addresses the public need for continuously improved information about the quality of drinking water both related to reassessment of current regulations and investigation into new potential contaminants. The National Academy of Sciences (NAS) and other independent organizations continue to express the need for drinking water research.

Evidence: Research Provisions of the Safe Drinking Water Act (42 U.S.C. Section 300j-1 and 300j-18); NAS reports including - Public Water Supply Distribution Systems: Assessing and Reducing Risks-First Report (2005), Indicators for Waterborne Pathogens (2004), Arsenic in Drinking Water (2001 Update, 1999); International Life Sciences Institute reports characterizing disinfection by-products (1993, 1994, 1995, 1999)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Drinking Water Research Program (DWRP) actively coordinates research agendas with other drinking water research organizations. Within EPA, the DWRP coordinates research with the Office of Water's Technical Support Center (TSC). The two programs collaborate on development of analytical methods and provision of technical assistance. Additionally, members from both the TSC and the DWRP serve on the Research Coordination Team which plans and implements DWRP research. The other major drinking water research organizations in the U.S. are the American Water Works Association Research Foundation (AwwaRF) and the Water Environment Research Foundation (WERF) whose missions are to serve the research needs of their membership, the water utilities. The DWRP coordinates with AwwaRF and WERF by participating on their respective research planning committees. Finally, the DWRP coordinates with international drinking water research programs through its membership in the Global Water Research Coalition.

Evidence: Research Coordination Summary with Office of Water Technical Support Center and External Organization; AwwaRF - Research Chairs Meeting Summary (October 20-21, 2004); Biological Destruction of Perchlorate and Nitrate in Ion Exchange Concentrate (RFP 3137); 2005-2006 WERF Long-Range Plan and Annual Report; Global Water Research Coalition website: www.globalwaterresearchcoalition.net/aboutgwrc.htm); DWRP Research Coordination Team Roster (July, 2005); DWRP/TSC Coordination meetings/workshops (e.g. NERL Research Planning Meeting with OGWDW/SRMD (October 15, 2003), Virulence Factor Activity Relationships Workshop Draft Summary Report (October 28-29, 2004), Feasibility of Using DNA-RNA Microarrays and Related Technologies for High Through-put Detection of Waterborne Pathogens - DWRP/TSC research status and planning workshop); DWRP/TSC co-developed drinking water methods and guidance (e.g. Method 332.0, Method 556, Quality Assurance/Quality Control Guidance for Laboratories Performing PCR Analyses on Environmental Samples).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Drinking Water Research Program (DWRP) design is free of major flaws. The program has an unambiguous, focused design, as evidenced by the Program Design Model developed jointly with the Office of Water. The DWRP targets resources toward research activities that create products for OW and other key clients, thereby influencing OW outcomes. The DWRP has established Long Term Goals that are equivalent to client-oriented short-term outcomes. OW uses DWRP products to drive intermediate and environmental outcomes leading to the long-term outcome of protecting public health. An independent, expert review panel, EPA's Board of Scientific Counselors, conducted a review of the DWRP in the third quarter of FY05 and was charged with reviewing the DWRP program design. Preliminary findings indicate no major design flaws.

Evidence: Drinking Water Research Program Design Model; Board of Scientific Counselors Drinking Water Subcommittee Outbriefing - Preliminary Findings

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The Drinking Water Research Program (DWRP) effectively allocates resources to intramural and extramural beneficiaries to fund specific activities as established in the DWRP Multi-Year Plan. Ultimately, the Office of Water (OW) uses products produced by beneficiaries of DWRP resources to help it implement the Safe Drinking Water Act. The Office of Research and Development collaborates with OW when developing the DWRP Multi-Year Plan which helps ensure OW will benefit from ORD research products.

Evidence: Research Provisions of the Safe Drinking Water Act (42 U.S.C. Section 300j-1 and 300j-18); Drinking Water Research Program Design Model; Drinking Water Research Program Multi-Year Plan (2003)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Under the recently reorganized, outcome-focused long term goal structure, the Drinking Water Research Program will strive to increase the utility of its new scientific data and leading-edge approaches for the Office of Water (OW) in their (1) Six-Year Review decisions (regulated contaminants) and (2) Contaminant Candidate List (CCL) decisions (unregulated contaminants). Planned revisions (Spring 2006) to the Multi-year plan will reflect this realignment. The DWRP will measure its progress in meeting the two long term goals through independent expert review. The reviewers will rate the program on a scale of 1 to 5 based on a series of nine survey questions. Each number in the rating scale has a specific definition related to the utility of the program's research products and the strength of the evidence provided by ORD. Three of the questions address the use of ORD products; the remaining six address the utility (usefulness for an intended purpose) of those products. After determining an overall program rating, the panel will provide a brief narrative summary of the score, which will allow the Agency and others reviewing the program a detailed comparison of evaluations through time.

Evidence: EPA ORD Drinking Water Research Program Long Term Goals and Measures -- summary of the measures and evaluation methodology.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The long term measures were finalized in late 2005. At that time, the program proposed targets and timeframes; however, there was not sufficient time to review these targets and develop a sufficient justification. Consequently, the 2005 PART does not include targets and timeframes for the long-term measures.

Evidence:

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The DWRP has developed two annual performance measures and is developing a third annual outcome measure. Together, these measures will adequately monitor annual progress toward meeting the Drinking Water Research Program's (DWRP) two Long Term Goals. There will be two of each annual performance measure to correspond to the two Long Term goals of the program (6 total annual performance measures). The first is an output measure that tracks progress towards completing key research products which are used by OW to make regulatory decisions. The DWRP coordinates with OW to develop a Multi-Year Plan (MYP) which sets a schedule for key products needed to support OW regulatory decisions. This measure will assess the DWRP's adherence to the MYP. The second measure, an intermediate outcome measure, will track the percentage of DWRP products used by OW as the basis of or in support of regulatory decisions. The measure will be based on 5 years of data. The final measure, an outcome measure, is currently under development. As proposed, it will track OW use of DWRP products through a survey instrument. This measure will go beyond the intermediate outcome measure to gather more detailed information about the utility of DWRP outputs to the program's major client, OW. All new measures will be included in GPRA documents - FY2007 Annual Plan (draft in OMB submission and final in Congressional Justification) and FY2006 annual report.

Evidence: Drinking Water Research Program Annual Performance Goals; Drinking Water Research Program Multi-Year Plan, pages T-1.1 - T3.4; July 22, 2005 Memorandum from Gregory Sayles, Acting Program Director for Drinking Water Research RE: Client Survey for Drinking Water Research Programs - commits DWRP to developing the client survey measure with OMB consultation.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and targets have been developed for the intermediate outcome annual measure which assesses the program's progress in delivering research outputs supporting regulatory decisions. The program has not yet provided evidence of how these targets are ambitious. Additionally, the program is working to develop baselines and ambitious targets for the output measure assessing the percentage of products used as the basis of or in support of Office of Water regulatory decisions.

Evidence: Drinking Water Research Program Multi-Year Plan, pages T-1.1 - T3.4.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: There is evidence that grantees support the overall goals of the program and that grantees measure and report on their performance as it relates to accomplishing the program's goals. However, the evidence does not indicate how other program partners such as contractors or signatories to cooperative agreements support the program goals or measure and report on these goals. The Drinking Water Research Program (DWRP) works with the Office of Water (OW) to develop a Multi-Year Plan (MYP) for research. The MYP is laid out based on the regulatory needs of OW and corresponds with the DWRP's long term goals of increasing use of products by OW in regulatory decisions. The MYP represents all research funded by the DWRP, including extramural funding to grant recipients. This is seen primarily in the list of expected products and their schedule for completion. All grantees are made aware of the MYP expectations- intramurally through ORD management and the DW National Program Director and extramurally through the STAR Grant Project Officers. STAR Grant Requests for Applications are developed based on the MYP needs and therefore are linked to the long term goals of the program. Grantees submit annual progress reports, which Grant Project Officers review to determine how well the grantees are performing.

Evidence: DWRP MYP PAGES T1.1-T3.4 (NCER goals are for STAR grantees); EXAMPLE STAR RFA: Development and Evaluation of Innovative Approaches for the Quantitative Assessment of Pathogens in Drinking Water (FY2005); EPA Order 5700.6 A1: Grant's Administration Division Policy on Compliance, Review, and Monitoring

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Program wide evaluation of the Drinking Water Research Program (DWRP) by independent, expert panels began in the third quarter of fiscal year 2005 and will recur every 4 to 5 years. The reviews will be conducted by EPA's Board of Scientific Counselors who will evaluate the DWRP's relevance, design, planning, performance, product quality and utility to clients, and scientific leadership. In order to maintain independence, the DWRP requires panelists to submit financial information to minimize and prevent conflicts of interest. In the past, ORD has used this information to ask panelists to recuse themselves.

Evidence: Charge to the BOSC Drinking Water Subcommittee; BOSC Drinking Water Subcommittee Outbriefing - Preliminary Results; BOSC Fact Sheet; EPA Confidential Financial Disclosure Form for Special Government Employees Serving on Federal Advisory Committees at the U.S. Environmental Protection Agency; Executive Branch Confidential Financial Disclosure Report; Commitment to perform regular evaluations - Memorandum RE: Independent Program Evaluation for the Drinking Water Research Program to be Performed Regularly (signed 7/22/05).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: There is no thorough evidence that the Drinking Water Research Program (DWRP) actively links its budgetary resources with annual performance targets. Limited evidence indicates that the DWRP can show allocation of direct and indirect resources between the two long term goals; however, these budgetary resources are not adequately linking to performance targets to show the performance implications of funding decisions. Finally, the annual OMB submission and Congressional Justification both present budget and performance information; however, the presentation of these does not make clear the impact of funding, policy or legislative decisions on expected performance.

Evidence: FY2005 Congressional Justification.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Drinking Water Research Program (DWRP) has taken meaningful steps to correct its strategic planning deficiencies. The DWRP has recently developed outcome-oriented Long Term Goals, and is developing measurable output-oriented and outcome-oriented Annual Goals. The DWRP has also established recurring program-level independent, expert reviews (first review was held in the third quarter of FY2005) that will determine relevancy of the program, appropriateness of the design and planning of the program, progress toward achieving Long Terms Goals, and quality of the research products. The first review also assessed the adequacy and utility of the revised Long Term and Annual Goals.

Evidence: DWRP Long Term Goals; DWRP Annual Performance Goals; Charge to the Board of Scientific Counselors (BOSC) Drinking Water Subcommittee; BOSC DW Subcommittee Outbriefing - Preliminary Findings.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Drinking Water Research Program (DWRP) has not conducted any comparisons of the potential benefits of its approaches with alternatives either within or outside the program.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Drinking Water Research Program (DWRP) employs a prioritization process on several time scales including yearly prioritization of current and proposed work, and biannual updating of the Multi-Year Plan. The DWRP uses these priorities in constructing its budget request, through the Office of Research and Development (ORD) and EPA. ORD and the DW National Program Director also use this prioritization process to implement annual appropriations from Congress. For example, in Fiscal Year 2005, Congress appropriated $1.6 million less than requested. The DWRP used their prioritized list of projects to decide which work would not be funded for that fiscal year.

Evidence: October 14, 2003 Memo from Mike Shapiro, Deputy Assistant Administrator for the Office of Water RE: OW Alternative Goal 2 FY05 and FY06 Contingency Pool; DWRP Research Coordination Team; ORD FY 2005-2006 Contingency Plan Development Process; DWRP Multi-Year Plan, page 11; Impact Summary of FY 2005 Appropriation.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Evidence indicates that the Drinking Water Research Program (DWRP) regularly collects high-quality performance data; however, evidence for the use of this data to adjust program priorities, allocate resources, or take other appropriate management actions is lacking. The DWRP reports performance progress quarterly. The DWRP grantees are required to report annual progress and final results, including significant accomplishments which are posted on a public website. Annual goals and measures are tracked in ORD's Integrated Resource Management System (IRMS). Associated data in IRMS includes baseline information, Strategic Goal, Objective, and Sub-objective linkages, Multi-Year Plan (MYP), Long-term Goal, and responsible organization for each performance measure. ORD's Deputy Assistant Administrator for Management is provided with quarterly updates on the status of ORD goals and measures. This information is used to inform the annual planning process as well as to update the MYP. Planning summaries and other information for each program are provided by the National Program Directors (NPDs) to ORD's Executive Council (EC) to inform the EC's discussions of planning and budgeting for the year. These materials include a contingency pool representing a significant portion of ORD's research program with a statement for each area in the contingency pool explaining performance impacts if the reduction was implemented. These materials continue to be used throughout the budget and operating plan development processes.

Evidence: Integrated Resource Management System Annual Performance Goals and Measures Quarterly Report: FY 2004 Drinking Water Research Program; Science to Achieve Results (STAR) Grants Program (http://es.epa.gov/ncer/grants); Grant Terms and Conditions (http://es.epa.gov/ncer/guidance/tscs99.html ); Grant Progress Reports(http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/research.search/type/3

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Drinking Water Research Program (DWRP) incorporates program performance into personnel performance evaluation criteria. Senior managers are accountable for specific performance standards relating to program goals, including progress toward achieving the targets and timeframes described in the multi-year plans. The research program also monitors progress against GPRA targets. Some research is conducted extramurally. Contracts and grants, statements of work, deliverables, costs, and schedules are written into award terms. ORD project officers and Work Assignment Managers are responsible for seeing that agreements are awarded and managed according to government regulations to give value to the government and the public.

Evidence: Example Performance Agreements of DWRP managers showing quantifiable performance standards which hold managers accountable for cost, schedule and performance results; Grant Progress Reports (http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/research.search/type/ 3); EPA Project Officer Online Recertification Training.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The Drinking Water Research Program (DWRP) has consistently obligated its two year funds in a timely manner. Each year, the DWRP develops an Operating Plan with the proposed budget and updates it upon enactment of a Congressional appropriation. The Operating Plan tracks the DWRP funds explicitly. EPA also works with grantees to ensure that their work plans reflect the Agency's Strategic Plan and Operating Plan and that recipient spending is consistent with the approved workplan. Each program office and grants management office conducts post-award monitoring of assistance agreements, including monitoring the draw-down of funds against grantee progress on workplan tasks and deliverables. All grantees are required to submit annual or more frequent financial status reports.

Evidence: DWRP Integrated Resource Management System (IRMS) Budget Execution and Operating Plan Data for FY 2001 to FY 2004: End of Year obligations during this period were at least 92% of available resources. DWRP IRMS Budget Execution and Operating Plan Data for FY2005: As of June 30, 2005, the DWRP had obligated 87% of its two-year program resources; EPA Order 5700.6 A1: Grant's Administration Division Policy on Compliance, Review, and Monitoring; National Center for Environmental Research Calendar Year 2005 Post-Award Monitoring Plan for Assistance Agreements.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Drinking Water Research Program (DWRP) and OMB have not agreed on an adequate efficiency measure though the program continues work on developing one.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Drinking Water Research Program (DWRP) actively coordinates research agendas with other drinking water research organizations. Within EPA, the DWRP coordinates research with the Office of Water's Technical Support Center (TSC). The two programs collaborate on development of analytical methods and provision of technical assistance. Additionally, members from both the TSC and the DWRP serve on the Research Coordination Team which plans and implements DWRP research. The other major drinking water research organizations in the U.S. are the American Water Works Association Research Foundation (AwwaRF) and the Water Environment Research Foundation (WERF) whose missions are to serve the research needs of their membership, the water utilities. The DWRP coordinates with AwwaRF and WERF by participating on their respective research planning committees. Finally, the DWRP coordinates with international drinking water research programs through its membership in the Global Water Research Coalition.

Evidence: Research Coordination Summary with Office of Water Technical Support Center and External Organization; AwwaRF - Research Chairs Meeting Summary (October 20-21, 2004); Biological Destruction of Perchlorate and Nitrate in Ion Exchange Concentrate (RFP 3137); 2005-2006 WERF Long-Range Plan and Annual Report; Global Water Research Coalition website: www.globalwaterresearchcoalition.net/aboutgwrc.htm); DWRP Research Coordination Team Roster (July, 2005); DWRP/TSC Coordination meetings/workshops (e.g. NERL Research Planning Meeting with OGWDW/SRMD (October 15, 2003), Virulence Factor Activity Relationships Workshop Draft Summary Report (October 28-29, 2004), Feasibility of Using DNA-RNA Microarrays and Related Technologies for High Through-put Detection of Waterborne Pathogens - DWRP/TSC research status and planning workshop); DWRP/TSC co-developed drinking water methods and guidance (e.g. Method 332.0, Method 556, Quality Assurance/Quality Control Guidance for Laboratories Performing PCR Analyses on Environmental Samples).

Does the program use strong financial management practices?

Explanation: The Drinking Water Research Program (DWRP) has no material weaknesses as reported by the IG and has procedures in place to minimize erroneous payments. The DWRP follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. The existing system of controls and accountability are based on GAO and other principles to ensure that improper payments are not made. ORD organizations receive resource allocations at the program level (Goal, Objective, Program/Project). In turn, ORD organizations develop implementation plans to execute their allocated resources. Implementation plans detail the planned execution of allocated resources at the goal, objective, program/project and task level of detail. ORD has trained Funds Control Officers (FCOs) who are responsible for certifying the availability of funds as to the correct purpose, time and amount.

Evidence: EPA Financial Management Five-Year Plan 2005-2009 (www.epa.gov/ocfo/finstatement/5yearplan2005-2009.pdf); FY 2006 Congressional Justification (www.epa.gov/ocfo/budget/index.htm); IG's Report on EPA's FY 2003 and FY 2004 Financial Statements (www.epa.gov/ocfopage/finstatement/2004ar/aro4_section3_3.pdf, pages 208 -209) EPA, including the DWRP, did not have any material weaknesses.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The most compelling management deficiency in the Drinking Water Research Program has been the lack of a program director with the sufficient authority and responsibility to manage the budget, establish a research agenda including measures of accountability, and to conduct evaluations of program management and any necessary remediation. In 2005, EPA corrected this situation by appointing National Program Directors (NPDs) to most of the ORD Research Programs, including Drinking Water (to be filled permanently soon, currently filled as a detail), with primary responsibility for managing the program. Annual planning requires the Water Research Coordination Team (RCT) to compare planned deliverables for Office of Water to OW's current priorities. The RCT responds with reprioritization of planned and current research. Finally, in FY 2004, no Agency- or program-level material weakness was identified for the Drinking Water Research Program by the Federal Managers Financial Integrity Act (FMFIA) annual review process.

Evidence: EPA Management Integrity and Federal Manager's Financial Integrity Act (FMFIA) (www.epa.gov/ocfo/integrity/integrity.htm); Research Coordination Team; NPD Press Release; IG's Report on EPA's FY 2003 and FY 2004 Financial Statements (www.epa.gov/ocfopage/finstatement/2004ar/aro4_section3_3.pdf, pages 208 -209) EPA, including the DWRP, did not have any material weaknesses.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Evidence indicating that grants are awarded based on a clear competitive process is strong. All of the DW research grants are awarded through ORD's competitive STAR extramural grants program using external scientific peer reviewers to rate applications. Pure competition among the applicants is conducted where awards are given based on ranking for scientific merit and program relevance. Renewals are rarely granted and governed by a revised competition order. EPA's process for soliciting and selecting proposals is clearly articulated on its website. Over 90% of the DW program grants have been awarded to researchers who have not previously received individual STAR grants. In the future, to receive a YES to this question, the program will have to show how it competes awards to other non-grant program partners.

Evidence: To attract new investigators, research solicitations are announced in the Federal Register, posted on the National Center for Environmental Research (NCER) website for at least 90 days, emailed to institutions and individuals that have indicated an interest in receiving them, distributed at scientific conferences, and disseminated to researchers at other federal agencies; NCER Web site: http://es.epa.gov/ncer/; EPA National Center for Environmental Research website: RFA announcements (FY 2005 Example - http://es.epa.gov/ncer/rfa/2005/2005_pathogens_drinking_water.html); National Academy of Sciences review, "The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program," pp. 30, 34, 152.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Evidence of strong oversight practices of grantees is apparent. Grant project officers monitor grantee performance, including submission of annual progress reports and compliance with federal requirements. Cost reviews lie at the heart of the DWRP fiduciary responsibilities. To satisfy the requirements contained in OMB Circulars A-122 (Cost Principles for Nonprofit Organizations), A-87 (Cost Principles for State, Local, and Indian Tribal Governments) and A-21 (Cost Principles for Educational Institutions) as articulated by EPA's Cost Review Guidance (GPI-00-05), EPA has developed a cost review template for Project Officers (PO) to use to document their cost reviews. The template is currently undergoing internal review and will become a requirement in the first quarter of FY 2006. Also, grantees provide a list of publications, presentations, and other activities on an annual basis and at the end of their grant period. Project officers also conduct formal site visits to monitor the progress of active grants. Finally, the STAR program sponsors workshops to bring together the DW grantees with ORD scientists and representatives from the Office of Water to inform one another of scientific results and upcoming policy considerations. Prior to award Project Officers closely review the budgets and often request additional explanations or modifications. Then, in addition to the scientific progress reports, Project Officers get an annual report on the budget from grantees and POs have access to the EPA funding system which permits them to review the current and entire expense records on any EPA grant at any time. In the future, to receive a YES to this question, the program will have to demonstrate its oversight practices of non-grant program partners.

Evidence: Project officer site visits conducted on a minimum of 10% of active grantees; EPA conducted a total of 7 site visits in FY2004, out of 28 active drinking water program grants; FY2004 Site Visit Schedule; NCER website for annual reports, publications, activities, and Workshop Proceedings (www.epa.gov/ncer); Project Officer Grant Review Template.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Evidence of collection of grantee performance data is strong. An annual progress report is submitted by each grantee and posted on the EPA National Center for Environmental Research website. Reports are distributed to EPA staff to disseminate to interested parties. These reports include summaries of progress in relation to project objectives as well as publications of research results. Grantees also present results at the multitude of DW-related national and international scientific conferences held annually. Almost 200 publications resulting from the 62 DW grants awarded since the program began in 1996 are further evidence of grantee performance. The DW grants have been used to train at least 36 undergraduate students and contributed to the support of over 100 graduate students, 45 post-docs and 2 visiting scientists. Results from grants have also been used to achieve additional support from other federal agencies and organizations. In the future, to receive a YES to this question, the program will have to demonstrate evidence of how it collects performance data of non-grant program partners.

Evidence: EPA National Center for Environmental Research website: progress reports and publications lists (http://es.epa.gov/ncer/); National Academy of Sciences review, "The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program," Summary

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The Drinking Water Research Program (DWRP) funds internal research and an external, competitive (STAR) grants program. The internal program funds are not allocated via a competitive grants approach. Rather, funds are distributed to high priority areas, identified by the program's primary client, the Office of Water, through the Research Coordination Team composed of representatives from OW and ORD. Internal non-competitive funding is justified because (1) ORD can address complex, multi-disciplinary DW issues inside one organization by integrating the work of its highly-skilled life scientists, physical scientists and engineers, and (2) unique research facilities are available in ORD. EPA initiated formal independent external expert reviews of the quality, relevance, and performance of the DWRP in the 3rd quarter of FY2005.

Evidence: Drinking Water Research Program Peer Review and Quality Assurance; ORD FY 2005-2006 Contingency Plan Development Process; Drinking Water Research Program External Peer-Review of Research Program Components; Unique facilities include the drinking water pilot plant and distribution system test loops in Cincinnati, OH and the Human Studies facility in Chapel Hill, NC; FY06 PresBud: $40.1M Non-STAR, $4.6M STAR; FY05 Enacted: $44.2M Non-STAR, $4.5M STAR

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Drinking Water Research Program has two approved long term performance measures and is working with the Board of Scientific Counselors to develop a baseline measure.

Evidence:

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Drinking Water Research Program has results for the output measure. These results indicate the program is not achieving its targets. Performance on the Contaminant Candidate List measure is diminishing and performance on the Six Year Review measure diminished from 2003 to 2004 but showed improvement in 2005. This is the first year the program has measured its progress in achieving multi-year plan targets, which must enter into the determination of achievement of annual performance goals. Given this, the program was preliminarily given a "small extent." However, if progress is not shown on these and the other annual measures in 2006, the answer to this question will change to a "no."

Evidence:

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Drinking Water Research Program and OMB have not agreed upon an adequate efficiency measure.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Drinking Water Research Program (DWRP) undertakes research that its primary client, the Office of Water, requests to inform implementation of the Safe Drinking Water Act. The other major drinking water research organizations in the U.S. are the American Water Works Association Research Foundation (AwwaRF) and the Water Environment Research Foundation (WERF) whose missions are to serve the research needs of their membership, the water utilities. In March 2005, EPA completed a bibiliometric analysis of DWRP research products using Thomson's Essential Science Indicators (ESI) and Journal Citation Reports (JCR) as benchmarks. The analysis found that "drinking water papers are more highly cited than the average paper" and that more than one-quarter of the drinking water publications are highly cited papers" when using ESI criteria. This analysis lends support that the DWRP performs favorably in the drinking water research community. However, a direct comparison of the performance of the DWRP with AwwaRF and/or WERF has not been completed.

Evidence: Research Provisions of the Safe Drinking Water Act; Bibliometric Analysis for Papers on Topics Related to Drinking Water, March 2005.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In June, 2005 the Drinking Water Subcommittee of EPA's Board of Scientific Counselors (BOSC) performed an independent, comprehensive review of the Drinking Water Research Program (DWRP). The BOSC evaluated the relevance, design, planning, product quality and utility to clients, and scientific leadership of DWRP research. The BOSC findings were critical but generally positive. Two of the key findings are that the program is relevant and critically important to EPA's mission and that research outputs lead to outcomes with respect to the Office of Water and other key clients. The BOSC did note that the program would benefit from making the process of transferring outputs to outcomes more transparent.

Evidence: Board of Scientific Counselors Drinking Water Subcommittee Outbriefing - Preliminary Findings; Review of the Office of Research and Development's Drinking Water Research Program at the U.S. Environmental Protection Agency - Final Report.