Detailed Information on the
Bureau of Reclamation - Science and Technology Program Assessment

Submit legislation establishing overarching authority to enter into cooperative agreements for Research & Development. Such authority would facilitate easier, stronger collaborative efforts with researchers at other institutions, and further increase competition for research funding.

Work with the Western Area Power Administration and the Bonneville Power administration to develop and submit any necessary legislation and enter into a Memorandum of Understanding with Reclamation to help fund, on a direct financing basis, hydropower research that directly benefits power customers.

Establish baselines for those performance measures which do not already have them, and collect performance data for those that already have baselines.

Examine whether or to what extent the Federal government should carry out water and hydropower-related research.

Measure: R&D products will generate a 10 to 1 return on Reclamations R&D investments in terms of the value of the water and power benefits derived when they are deployed.

Explanation:Water liberation can be achieved through demonstrations and deployments of R&D solutions that expand useable water supplies, conserve existing water supplies, or avoid water conflicts. The annual goal and measure targets a 10:1 annual return on the R&D investment in order to track progress, gauge effectiveness, and make program management changes as necessary to achieve the long-term goal. However, annual fluctuations above and below the 10:1 rate of return is anticipated. The six-year performance period starts in 2005 and ends in 2010. By the end of FY 2010, the accumulated amount of annual benefits over the six-year period will be able to project a continuing benefit to cost ratio of at least 10:1 into the future. The annual R&D investment is equivalent to the annual R&D budget request made by the Research and Development Office. The actual investment level will be adjusted to reflect the portion of the actual budget enacted where Congress does not direct the investment criteria. (FY 2005 requested budget $10,484,000, FY 2006 requested budget $9,709,000, future years assumed level with 2006 for target calculation purposes). Note: Improved goal and measures text was entered on June 11, 2005. The goals and measures developed through the 2003 PART process remain intact but the goal and measure text has been simplified, as described below, to be more intuitive. In addition, the method of projecting and reporting the outcome performance measure of producing a 10:1 return on the R&D investment has been modified to be more direct and simple. The improved approach also provides a much more obvious and direct correlation of the goal and measure to annual budgets. Originally, a hypothetical quantity projection of the combination of water liberated, cost reduced, or hydropower generated from R&D that would produce a 10:1 economic return was set as the goal. At the goal setting stage, projecting the mix of the 10:1 return between water liberated, cost reduced or hydropower is too presumptuous, and leads to the creation of a segregation of the goal into three different annual and three different long-term outcome goals. The improved approach results in one annual-outcome-goal, and one long-term outcome goal. The annual goal directly supports the long-term goal which is to achieve a 10:1 economic return. Goal accomplishment reporting will capture and report how each category of economic return (i.e. water, costs savings, and hydropower) contributed to the goal accomplishment.

Measure: The ratio of total program cost-share to total program budget will increase a minimum of 5% each year.

Explanation:The value of our R&D to others, and maximizing the Reclamation R&D capability per program dollar invested, will be demonstrated by increasing our R&D collaboration with those that have a vested interest in finding new solutions to Western Water and Power challenges. Increased collaboration will be measured by achieving a 34% percent increase in the ratio of partner R&D cost-share, per Reclamation R&D dollar, over a six-year performance period that begins in 2005 and ends in 2010.

Measure: Increase the number of Western Water and Power solution bulletins to R&D customers by 34% over a six-year period.

Explanation:We will more effectively and broadly share Water and Power solutions with those that can put them to work - Western water and power managers. Increased communications will be measured by achieving a 34% percent increase in the production of Water and Power Solution Bulletins over the six-year performance period that starts in 2005 and ends in 2010. Water and Power Solution Bulletins will be posted on a searchable web-enabled data base that will target the R&D customer through active outreach.

Is the program purpose clear?

Explanation: Reclamation's (BOR's) research program has a clear mission statement: facilitating the development and use of new scientific and technological solutions that contribute significantly to a safe, affordable, sustainable, and ample water and power supply. The purpose of the desalination research established by Congress is to develop more cost-effective, technologically efficient, and implementable means to desalinate water.

Evidence: Tab 1.1-1: Web page postings of Science &Technology (S&T) Program Goal Statement show a clear, focused program mission stated in two separate places at www.usbr.gov/research.1.1-2: Program PowerPoint slide that shows program mission goal that has been presented at recent National Water Resources Association (NWRA), Western Coalition of Arid States (WESCAS), and other conferences. 1.1-3: Replies from NWRA members showing strong support for S&T Program Goal.1.1-4: P.L. 104-298 (Water Desalination Act of 1996) states the purpose of the legislation.1.1-5: The home page of Reclamation's Desal website shows the purpose of the program on the first page. The website is located at <www.usbr.gov/water/desal.html>.The Reclamation Act of 1902, and Public Law 92-149 provides authority for research focused on BOR mission needs.

Does the program address a specific and existing problem, interest, or need?

Explanation: Growing shortages of water and escalating water conflicts pose many challenges for Reclamation water managers and water users. Research is customized to find innovative solutions focused on BOR core end outcomes of water and power deliveries and on the specific needs of our stakeholders (i.e., Reclamation resource managers and the external water managers and water users served by Reclamation project waters). The program is designed to ensure research is targeted to those problems, interests, and needs. The program has a multi-objective steering team that includes internal and external stakeholders, and serves to steer, validate, and prioritize program needs and direction. The desalination research also serves national water supply needs.

Evidence: Water supplies in the U.S. are becoming progressively more scarce. 1.2-1: US Census and USGeological Survey (USGS) data show the Western US has an exploding population base, and a fixed amount of fresh water. 1.2-2: WATER2025 initiative describes this problem. 1.2-3: BOR's prime focus is in West. 1.2-4: Program addresses unique problems and solutions, pg. 1. 1.2-5: National Research Council (NRC) (2001) reports water problems & research needs. 1.2-6: The S&T Roadmap guides research proposals and program decisions to focus on four main research areas. 1.2-7: Nested in the 4 research focus areas are 20 mission-specific R&D output areas, each with specific goals and objectives. 1.2-8: Sect. I.F, I.G, and VI of the proposal form shows that research is focused on problems specific to BOR challenges, needs, and responsibilities. 1.2-9: Letter from WESCAS. 1.2-10: Desalination R&D brochure; 1.2-11, 'Desalination and Water Purification Technology Roadmap', pp 8-28; 1.2-12: the Multi-State Salinity Coalition article; 1.2-13 & 14, two articles; and 1.2-15: Golden Workshop Report.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: No other Research & Development (R&D) programs are dedicated to developing solutions for Reclamation water managers/users and their water needs. The program was designed to focus on these needs (see response to Question 1.2 and 4.4). It uses multiple mechanisms to avoid duplication and catalyze collaboration and coordination with others involved in water resources research. Subsidized agricultural water market prices do not provide sufficient private sector incentive for the R&D necessary. Private sector investments in desal. focus on incremental change of specific products; profit margins are too thin to support the high risk and long development times for basic desal. research. The mission of desalination cost reduction is unique, but other agencies work with the same technologies as they relate to their military mission requirements of reliability and portability. To avoid redundant research and facilitate information sharing, the program manages an interagency consortium on desalination and has a Memorandum of Understanding (MOU) with other research funding non-profits.

Evidence: 1.3-1: A federal research coordinating committee ensures no duplication of research activities and encourage collaboration. 1.3-2: Established a multi-agency and stakeholder steering team that helps avoid duplication, encourage collaboration, and identifies R&D entities having ability to compliment or support BOR efforts. 1.3-3: Notes summarize the unique niches served by BOR R&D as applied and focused on solutions to BOR problems, while the USGS focus is on more basic R&D plus water data collection and dissemination. 1.3-4: Collaboration planning document. 1.3-5: Section V of proposal form and proposal ranking factors avoids duplication by encouraging coordination with partners having similar objectives through resource leveraging. 1.3-6: A memo addressing collaboration with USGS and the National Institutes for Water Resources (NIWR). 1.2-8, Section I.G of proposal form and ranking factors avoids duplicative efforts. 1.3-7: Consortium minutes. 1.3-8: MOU for Research Task Force. 1.3-9: BOR desalination research plan. 1.3-10 Excerpts from proposals about the absence of possible funding from other sources.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program is efficient and produces considerable results with a very small budget. Management is 4% of program budget. Program established a competitive environment and funds intramural & extramural research in a timely manner. It has established means to spread awareness and application of results and has external peer review mechanisms. Effective business practice improvements focus efforts on current and future BOR water management needs and goals, promote resource leveraging, avoid duplication, and evaluate costs and benefits. One area for improvement is the lack of overarching cooperative agreement/grant authority for R&D. Such authority would facilitate easier, stronger collaborative efforts with researchers at other institutions; and further increase competition for research funding. Other agencies such as Corps of Engineers (COE), USGS, National Park Service (NPS), and Fish and Wildlife Service (FWS) have such authorities. Another area for improvement is where power marketing agencies benefit from BOR R&D that lead to cheaper power,but do not fund BOR R&D.

Evidence: 1.4-1: New competitive call for proposals. 1.4-2: External award for Exceptional Process Improvement. 1.4-3: Western Coalition of Arid States letter shows stakeholder value/low cost. 1.4-4: 4% program management/admin costs based on actual expenditures. 1.4-5&6: Proposal form/guidelines ensure focus on needs & resource leveraging. New on-line system for efficient program workflow & progress reporting. 1.4-7: Proposals awarded with greatest potential; best benefits; high leveraging. 1.4-8: Reward innovative, high quality R&D at reasonable cost, penalize duplication of capabilities. 1.4-9: Independent steering team identifies R&D priorities. 1.4-10: Project partner list. 1.4-11: External reviewers ensure effectiveness. Ex: removal of a flaw: initially offered $25k coop. agreements for extramural desal. projects but efforts were too small to be effective, ensure completion, and the project admin. was a large percentage of costs. Increasing fed. share to $100k ensured useful R&D accomplished. See also 1.2-11 Desal Research Roadmap, pp 38-45. 1.4.12 Extramural desal. proposal process.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program targets Reclamation resource managers and the external water managers and water users served by Reclamation project waters. Since 2001, the S&T Program has been restructured to ensure maximum focus on program stakeholders and their priority needs, to involve them throughout the R&D process, and provide new R&D results. The program has also established program goals that measure performance in terms of the core values of stakeholders and their expected outcomes of water, power, and avoided costs. Internal & external stakeholders have stated support for program goals and practices. Since 1989, Reclamation has held desal workshops with stakeholders and conducted studies to identify needs and promising technologies. Reclamation used the information obtained from these activities to guide the program. Beneficiaries are targeted through the research solicitation process, which is competitive, merit-reviewed and cost-shared.

Evidence: 1.5-1: Steering team guides program direction & priorities. 1.5-2: Outreach workshop for stretching ag. water supplies & invasive species workshops demonstrate outreach & involvement of stakeholders.(also posted on BuRec website). 1.1-3 NWRA members show strong support for S&T Program Goal. 1.2-9: Letter from WESCAS states support for program. Both Steering & BOR Leadership Teams show strong support for program practices & management(see 2.6). 1.4-11: Stakeholders influence direction of efforts through relevancy review. 1.4-7, Relevancy proposal ranking criteria rewards proposals with greatest potential for high use of R&D outputs. Ranking process rewards proposals that work with stakeholders throughout the research process. 1.2-8, Sections I.F, I.G, & VI of the program proposal form shows that research is focused on problems specific to BOR. 1.2-6 & 1.2-7, S&T Program Roadmap, effectively targets proposals to support BOR's core mission. 1.5-3, 'Desalting Needs References,' shows how needs and beneficiaries are identified. 1.5-4, -5, -6 and -7, the current Broad Agency Announcements, describe how research projects are selected.

Does the program effectively articulate potential public benefits?

Explanation:  

Evidence:  

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation:  

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has 3 long-term outcome goals and measures : 1) Quantity of water liberated, 2) Kilowatts (kW) of power generated, and 3) avoided operational costs. The program contributes to safe, affordable, sustainable, and ample water and power supplies by developing and implementing solutions that liberate water for increased use, or more dependable beneficial use, for BOR stakeholders; solutions that contribute to increased power generation, and/or that reduce or minimize operational costs so program stakeholders can use their financial resources more effectively. All outcome goals try to achieve at least a 10:1 economic return of the federal S&T Program investment and include extramural desalination research contributions. Program efficiency measures are in terms of outputs that contribute to outcomes and pursue a 34% improved efficiency over 6 years. The desal. program has 3 additional long-term performance measures under review by the NRC: 1) Reduce the cost of desal. & water treatment; 2) Reduce environmental impact of facilitiy operations; and 3) Increase technology transfer and awareness.

Evidence: 2.1-1: Explains long-term outcome goals and measures; long-term goal reporting period is 6 years. 2.1-2, Section 1.1, the outcome goals of water liberated, power generated, and operational costs avoided support the program's goal and purpose. 2.1-3: Table demonstrates alignment of the 3 program outcome goals to Government Performance and Results Act (GPRA) goals. 2.1-4, Section III.A explains the long-term efficiency measures in terms of two key outputs: (1) Resource leveraging measures the value others place on S&T efforts and also directs more resources and coordination toward issues shared by partners; (2) Production and dissemination of S&T Bulletins that target program beneficiaries and peers with relevant findings. 2.1-5: Slides explaining output efficiency measures have been presented at NWRA, WESCAS, Performance Institute, Department of the Interior (DOI), et al. 2.1-6: S&T Bulletin template and production flowchart. Objectives in the Broad Area Announcement (BAA) (Section A.2.2) can be found in Tabs 1.5-4 to 1.5-7. Tab 1.2-10, Table 1, page 11, has desal long-term performance measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long-term goals have a 6-year timeframe. Outcomes target a minimum 10:1 economic return on the S&T investment in terms of outcomes that result from deployment of R&D outputs. Long-term output goals catalyze progress toward outcome goals. Output-based goals pursue greater levels (minimum 34% increase over 6 years) of (1) resource leveraging and (2) S&T Bulletins for more effective and timely transfer of new tools and knowledge to the end-user of the research outputs. Targets are ambitious considering the relatively small size of the program, diversity of R&D needs, and the fact that new research outputs must be put into use in order to achieve outcome goals. Desalination research targets described are ambitious. The desalination research roadmap has developed specific long-term (2020) targets that are very ambitious and are currently under review by the NRC. The approved desal roadmap targets will then become additional program targets.

Evidence: 2.2-1: Section II.A and III.B of S&T Program Goal document shows the specific, quantified program outcome targets and output targets over a 6-year reporting period (FY2005-10). 2.2-2: For the relatively small size of the program, achieving a 10:1 return on the R&D investment in terms of stakeholder outcome benefits achieved is considered ambitious. Likewise, a 34% increase over 6-years in the production efficiency of key program outputs is also considered ambitious. 2.2-3: Section II.E shows the established implementation schedule for program goals, baselines, and performance measures. Baselines for output measures are based on incremental annual improvements over the previous year. The FY2004 accomplishment will serve as the initial baseline for the FY2005-FY2010 reporting period. Establishing baselines for outcome measures are also scheduled by the end of FY2004. Tab 1.2-12, Table 1, p 11, specific needs-driven desal technology based targets and critical objectives were developed (i.e., long-term performance measures).

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: Annual performance measures are tied directly to annual budget proposals. To achieve the 34% long-term increase in the leveraging and knowledge transfer output goals requires a 5% annual increase in each measure. Output goal measures are normalized with respect to S&T Program investments to create an efficiency ratio that measures production vs. costs. Desalination research projects, partnerships, publications and presentations are required to address the measures stated in Question 2.1. The program management plan that stems from the desalination research roadmap will have annual performance measures based on critical research areas.

Evidence: 2.3-1: Section II.A and III.B of S&T Program Goal document shows the specific, quantified program annual outcome targets and output targets. 2.3-2: Section III.A of the S&T Program Goal document explains the program annual efficiency measures in terms of two key outputs 1) Resource leveraging that measures the value others place in program efforts and also directs more resources and coordination toward common issues shared by partners and 2) Production and dissemination of S&T Bulletins that target program beneficiaries and peers about relevant findings that can facilitate use of new solutions. 2.5-7 shows the evaluation form used. The stakeholder-driven desal research roadmap contains critical objectives based on needs, and matches technologies to address those needs. 2.3-3 shows an example of current efforts on development of one aspect (concentrate management) of the desalination research roadmap.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: Annual outcomes target a minimum 10:1 economic return on the annual research investment in terms of outcomes that result from deployment of R&D outputs. Annual output-based efficiency goals pursue greater levels (minimum 5% increase over previous year) of (1) resource leveraging and (2) timely transfer of new tools and knowledge to the end-user of the research outputs. Targets are ambitious considering the relatively small size of the program and the fact that new research outputs must be put into use in order to achieve outcome goals. Now that goals are established, baseline data will be collected in 2004.

Evidence: 2.4-1: Section II.A and III.B of S&T Program Goal document shows specific, quantified annual outcome and output targets. 2.4-2: Both annual outcome and output targets are established to show continued improvement and progress toward meeting long-term targets so that timely corrective actions can be taken toward achieving the long-term goals. 2.4-3: Section II.E of the S&T Program Goal document shows the established implementation schedule for program goals, baselines, and performance measures. Baselines for output measures are based on incremental annual improvements over the previous year. The FY2004 accomplishment will serve as the initial baseline. Establishing baselines for outcome measures is also scheduled by the end of FY2004. 2.4-4: Since program goal development activities did not begin until FY2002, determining if the annual and long-term targets are achievable is based on an analysis of recent case histories. Case histories, in Section II.D of the S&T Program Goal document, indicate targets are achievable and serve as a baseline indicator.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Recipients of funding must demonstrate how their efforts will contribute to research mission, goals, and performance measures. Proposal ranking and funding decisions consider alignment, potential contribution to goals, and past success in meeting goals. Accepted proposals become performance contracts. Failure to perform in accordance with contract jeopardizes continued and future funding. Technical Service Center (TSC) management has committed to supporting program goals and measures. Cooperative extramural research agreements contains the BAA as well as the proposal as part of the "contractual document", thus incorporating program goals. External cooperative partners are informed that collaborative efforts contribute toward BOR mission and research goals and that success and continued funding depends on results toward these goals. Recipients of Cooperative Agreements work on the projects specified in their proposals and federal project managers monitor work to ensure it stays directed to the long-term goals of the program and the specific project goals.

Evidence: 2.5-1: Proposal, sections I.F & V, require demonstration of contributions to goals. 2.5-2: Presentation used with potential partners/contractors shows goals and passion we expect from program partners. 2.5-3: Program stipulates that proposers commit to work toward and report on progress toward goals. 2.5-4: Annual project progress report and guidelines. Progress report is generated for each proposal by an on-line system. All "contractors" must report on their project progress as well as their progress toward their committed contributions toward program goals. Annual resubmittal for project review of progress is required for continued funding and to ensure continued relevance. 2.5-5: TSC, the primary contractor for intramural R&D, has provided a signed agreement to commit to and work toward the S&T Program goals. 2.5-6: Cooperative Research and Development Agreement (CRADA) form informs cooperating partners that collaborative efforts will contribute toward BOR mission. 2.5-7& 8 Pre- and full-proposal evaluation forms contain program objectives, page 4. See also 1.5-6, pgs 33 - 38

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Program steering team (external & internal program customers, stakeholders, peers) meets annually to provide strategic guidance, non-biased program evaluation, review & prioritize needs, assess progress, & review R&D projects. Periodic briefings on program practices & goals are provided to BOR's leadership & stakeholders with feedback requested. Annual, independent, technical & relevancy review of all new proposals and multi-year projects ensures high quality, a focus on BOR priorities, and progress. Annual reviews of multi-year projects ensures continued relevance & progress. Non-biased peer review groups used for certain cross-cutting R&D areas such as water operations and hydrologic process modeling. Independent peer reviews of the desalination R&D activities were done in 1993, 1995, 1998, & 1999. Presentations are made to critical audiences to get feedback. The 1998 peer review by NWRI was the most significant & forms the basis for the development of the desalination research roadmap, proposed facilitation role for the DOI and legislative concepts.

Evidence: Steering team roster (see1.3-2). 2.6-1: 2001 S&T steering team summary of issues & program responses. 2.6-2: 2001 S&T steering team reviews proposed program practice improvements shows support for practices. 2.6-3: 2002 S&T steering team evaluations & recommendations for improvements show strong support for new practices, focus, & accomplishments. 2.6-4: 2003 evaluations from steering team & BOR leadership show strong support for program management. 2.6-5: Existing & planned independent panels provide technical and user relevancy review of specific R&D areas. Each proposal is reviewed for mission relevancy and annual progress by independent BOR customers and program management (see 1.4-7 & 2.5-4). 2.6-6, 1998 NWRI desal R&D peer review contract; 2.6-7, NWRI peer document; 2.6-8, Changes planned or implemented in response to 1998 review. 2.6-9, Response from a committee member; 2.6-10; a less formal review in 1999. 1.2-11: Desal roadmap addresses program improvements & involved independent experts, and has NRC review underway. 1.3-9 shows planned improvements for desal R&D.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests are directly linked to program goals & show impact of funding levels on goals, although this level of detail is not always clear in OMB & Congressional requests. Annual output goal measures are based on production efficiency relative to program funding. Linking priorities to funding allocations identifies program impacts. Budget requests and cost reports include complete program costs and all direct and indirect costs are tracked and reported on each R&D project. BOR will start Activity Based Costing (ABC) in FY04 that will the numerous activities that support BOR GPRA goals. Desal budget requests are based on moving technologies from the proof-of-concept stage to piloting and demonstration. A promising new technology or process proven in a 1-year research/laboratory study ($100k maximum) in most cases would be funded at the pilot stage ($270k maximum) for 2 additional years. Funding to date has not allowed 2-year demo projects ($1M maximum). The desal roadmap & related management plan will better match budget requests to annual & long-term performance goals.

Evidence: 2.7-1: S&T Program Goal document includes a breakdown of program goals and budgets according to each of the four program focus areas. The goals, associated performance measures, and budgets are linked so that impacts and achievements with different funding levels can be clearly demonstrated. 2.7.2: The program steering team priorities with overlay of budget scenarios was used to show program impacts during BOR's FY05 budget formulation process. 2.7.3. Full Costing: Comments from the DOI response to 9/2002 GAO audit shows that the BOR cost accounting system is detailed and comprehensive and provides the full costs (direct and indirect) of its programs and activities and that full costs are reported in financial statements and budget reports. The program budget allocation pie chart shows the total costs of the program. Indirect costs are included in each category shown (1.4-4). The pie chart is used during annual budget formulation processes. 1.5.6&7 describe funding levels and task descriptions. 1.2-11, pg 29, Fig 10, desal roadmap has a very broad discussion of budget needs.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Director of Research and S&T Program Coordinator were new to their positions in FY2000. At that time, program practices and capabilities were assessed through interviews with numerous researchers, BOR managers, and water users. While areas of research were relatively productive, the program management lacked essential elements. As a result, significant new program practices implemented during FY01-FY03 included clear definition of program mission, development of a S&T Program roadmap that puts a sharp focus on BOR stakeholder needs, based competitive Request for Proposal (RFP) & research selection on merit and relevancy, establishing program goals and measures, expanding steering team to include external stakeholders and other agencies, and having the steering team set priorities. The program uses the steering team & periodic briefings with program end-users to provide ongoing input on strategic planning improvements. Many of the corrective steps recommended in the 1998 peer review of desalination research have been taken or are in the process of being implemented.

Evidence: Assesment by new management in FY 2000 found some research areas to be productive but program lacked many essential elements including: priorities, assurance of relevancy, program goals and measures, accountability, clear focus on BOR customers, objective proposal peer review, broad competition for funds. Significant new program practices implemented during FY01-FY-03 to correct these deficiencies are explained in sections 1 & 2 along with associated evidence. Long-term reauthorization of extramural desalination research and steady levels of budget requests will facilitate development and implementation of a management plan that is being created from the desalination research roadmap framework. Evidence contained in the responses to Section 1.1.-1.5 and Section 2.1- 2.4 are relevant for this response, especially sections 1.4 and 2.1-2.4

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals and used the results to guide the resulting activity?

Explanation: BOR currently has no authority to construct and operate research facilities. Congress directed that a new research facility be planned for desalination. As such, no programmatic alternative analysis was performed. To plan this facility, an Executive Committee of local stakeholders and technical experts was formed to conduct a systematic study of the potential roles for a Tularosa Basin desalination research facility. The study's draft Report to Congress (2.CA1-1) and Environmental Assessment contain recommendations based on alternatives analysis for the facility mission, location, conceptual design, site layout, anticipated costs, organizational structure, and accelerated design/construction process. The program is employing a performance-based design-build contract to reduce costs and eliminate unnecessary contracting delays.

Evidence: The Executive Committee met monthly from January 2002 through August 2002 in Las Cruces, NM and Alamogordo, NM to gather data, analyze alternatives, and select viable options for the development of the Tularosa Basin National Desalination Research Facility. 2.CA1-2, Minutes of Meetings. Site investigations and public meetings were also a part of the process to ensure the best possible product. In addition, BOR held two meetings of Denver technical divisions in April 2003 to validate the Executive Committee's recommendation for a performance-based design-build contract. They concluded that the private sector, performance-based contract was the best alternative for executing the construction of the facility when compared to in-house design-bid-build efforts by BOR or Sandia National Labs.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: The program generates significant research, and leverages significant funds for its size. Alternative R&D projects are identified through a BOR-wide call. Program reviews all proposals to see if BOR is best positioned to conduct/lead the research. Reclamation also evaluated the business practices and efforts of others such as NIWR, California Bay-Delta Program (CALFED), National Science Foundation (NSF), and Department of Energy (DOE) to develop and customize their practices to serve BOR-specific needs. Adopted best practices from NIWR and DOE. Other agencies performing desal. research have different goals and focus on mission needs (e.g., for the military, field reliability and portability are key factors in research rather than commercialization, cost reduction, and concentrate disposal). Reclamation periodically meets with other government agencies, the military and professional research organizations to compare efforts and leverage resources. Within the research roadmap framework, a proposed course of action was developed that currently optimizes benefits within the water community.

Evidence: BOR evaluated R&D business practices of other agencies through: talking to the President of NIWR & the Corps' R&D Director; touring the on-line CALFED proposal/review process; reviewing published literature on DOE; and talking with participants in these and other programs via the Water Resources Research Coordinating Committee (WRRCC). Annual RFP and proposal rating forms show the assessment and comparison of benefits, value, and other factors when selecting proposed efforts. The benefit to BOR's mission, costs, potential for broad application and use, and if BOR is best positioned to conduct or lead the proposed research are factors with the greatest weight. 1.3-7: At consortium meetings BOR compares programs with other federal agencies. 2.RD.1-1: Regular meetings with the reuse/desal research task force enables comparison of benefits/allocation of resources among participants. 1.2-11: the desal roadmap, pg 45, provides a mechanism for future comparison of efforts. 2.RD1-2. pg 2, contains a discussion of the NRC review. BOR's competitive process to compare proposals & select the most beneficial for stakeholders appears at 1.4.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: S&T Program steering team reviews/adjusts priorities on annual basis. Funding targets for the four program R&D focus areas in the call for proposals and in the annual budget submitted to Congress are framed around these priorities. Program uses these priorities down to each of the 20 individual research output areas to guide proposal funding decisions. Proposal ratings, administration priorities and line item flexibility also influence funding decisions. Annual budget requests are directly linked to annual & long-term program goals/measures, and clearly show impact of funding levels on goals. Linking steering team priorities to funding allocations also identifies program impacts associated with funding and policy decisions. The prioritization (evaluation) process for desal is described in the answer to question 3.CO1.

Evidence: 2.RD2-1: Steering team prioritizes 20 R&D output areas on the S&T Program roadmap. The roadmap defines each output area. FY2003 funding allocations to each priority area show that priorities are being used and correlate reasonably well with priorities. Correlation anomalies with the priorities are the result of administration priorities and line item flexibility limitations that exists under the FY2003 program line item structure. 2.RD2-2: Survey of BOR managers on the primary causes of water conflict. This information will be factored into program priority updates; it is fairly consistent with steering team priorities. 2.RD2-3: RFP shows funding targets based on steering team priorities. Priorities are in all desal BAAs (1.5-4 thru 7, section A.2.2). Desalination research roadmap priorities 1.2-11: the desal roadmap, pg 45, provides a mechanism for future comparison of efforts. 2.RD1-2. pg 2, contains a discussion of the NRC review. Our competitive process to compare proposals & select the most beneficial for our stakeholders appears at 1.4

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Timely and credible performance data is collected via annual & long-term program goal performance reporting including an on-line system that allows realtime reporting. On-line outcome goal reporting capability is being developed. Progress report is accessible to the R&D contractor & program management so that progress can be reviewed to ensure credibility. Multi-year projects resubmit & annually update progress. Progress report requires 3rd party contact info to verify progress and demonstrated performance is an essential factor for funding. Progress report & program performance measures help identify best practices, provide insights on problems, and help identify corrective actions to take. The program steering team provides annual evaluation and input for program improvements. The Contracting Officer's Technical Representative (COTR) approves payment based upon actual accomplishment of deliverables. Extramural desal has proposed an annual peer review process of project accomplishment in relation to the program's goals and would help identify management deficiencies.

Evidence: Progress report template and progress reporting guidelines are documented under evidence section 2.5-4. Existing progress report template and progress reporting guidelines demonstrate system established to collect regular, timely, credible performance information. Steering team evaluations also provide annual program performance input which is used to manage the program and improve its performance (see sections 4.5 and 2.8). 3.1-1 shows an example of the COTR responsibilities to ensure credible and timely performance of extramural researchers. All extramural research projects require quarterly reports, a visit by the COTR, formal presentation, and final report. The proposed peer review process is described in Tab 1.3-9, pg 4.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: The annual performance plans for the Director of Research and the S&T Program Coordinator incorporate program performance elements. Some managers in BOR's Technical Service Center (the primary S&T Program contractor) do not require performance related to research accomplishments; each desalination task area is assigned to a staff member who reports to the Group Manager and their performance as leaders and monitors is part of their interim and annual performance reviews. 3.2-3 Assignments are shown in the minutes of the DWPR meeting. Agency GPRA goals do not contain explicit performance related to incorporating new technologies and solutions into resource management practices. For extramural research, staff sit on the advisory boards of the program partners to guide the programs of these organizations. All partners and contractors are held accountable for their performance.

Evidence: 3.2-1: Annual performance plans for the Director of Research and the S&T Program Coordinator include achieving specific program results. Reviews have been stellar. 3.2-2: Primary program contractor is BOR's TSC whose performance plans include requirements to 'manage program accomplishments and provide quality service to clients'. All program R&D projects are required to annually submit a relevant progress report that demonstrates adequate progress toward program goals and key project tasks in order to be considered for continued funding (see section 2.5-4). Tab 1.5-5 section F.3.2 lists past performance as an award factor and section F.1.1 (b) evaluates the managerial capabilities of the proposer. Assignments of Task Leaders and individual project monitors are shown in the minutes of the DWPR meeting in Tab 3.2-3. Performance standards hold staff accountable (example at 3.2-4).

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Since 2001 BOR makes funding decisions and awards project funds by 10/01, contingent on appropriations action. Program annually obligates 100% of enacted budget. Proposals require a detailed task-based funding schedule. The progress report requires task-by-task tracking of progress. We use the Federal Finance Accounting system to track expenditures against each project monthly and more frequently when necessary. Tracking includes auditing categories of expenditures to ensure consistency with project intent and working with research contractor to correct inappropriate charges. As soon as the appropriation is received, a cost centered budget is developed for specific projects and is formally reviewed for accomplishment three times a year. Adjustments, generally modest, are made as the year proceeds and more accurate cost figures become available. As work is invoiced, the COTR matches accomplishments before recommending payment. At the end of the fiscal year, finances are monitored daily.

Evidence: 3.3-1: Funding approval notices show timely awards at start of FY. 3.3-2: Memo from TSC, primary contractor of program R&D, stating that awards have been consistently made at the beginning of the FY. 3.3-3: End-of-year carryover & accomplishment summary report for FY98-02 show good accomplishment & fund management with minimal carry over. 3.3-4: Sect. II & III of the program proposal form and Progress Report Sect. X.C and X.E show the required task-based cost and schedule plan. 3.3-5: Monthly and year-end spending reports at program and individual project level are used to track expenditures. 3.3-6: Federal Finance System report showing the detail of expenditure categories. 3.3-7: The desal budget plan for FY02 was spent for intended purposes: 57% for cooperative agreements, 32% for partnerships, 3% on technology transfer and 7.4% on program administration. 3.3-8 includes two examples (after 9 and 12 months) of monthly financial sheets showing expenditures by project. The final sheet indicates an unspent balance of $100 for FY02.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Program output measures reflect production/unit cost. Program admin is 4% of funding for research overall, and 7.4% for desal. Significant business process re-engineering has occurred over last two years including a competitive RFP process. Most intramural research is done by staff in the TSC which is subject to A-76 competitive sourcing requirements. IT improvements have been implemented with further modules underway to accomplish the program and streamline efforts. Automated, web-based system is speeding proposal submittal, peer review, and award processes and includes progress reporting and performance measure reporting for program goals (outcome goal reporting under devel). It enables others see what products are expected each year, attracts partners (which increases program cost effectiveness), & facilitates sharing of findings. Extramural desal research provides funds based on merit-reviewed, cost shared, and competitive cooperative agreements. Cooperative agreements permit revision as efficiencies are discovered. All proposals are reviewed for appropriateness of cost.

Evidence: BOR has de-layered management structure since 1994. Under A-76 competitive sourcing, TSC will convert 63 positions by 9/2003. BOR will convert additional 550 positions by FY06. Annual & long term program efficiency measures (2.1-4, 2.1-5, 2.2-3) are in terms of production/unit cost. Efficiency measures focus on key program outputs: 1) resource leveraging & 2) sharing information. 1.4-4: program cost categories w/ management/admin at 4%. 1.4 documents program business practice improvements implemented over past couple years. 3.4-1: Flow chart shows our online proposal process. System uses web-enabled data base for easy, efficient, effective entry, review, tracking progress, & dissemination of information. 2.1-6: On-line S&T Bulletin draft template & production flowchart. Template & process to be finalized by the end of FY03 & implemented as an on-line web-enabled data base. Award of program R&D projects at the start of the past prior 3 fiscal years (3.3-1 & 3.3-2). 3.4-2 Cost reasonableness is an evaluation factor on desal evaluation form and on S&T Program eval form(see 1.4-8).

Does the program collaborate and coordinate effectively with related programs?

Explanation: Meaningful and relevant coordination and collaboration is a fundamental tenet of the program with numerous mechanisms in place to encourage and accomplish effective collaboration. Program management coordinates desalination efforts within the federal community, professional research organizations, the military, and various water authorities through several distinct mechanisms. Researchers also coordinate and collaborate with their counterparts at other federal agencies and research institutions. Researchers have both formal and informal relationships and communications with researchers in related fields. Reclamation coordinates desalination efforts within the Federal community, professional research organizations, the military and various water authorities through the Interagency Consortium.

Evidence: River modellers coordinate via the Interagency Hydrologic Modeling Committee; USGS, Tennessee Valley Authority (TVA), and COE collaborate on Riverware development; BOR coordinates with the Department of Agriculture (USDA), COE, and several universities on invasive species R&D, and with the Electric Power Research Institute (EPRI) on power efficiency and related issues. 3.5-1: WRRCC ensures no duplication of water resources research. 3.5-2: Steering team helps collaboration and avoids duplication, with collaboration via the Desal Consortium and the Recycling/ Reuse Task Force. Ex: 2.3-3, Leveraged funding by three of the five members of the Reuse Task Force. 3.5-3 By partnering with the Navy, identified research opportunities 3.5-4, Awwa Research Foundation (AWWARF)/BOR Workshop led to joint funding of 2 projects. 3.5-5, Work with the Middle East Desalination Research Center via State Dept support. 3.5-6: List of partner entities for FY2003 S&T projects. 3.5-7: Use Cooperative Research and Development Agreements (CRADAs) to collaborate with other organizations.

Does the program use strong financial management practices?

Explanation: Budget Analyst uses the Federal Finance Accounting system to track expenditures against each project on a monthly basis, and more frequently when necessary. Tracking includes auditing categories of expenditures to ensure consistency with project intent and working with research contractor to correct inappropriate charges. Recent General Accounting Office (GAO) audit to evaluate cost accounting and cost recovery practices for Reclamation's Bureauwide programs found no anomalies with the S&T Program and that activities funded were consistent with Reclamation's authorities for research. Every extramural research agreement has a financial plan and schedule supervised by the COTR and the Contracting Officer. The COTR certifies that the information is accurate and timely. The Acquisitions and Assistance Management Services Department assures that the financial systems meet statutory requirements. Cooperative agreements are periodically audited including a recent one by the Inspector General's office. The TSC has a budget group that monitors spending and overspending.

Evidence: Strong program financial management practices are fully documented in Section 3.3. 3.6-1: GAO audit report number GAO-02-973 found no anomalies with the S&T Program and that activities funded were consistent with Reclamation's authorities for research. 3.6-2 Forms showing staff hours, lab rates, and equipment costs by task are required in all research cooperative agreements. Cooperative agreements are periodically audited, including a recent one by the Inspector General's office. The Technical Services Center has a budget group that monitors spending. The Science and Technology Office has a Budget Analyst who monitors expenditures through the Federal Financial System.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Because of feedback and suggestions from steering team, BOR's leadership team & others, and because the NRC's 2001 report also identified water resource R&D coordination among federal agencies as a concern, program expanded the steering team membership to include stakeholders, USGS, ERS, & ARS and established the Federal Water Resources Research Coordinating Committee to better coordinate and catalyze federal water resources research collaboration across the federal government. We have developed plans and actions to enhance collaboration and coordination with USGS & NIWR, integrated coordination and collaboration incentives and guidance within the proposal process. BOR leadership also identified better outreach to end-users as an improvement need, resulting in monthly activity reports, the newly-established S&T bulletin concept, and research exchange workshops. To respond to Presidential Management Initiatives, developed program goals and performance measures & implemented web-enabled efficiencies to better manage program workflow.

Evidence: Program coordination improvements are fully documented in Section 1.3, 1.5, and 3.5. Program goals and performance measures are fully documented in Sections 2.1 to 2.4. S&T Program steering team evaluations & responses to other program management concerns are contained in section 2.6 explanation and evidence. Web-enabled IT efficiencies are fully documented in section 3.4-1.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: Not applicable as facility construction is not yet fully authorized. However, construction of the Tularosa Basin National Research Facility will be managed by clearly defined deliverables, performance characteristics and carefully planned cost and schedule goals identified in the Draft Report to Congress: Tularosa Basin National Desalination Research Facility Study (Draft Report) of September 2002. See Tab 2.CA1.1

Evidence: 2.CA1-1: The Draft Report documents that the mission of the facility is to conduct R & D activities in desalination of brackish ground water, concentrate disposal issues, and renewable energy/ desalination hybrids. The facility is also to function as an information center for the public. The draft report to congress contains a mission statement (pg. 1), operational guidelines (pg. 30), conceptual design (pp. 11-14), facility layout (pp. 15-28), organizational structure ( pp. 30-32), cost breakdown (pp. 29, 33), and streamlined performance-based design-build process (pg. 34). 2.CA1-2, Minutes of Meetings, provides an elaboration of these elements. The program was further refined through organizational meetings to plan the scope of work, cost and schedule for a performance-based design-build contract to be issued July 15, 2003. The defined deliverables are specified through floor plans, architectural views and site layouts. BOR established a 24-month schedule for completing the project, which will reduce costs and project duration. 3.CA1-1: shows a draft statement of objectives.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Not directly applicable as this program uses Cooperative Agreements instead of grants. The agreement cycle begins with a widely distributed announcement and request for preproposals. All preproposals are assessed based on merit of concept and appropriateness to our published program goals. The proposals are evaluated by at least three individuals, including some individuals outside Reclamation.

Evidence: Examples of publicity for the competition are shown in 3.CO1-1 to -3 and 2.RD1-2. The procurement is widely advertised on Reclamation's Water Treatment website, through Government procurement sites, and by our newsletter that receives wide circulation in the water community. About 40% of preproposers are requested to submit a full proposal. 2.5-8: The proposal rating sheet, which uses ten factors shows the importance of factors in rating. About 40% of proposals are funded. Recipients of awards between FY98 and FY02 are listed in Tab 3CO1-4. In FY99, 8 of the 12 recipients were new, excluding two 2-year pilot projects. No awards made in FY00. In FY01, 5 of the 8 recipients were new. In FY02, 9 of the 10 recipients were new. The pilot scale projects under Task H are for two years. These are listed separately so there appear to be more repeat awards than there actually are. There is a unique case where a recipient was funded three cycles in a row. This was a technically difficult membrane development.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Not directly applicable because extramural research is handled through cooperative agreements in lieu of grants to ensure that BOR has sufficient oversight and influence over program data quality. The proposed work is required to be broken down into tasks and costs for labor, materials and supplies are provided for each task in the proposed budget and monitored closely. Each program is monitored by a GCAOTR (Grants and Cooperative Agreements Officer's Technical Representative) who is required to understand the content and track the progress of the program.

Evidence: One or two GCAOTRs, people familiar with the technology under study, are assigned to oversee each award. As an aid to oversight, each GCAOTR is provided with the Technical and Financial Proposals, and with a sheet, shown in 3.1-1, showing contacts and a schedule for contacts. Early in the award period, the GCAOTR visits the study site. The Principal Investigator (PI) is required to submit quarterly progress reports. Telephone contact is maintained at least once a quarter. The GCAOTR is typically able to make meaningful contributions to the research program. Each invoice is reviewed for approval by the GCAOTR with program costs being checked against the detailed budget in the Financial Proposal. On completion of the program, the PI submits a final report and makes an oral presentation to researchers in Denver. The draft final report is reviewed and comments are submitted for incorporation in the published version. 3.CO2-1 shows an example of the comments provided by the GCAOTR. Three times a year, the GCAOTRs and the Group Manager meet to discuss program progress. See minutes of meeting, 3.2-3.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Not strictly applicable as extramural research is handled by cooperative agreements. New awards are summarized in the Water from Water newsletter, 3CO3-1, and the Bureau of Reclamation website.

Evidence: The results from each research project are published in a formal final report. These reports are available in hard copy from Reclamation and are delivered to those who request them. The reports can also be downloaded from Reclamation's Water Treatment website. These reports are also contained in a set of CDs that are being issued as part of the history of desalination and water treatment research. Note that the pilot plant projects produce a final report only at the end of the 2-year period, the first year generally being devoted to construction of the pilot plant rather than generation of data. The publication library from this program now contains almost one hundred technical reports. The list of reports is shown as Tab 4.1-6. A sample report, one of the shorter ones, appears as 3CO3-2. In addition BOR encourages presentation at technical conferences and in refereed journals.

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: The program uses an annual, BOR-wide, competitive call for proposals implemented in 2002 for all intramural R&D. Each proposal is evaluated for mission relevancy & technical merit. Relevancy review is conducted by BOR cadre of subject matter resource managers across BOR (external to program). Technical review is conducted by at least 3 technical experts. One from BOR's TSC & two external to TSC and ideally external to BOR (sometimes there is not expertise outside of BOR). Funded proposals are selected from those having both high technical and relevancy scores and which are aligned with steering team/Administration priorities. Proposal and selection process meets Circular A-11 definition for merit reviewed research with competitive selection using external (peer) evaluations. Approx 10% of awarded funds are sub-contracted to external entities for specific tasks through subsequent contracts and cooperative agreements. The proposal & selection process for extramural desalination R&D meets Circular A-11 definition for merit-reviewed research w/competitive selection & external (peer)evaluation(see 2.5-8).

Evidence: 3.RD1-1: Web page outlines the S&T Program competitive, merit-based process(also documented in Section 1.4). Peer review occurs on all intra&extramural research. Historically nearly all desal funds written in by Congress in highly variable amounts; however, in FY04 the Admin. requested $4.7M for desal R&D spread across several line items. In addition Congress earmarks in 02 & 03 about $2M/year for funding extramural weather modification research under BOR's drought authorities. Research Office manages weather mod research activities under drought authorities for BOR's Office of Policy & Program Services. On average, 86% of FY02 & 03 intramural research was competed (remaining funds were used for tech transfer, outreach, special studies, e-gov IT improvement, & program management). In FY02, 57% of all desal funds received were subject to competition. Remaining funds went to tech transfer & program admin; and at Congress direction: partnerships, clearinghouse develop/management, desal roadmap, Tularosa Basin studies. External NRC,NWRI desal assessments documented elsewhere.

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation:  

Evidence:  

Does the program adequately define appropriate termination points and other decision points?

Explanation:  

Evidence:  

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: Relevant, long-term outcome and output goals and measures are clearly established (see Questions 2.1 and 2.2). Functional schedule for establishing meaningful baseline and performance measure tracking are in-place and scheduled to be completed by the end of FY2004. The long-term goal performance reporting period is FY2005 'FY2010. As such, complete and accurate performance tracking consistent with the program goal objectives and framework are not available at this time. However, performance evidence to date suggests the program is on track, achieving stakeholder value and expectations, and making progress toward successful long-term goal accomplishments. A long-term performance goal in the authorizing legislation for desal required a report to Congress recommending demonstration plants. This report was provided in May 2001. Although these goals are new, the program also made significant progress toward achieving its prior long-term goals, which were revised and collapsed into the new goals.

Evidence: A variety of documents show the progress made toward both old and new program goals. These include: 4.1-1, Case history analysis in Section III.D of the S&T Program Goal Document; 4.1-2, Summary of various success stories indicating program achievements; 4.1-3, Letters from stakeholders recognizing value of the program; 4.1-4: Summary of collaborating/partner entities for FY2003 projects as evidence of achievements; 4.1-5, Summary of current Agreements as evidence that achievements are being made toward goals. High ratings and statements of support in Question 2.6 show satisfaction with achievements. The desal program has funded 84 cost-shared projects with 50% of the funding going to at least 27 universities. 4.1-6 is a list of reports distributed by the program. During this period, desalination costs are reported to have dropped 50%, part of which can be attributed to work in this program. 4.1-7, slide 12, shows the decreased costs of desal. A measure of the program's success is the growing interest in membrane bioreactors, recommended in the report to Congress, 4.1-8.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual outcome & output goals and measures are clearly established (see Questions 2.3 and 2.4). A schedule for establishing meaningful baseline by the end of FY2004 is in place. The first formal goal reporting year is FY2005, so complete performance tracking is not available now. However, evidence suggests the program is now on track, achieving stakeholder value, and making progress toward successfully accomplishing annual performance measures. The annual performance goals for the desal program are to fund projects within the priorities for that year and hold the contractors and partners accountable on individual projects. The desal roadmap and the management plan will set measurable annual performance goals. Initial program output goals for each S&T program R&D output area were established, linked to our S&T Roadmap, and posted on our website in FY2002. FY03 program R&D awards were targeted at the goals and program priotities.

Evidence: Annual goals and performance measures are strategically linked to the long-term goals to provide meaningful and relevant data about progress toward the long-term goals. Consequently, the evidence in Question 4.1 is also the evidence for Question 4.2. See evidence in section 1.2-7 for the output goals developed and posted on our website in FY2002 for each of the R&D output areas on our S&T Program roadmap. Stuctured goal measure tracking not implemented in past but is scheduled for FY2004 (see 2.2-3). However, program accomplishments consistent with goal objectives are documented in section 4.1 and are also posted on our website.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: Measures of output goals are normalized with respect to S&T investment to create a ratio of production to costs (see Sections 2.1-4, 2.1-5, and 2.3-2). Outcome goal measures are based on achieving a high return on the S&T investment. These measures will provide consistent, meaningful indicators of efficiency and cost effectiveness from FY2005 to 2010. Although formal progress against these measures is not scheduled to be reported yet, the program has shown improvement since 2001. The change in the desal program, recommended in the 1998 peer review, to develop a research roadmap is producing significant efficiencies. The roadmap facilitates setting priorities based upon expert advice and helping to realize efficiencies by combining similar testing for differing needs. The roadmap is also outlining areas of collaboration to provide the greatest payoff. Where others have pursued research separately, the roadmap provides a guide for collaboration. Our goal is to have the best and brightest engaged in the highest priority research.

Evidence: 4.3-1 is a Denver News article: 2003 Award for Exceptional Productivity or Process Improvement given to program. 4.3-2 is a DOI People, Land & Water news article on program tech transfer effectiveness. 4.3-3 is a letter from the WESCAS showing recognition of high level of progress with relatively small budget. Evaluations shown in Questions 2.6-3 & 2.6-4 recognize recent program improvements. Other program improvements addressing coordination & collaboration to increase efficiency and effectiveness of R&D investments are documented in section 1.3. Recent program business practice improvements to improve intramural R&D effectiveness & efficiency are documented in Question 1.4. Recent program IT improvements to increase efficiency appear in 3.4-1. Progress toward A-76 competitive sourcing implementation is included in Question 3.4. Progress toward program efficiency is documented in Questions 4.1 and 4.2. 4.3-4 contains an analysis of how a currently funded research project could reduce the cost of desalination through reduced operating expenses & increased investor confidence.

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: No other R&D programs are focused & dedicated to developing solutions for BOR water managers and water users and their unique western water needs. There are similar or complimentary R&D activities performed by USGS, COE, &USDA. Although there are some similarities with these organizations, BOR's mission and R&D needs are different. For example, COE is a flood control/navigation agency and BOR is a water supply agency. Also USGS primarily concentrates on basic research and problem definition while BOR concentrates on applied R&D and solutions to problems. BOR compared their program practices with NIWR which focuses on applied R&D by academia for state water managers, and found their practices to be compatible with NIWR practices. BOR also compared their program practices with COE and with organizations that concentrate on basic water resources research and found their own program practices are strong in comparison. Stakeholder feedback also indicates the program compares very favorably with other programs that conduct water resources R&D that compliments BOR efforts and focus.

Evidence: Program comparisons and results are further explained and documented in Question 2.RD1. BOR actions to better understand, coordinate and integrate with other complimentary water resources research programs are explained and documented in Question 1.3. The size of BOR's R&D is significantly smaller than the other federal R&D programs with activities related to water and agriculture (COE,USGS, USDA),has comparable to stronger processes, and has productivity. Power marketing agencies fund no R&D on power efficiency and reliability.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Comments from the program steering team (consisting of external stakeholders, academia, USGS Regional Directors, USDA, NIWR, and BOR customers (region and area offices)) indicate strong support for program management, practices, and results. External evaluations indicate the program has been moderately effective in the past, and in recent years has drastically improved; new performance measures should better track effectiveness. The program is also being reviewed as part of a National Research Council (NRC) review of federal water resources research to address Congressional response to the 2001 NRC report 'Envisioning a Water Resources Research Agenda for the 21st Century'. The NRC is currently reviewing the desalination research roadmap as the basis for future investments in the program and will determine if it is an effective way to address the nation's water needs. The program has never requested an evaluation that focused only on its effectiveness, although as a part of a nomination for a prestigious water prize, the program was used as a justification of the nomination.

Evidence: The independence and diversity of the steering team is documented in the steering team roster (see 1.3-2). Program evaluations and results are documented in Question 2.6. Other documentation of program impact and value is contained in evidence 4.1-1 and 4.1-2, and Question 4.3. The initial comments, due in late June, from the National Research Council's review of the desalination research roadmap, will indicate the effectiveness of this approach. Tab 4.5-1 is a letter of nomination that cites the accomplishments of the desalination program.

Were program goals achieved within budgeted costs and established schedules?

Explanation: Not applicable. No construction is occurring as the program's only capital project, the Tularosa Basin Deslination Research Facility, is not authorized.

Evidence:  

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation:  

Evidence: