Detailed Information on the
Federally Funded Research and Development Centers Assessment

The program should take necessary steps to ensure that at least 90% of its facilities keep scheduled operating time lost under 10%.

The program should improve how it measures the use and effectiveness of its FFRDCs.

The program will ensure increased timeliness of yearly project reports from investigators.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment of "significant achievement" that FFRDCs provide leadership in the development, construction, and operation of major, next-generation facilities and other large research and education platforms.

Explanation:Assessment by the external Advisory Committee for GPRA Performance Assessment of "significant achievement" in activities that provide leadership in the development, construction, and operation of major, next-generation facilities and other large research and education platforms.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment of "significant achievement" that FFRDCs provide for the collection and analysis of the scientific and technical resources of the U.S. and other nations to inform policy formulation and resource allocation.

Explanation:Assessment by the external Advisory Committee for GPRA Performance Assessment of "significant achievement" in the collection and analysis of the scientific and technical resources of the U.S. and other nations to inform policy formulation and resource allocation.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment of "significant achievement" that FFRDCs support research that advances instrument technology and leads to the development of next-generation research and education tools.

Explanation:Assessment by external Advisory Committee for GPRA Performance Assessment of "significant achievement" in support research that advances instrument technology and leads to the development of next-generation research and education tools.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment of "significant achievement" that FFRDCs expand opportunities for U.S. researchers, educators, and students at all levels to access state-of the-art S&E facilities, tools, databases, and other infrastructure.

Explanation:Assessment by the external Advisory Committee for GPRA Performance Assessment of "significant achievement" in activities to expand opportunities for U.S. researchers, educators, and students at all levels to access state-of the-art S&E facilities, tools, databases, and other infrastructure.

Measure: Percent of operational facilities that keep scheduled operating time lost to less than 10%

Explanation:This measures demonstrates that the agency's FFRDCs, as a whole, keep actual operating time lost to less than 10% of scheduled operating time.

Measure: Number of users of National Center for Atmospheric Research (NCAR) data sets with unique access addresses who have downloaded data within the last 12 months.

Explanation:These data sets consist of value added observational and model data and are used to further research and education activities within the atmospheric and related sciences. The measure bears directly on the efficiencies of NCAR in that increasing the number of users who download these data sets demonstrates an increased usage of scientific outcomes invested in this FFRDC without a significant change in funding.

Measure: Percent of observing time at the National Optical Astronomy Observatory (NOAO) that is awarded competitively through the NOAO allocation committee.

Explanation:The remaining time is NOAO Director's Discretionary time that can go to worthy proposals that went through competitive review but did not make the cut, to targets of opportunity (e.g. supernovae) or to NOAO staff after evaluation by telescope allocation committee. This percentage does not include engineering time (measured separately as part of the efficiency metric), telescope time committed under international agreements (e.g. telescope time given to Chile, the location of one of the NOAO observatories), or fixed-term observing time awarded by limited competitive review through divestment of older telescopes or as a result of partnerships with universities or consortia for technology development. While continuing to remain at only a small fraction of the total time, the NOAO Director is permitted to assign up to 25% of NOAO time through his/her discretion.

Is the program purpose clear?

Explanation: The purpose of NSF's investments in Federally Funded Research and Development Centers (FFRDCs) is to support "investments in research, development, and R&D policy that create unique, important and long-term capabilities for the Federal government" that cannot be met as effectively by the broad US university-based research community. This program provides an umbrella organization for NSF's FFRDCs, which "provide leadership in the development, construction, and operation of major, next-generation facilities and other large research and education platforms" and "provide for the collection and analysis of the scientific and technical resources of the U.S. and other nations to inform policy formulation and resource allocation." (NSF FY 2003-2008 Strategic Plan). The NSF Act of 1950 authorizes and directs the Foundation "to initiate and support basic scientific research and programs to strengthen scientific research potential and science education programs at all levels" and "to foster and support the development and use of computer and other scientific and engineering methods and technologies, primarily for research and education in the sciences and engineering." The five FFRDCs are uniquely positioned to provide capabilities and state-of-the-art instrumentation to probe fundamental questions in science and/or to address pressing scientific and technological issues facing the nation and the international community. The centers designated as FFRDCs by NSF in the 1970s are: National Astronomy and Ionosphere Center (NAIC), National Center for Atmospheric Research (NCAR), National Optical Astronomy Observatory/National Solar Observatory (NOAO/NSO), National Radio Astronomy Observatory (NRAO) and Science and Technology Policy Institute (STPI), formerly the Critical Technologies Institute.

Evidence: The purposes and roles of FFRDCs at NSF are articulated in: NSF FY 2003-2008 Strategic Plan, pages 18-19 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201) National Science Foundation Act of 1950, Functions of the Foundation (frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=browse_usc&docid=Cite:+42USC1862) National Science Foundation Authorization Act Public Law 107-368, Section 2 (www.nsf.gov/mps/ast/aaac/p_l_107-368_nsf_authorization_act_of_2002.pdf) For more information on individual FFRDCs, see: The National Center for Atmospheric Research (www.ucar.edu) National Optical Astronomy Observatory (www.noao.edu/outreach/aboutnoao.html) National Astronomy and Ionosphere Center (www.naic.edu/global/naic_mission.htm) National Radio Astronomy Observatory (www.gb.nrao.edu/) Science and Technology Policy Institute (formerly the Critical Technologies Institute) (www.ida.org/stpi/index.html)

Does the program address a specific and existing problem, interest, or need?

Explanation: The specific and existing needs addressed by each FFRDC vary across the different FFRDCs supporting astronomy, atmospheric science, and policy. For example, NOAO/NSO provide access to the international scientific community to a complement of optical and infrared telescopes and associated instrumentation in both the northern and southern hemispheres for studies of the universe and the Sun. NAIC operates the world's most powerful radar transmitter for studies of the atmosphere and solar system as well as the world's largest single-dish radio telescope. NRAO provides access to the international community to a large collection of radio telescopes for investigations of the universe. These include the Very Large Array and the Robert C. Byrd Green Bank Telescope. NCAR provides state-of-the-art observational facilities to the atmospheric sciences community that includes a High-Performance Instrumented Airborne Platform for Environmental Research (HIAPER). This research aircraft with more than 7,000 miles of range, with capabilities to reach and exceed 50,000 feet, and with 12 to 15 hours of endurance, has advanced, lightweight, low-power sensors and instruments customized to the aircraft's performance and missions. This facility provides capabilities to the community unique for the examination of an enormous range of long-standing and recent scientific problems. Broadly speaking, NSF's investments in the FFRDCs are spelled out in the NSF Act of 1950, as amended and reiterated in the NSF Authorization Act of 2002, Sec. 2. Findings and Sec. 3. Policy Objectives. Sec 2. Paragraph 4 notes that the research and education activities of the National Science Foundation promote the discovery, integration, dissemination, and application of new knowledge in service to society and prepare future generations of scientists, mathematicians, and engineers who will be necessary to ensure America's leadership in the global marketplace. Sec. 2, Paragraph 6 states that the emerging global economic scientific, and technical environment challenges long-standing assumptions about domestic and international policy, requiring the National Science Foundation to play a more proactive role in sustaining the competitive advantage of the United States through superior research capabilities. Sec. 3 describes the allocation of funds in relation to the policy objectives set forth by the NSF Authorization Act of 2002 "to strengthen the Nation's lead in science and technology," "to increase overall workforce skills," and "to strengthen innovation by expanding the focus of competitiveness and innovation policy at the regional and local level."

Evidence: Evidence demonstrating that FFRDCs address specific and existing problems can be found in: NSF FY 2003-2008 Strategic Plan, page 18-19 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201) National Science Foundation Act of 1950, Functions of the Foundation (frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=browse_usc&docid=Cite:+42USC1862) National Science Foundation Authorization Act Public Law 107-368, Section 2 (www.nsf.gov/mps/ast/aaac/p_l_107-368_nsf_authorization_act_of_2002.pdf) The National Center for Atmospheric Research (www.ucar.edu) National Optical Astronomy Observatory (www.noao.edu/outreach/aboutnoao.html) National Astronomy and Ionosphere Center (www.naic.edu/global/naic_mission.htm) National Radio Astronomy Observatory (www.gb.nrao.edu/) Science and Technology Policy Institute (formerly the Critical Technologies Institute) (www.ida.org/stpi/index.html)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The NSF FFRDCs are not redundant or duplicative of any other efforts because they support unique facilities and focused research activities to complement and enhance research, education and policy activities in several of the disciplines for which the Foundation has responsibility. NSF is the only Federal agency that dedicates its FFRDCs to supporting the research, education, and policy needs of American universities (except for STPI, which supports the Office of Science and Technology Policy). The long-term support for the FFRDCs, the level of funding, and the breadth of capabilities at the FFRDCs is not matched in other modes of support employed by NSF, and no other program inside or outside the NSF offers such wide access to facilities. The Foundation has a responsibility to consider what large facilities are needed to maintain the nation's leadership in science and engineering. NSF consults with other agencies to avoid duplication and cooperates with other agencies and with international partners in constructing facilities, establishing observational experiment designs, and paradigms for data collection, management and sharing. With respect to large and midsize facility projects undertaken at the FFRDCs, NSF senior management reviews and compares opportunities of competing projects and forwards recommendations to the NSB for subsequent review and approval. Interagency and international agreements and understandings are active and on file for most facilities projects, demonstrating the commitment of NSF to efficient and effective coordination of efforts. External and internal reviews of FFRDCs address the question of the benefits for the community of activities at the Centers.

Evidence: NSF's FFRDCs are designed not to be be redundant or duplicative of other efforts. The agency has specific, statutory authority to evaluate the status and needs of the various sciences and engineering and to consider the results of this evaluation in correlating its research and educational programs with other Federal and non-Federal programs. National Science Foundation Act of 1950 under the "Functions of the Foundation (42 U.S.C. 1862, Sec 3.a)(5)"; http://assembler.law.cornell.edu/uscode/html/uscode42/usc_sec_42_00001862----000-.html). STPI, formerly the Critical Technologies Institute (CTI) was established as a FFRDC by the National Defense Authorization Act for Fiscal Year 1991 [P.L. 101-510] and the National Defense Authorization Act for Fiscal Years 1992-1993 [P.L. 102-190].

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: No major design flaws have been identified that would limit the effectiveness or efficiency of the NSF FFRDCs. NSF relies on the external review process, NSF Program Officers in their oversight capacity, and Committees of Visitors (COVs) to ensure that facilities are effectively serving their intended communities, and to recommend changes to improve program effectiveness and efficiency. These measures ensure that supporting the acquisition and operation of infrastructure is the most efficient and effective method of facilitating the science in question. The FFRDCs have "user groups" and advisory committees that communicate regularly with NSF and facilities managers. Merit review by peers has been recognized as a "best practice" for administering R&D programs. No strong evidence exists that an alternative design would better achieve the purpose of the NSF FFRDCs.

Evidence: Evidence showing that NSF's FFRDCs are free of major flaws that would limit effectiveness or efficiency is shown in the following documents: FY 2004 Performance and Accountability Report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=par); Report to the NSB on the NSF Merit Review Process - FY 2004 (http://www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf)August 2004 OMB/OSTP Guidance Memo (http://www.ostp.gov/html/m04-23.pdf); COV Reports (list of all reports - http://www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - http://www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate http://www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: NSF's FFRDCs support unique facilities to enable research and education activities for strategically targeted disciplines for which the Foundation has responsibility. In addition, the FFRDCs provide unique capabilities in science, technology, education, and underlying intellectual assessment necessary for policy makers. These capabilities are not broadly available within the university or college environments (or, in the case of STPI, within OSTP). The peer review process for access to specific facility resources and/or time ensures effective targeting of funding so that results of investments will reach the intended beneficiaries. Committees of Visitors ensure relevance to community needs.

Evidence: Evidence demonstrating that NSF's FFRDCs are effectively targeted can be found in: MyNSF, (formerly NSF's Custom News Service) is available at www.nsf.gov/mynsf/ Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf) COV Reports (list of all reports - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The NSF FFRDCs have focused, long-term plans in place that specify institutional goals and expectations for research, education, and policy outcomes. These plans are reviewed and updated periodically. The FFRDCs serve well defined communities of scientists and policy makers and provide unique facilities and intellectual capabilities and capacity that serve these communities. It is the service to these communities that defines the long-term plans developed by each Center. In addition, the facilities provided by the FFRDCs are a subset of the Tools Strategic Goal -- providing "broadly accessible, state-of-the-art S&E facilities, tools, and other infrastructure that enable discovery, learning and innovation" and thus reflect another driver for plans and measures implemented by the FFRDCs. This strategic goal reflects the parts of NSF's mission directed at programs to strengthen scientific and engineering research potential, and to support the development and use of tools and other scientific methods and technologies.

Evidence: NSF's FFRDCs have specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program as found in the Measures Tab of this document.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The FFRDCs enable discoveries or enhance productivity of NSF research or education communities: The goal of "significant achievement" requires external assessment of facility development and provision, science, and engineering outcomes based on knowledge of science achievement on a world-wide stage. The FFRDCs engage in a multitude of partnerships to support and enable development of large facilities and address grand challenges. FFRDCs contribute to the solution of major unanswered scientific questions.

Evidence: Evidence of NSF's FFRDCs ambitious targets and timeframes can be found in the Measures tab of this document. Other evidence is found in: FY 2004 AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); NCAR strategic plan (www.ncar.ucar.edu/stratplan/plan.pdf) High-Performance Instrument Aircraft Platform for Environmental Research - HIAPER (www.hiaper.ucar.edu/); Large-scale projects in which NOAO/NSO are partners (Advanced Technology Solar Telescope - www.atst.nso.edu Large Synoptic Survey Telescope - www.lsst.org Thirty Meter Telescope - www.aura-nio.noao.edu/)

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Each year, performance indicators that demonstrate progress toward achieving the long-term goals of the facilities operated and managed by the FFRDCs are delineated in the annual performance budget. There is also an annual cost and schedule goal for construction and upgrade of facilities and an annual goal related to facility operations. In the beginning of the fiscal year, FFRDCs submit annual program plans and at the end of the year submit accomplishment reports which form the basis for GPRA nuggets highlighting the activities in scientific research and engineering as well as education and outreach activities.

Evidence: NSF's FFRDCs have specific annual performance measures that demonstrate progress toward achieving the long-term goals of the program are found in the Measures Tab.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines for the annual performance measures are obtained from internal NSF sources and ambitious targets, commensurate with the budget environment, are set under the 'Measures' tab. In addition to program measures, FFRDCs set performance targets in their annual program plans that outline ambitious targets for scientific research, engineering, technology assessment, education and outreach activities.

Evidence: Evidence of NSF's FFRDCs baselines and ambitious targets for its annual measures can be found in the Measures Tab; NSF's Enterprise Information System; FY 2004 AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); and the FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par).

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Each FFRDC engages a multitude of partners that commit to and work toward the goals of the Center. Purpose, responsibilities, and requirements for all partners are spelled out in Cooperative Agreements for which four the FFRDCs are funded or, in the case of the fifth (STPI), in the contract for the management and operation of the Center. The terms of these Cooperative Agreements and the STPI contract explicitly and/or implicitly define the responsibilities, goals and expectations of the Awardee and NSF. In addition, these Cooperative Agreements and the STPI contract specifically require annual program plans which set goals for research, construction/upgrade and operations, as relevant. Memoranda of Understanding (MOUs) and Memoranda of Agreement (MOAs) exist between NSF and partnering organizations as well as between the FFRDCs and partners.

Evidence: Evidence demonstrating that all partners commit to and work toward the annual and/or long-term goals of the program is found in awardee program plans. Annual program and/or stragetic plans can be found at: www.noao.edu/dir/reports_archive.html (NOAO); www.nso.edu/general/docs/ (NSO); and NCAR www.ncar.ucar.edu/stratplan/plan.pdf). Awardee project reports are reviewed and approved on an individual basis prior to release of additional funds for the project. Other evidence may be found in NSF's Grant General Conditions (e.g. Article 9 of the NSF-UCAR Cooperative agreement states in part "the Awardee shall provide an Annual Scientific Report that describes the scientific and facilities programs conducted by NCAR staff, NCAR visitors and community participants in the preceding fiscal year.") (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf98gc1a). Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations are conducted regularly at multiple levels in order to inform program improvements and influence program planning. Each activity at NSF is reviewed once every three years by a COV. Advisory Committees review and approve COV reports. Since FY 2002, the Advisory Committee for GPRA Performance Assessment makes use of COV reports in its assessment of performance for each Tools indicator applicable to FFRDC facilities on an NSF-wide basis. NSF conducts workshops and various aspects of the FFRDCs have been reviewed by external entities such as the NAS. NSF staff and external experts conduct site visits at NSF-supported FFRDCs. All these activities inform NSF senior management and contribute to development of plans for the agency. NOTE: The weight of this question has been increased to reflect the importance NSF places on the conduct of independent evaluations to support program improvements and evaluate effectiveness.

Evidence: Evidence demonstrating that independent evaluations of NSF's FFRDCs are of sufficient scope and quality to evaluate the program's effectiveness can be found in: Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity GAO-03-454 May 2, 2003 (www.gao.gov/new.items/d03454.pdf); COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc) Advisory Committee reports, including the Advisory Committee for GPRA Assessment (AC/GPA) Report www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The NSF FY 2006 Budget Request was built around the Administration's R&D Criteria, thereby highlighting specific performance information for NSF's investment portfolio. The budget also clearly presents the resource request and outlines the activities that will be supported with the funds. NSF's performance structure provides the underlying framework for NSF's annual budget request. Each major NSF organization - i.e. the directorates - ties its budget directly to NSF's performance framework. This is further documented in the performance summary included in each organization's chapter of the budget, which ties the budget directly to the PART assessments. NSF also displays the full budgetary cost associated with the new program framework defined in the NSF Strategic Plan FY 2003-2008 in the Budget Request. With respect to presenting the resource needs in a clear and transparent presentation, the NSF budget displays resource requests by structural component and by performance goal. This presentation is based on consultations over the past year with key Congressional and OMB staff, and it also incorporates recommendations from the 2004 report on NSF by the National Academy of Public Administration. The purpose of this presentation is to highlight the matrix structure that NSF employs, with the major organizational units each contributing to the goals and investment categories established in the NSF Strategic Plan. This revised presentation contains additional information on the portfolio of investments maintained across NSF.

Evidence: The annual NSF Budget Request to Congress presents the long-term goals of the FFRDC program and the resources needed in a complete and transparent manner. Additional information may be found in the NSF Strategic Plan FY 2003-2008 (www.nsf.gov/od/gpra/Strategic_Plan/FY2003-2008.pdf); NSF Budget Cost Performance Integration Plan (Internal Document); and National Science Foundation Governance and Management for the Future, National Academy of Public Administrators. Evidence showing that budget requests are tied to accomplishment of the annual and long-term goals of the FFRDC program include the NSF Budget Cost Performance Integration Plan (Internal Document) and National Science Foundation Governance and Management for the Future, National Academy of Public Administrators, Order Number 04-07 (www.napawash.org/resources/news/news_4_28_04.html). Evidence specific to FFRDCs can be found in the following documents: National Academy of Sciences study, "Strategic Guidance for NSF's Support of the Atmospheric Sciences" (www4.nas.edu/webcr.nsf/ProjectScopeDisplay/BASC-U-03-03-A?OpenDocument); and Astronomy and Astrophysics Decadal Survey, "Astronomy and Physics in the New Millenium" (www.nap.edu/books/0309070317/html/). Detailed plans for FFRDC projects are included in the FY 2006 Budget Request to Congress (www.nsf.gov/bfa/bud/fy2006/toc/htm). These details include Principal Scientific Goals, Principal Education Goals, Partnerships and Connections to Industry, Associated Research and Education Activities, Management and Oversight, Current Project Status, Funding Profile, Renewal or Termination, and Science Support. Full budgetary costing for the Tools Strategic Goal is included in the FY 2006 Budget Request to Congress. Also relevant are Site Visit reports and Annual Reports.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Although deficiencies per se have not yet been noted, the Committees of Visitors (COV) process and Directorate Advisory Committees (ACs) provide valuable constructive feedback on an ongoing basis concerning areas where strategic planning can be strengthened, and in response the agency takes steps to address those issues. The FFRDCs are guided by strategic and other planning documents. NSF solicits public feedback on the agency's goals and planning processes as part of each independent (external) assessment of agency activities which include FFRDCs. Nevertheless, NSF has in place steps to address specific weaknesses, if identified.

Evidence: Evidence showing that NSF's FFRDCs strategic planning activities are effective include: COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp); 2005 COV report for the Division of Astronomical Sciences (www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf); 2003 COV report to the Advisory Committee of the Geosciences Directorate (www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); NSF Strategic Plan FY 2003 - FY 2008 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); and Directorate Advisory Committee minutes (for example, www.nsf.gov/mps/advisory.jsp and www.nsf.gov/geo/advisory.jsp).

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: Cost/benefit analysis and risk management are aspects of the planning and decision-making processes when FFRDC facility investments are considered. Alternative approaches, including cost and risk and utility for research, are considered in advance of project initiation. The FFRDCs conduct research and development to support these choices and the decision-making process. This can involve supporting research and/or facilities in a university context rather than a federal laboratory, because the university environment might offer better educational opportunities than would be available at a federal laboratory. For example, the Division of Atmospheric Sciences de-commissioned an aircraft at NCAR and began to sponsor a similar aircraft at the University of Wyoming because, in this particular case, NSF could better achieve its strategic goals by supporting a facility at a university. Design studies examine tradeoffs between different concepts, such as selection of alternate sites (e.g., ALMA, a telescope coordinated by NRAO), or technical and performance capabilities of commercially available products (e.g. the NCAR-operated research aircraft HIAPER). These same criteria are used in other decisions affecting the FFRDCs, such as the competition for the management. The Advisory Committee for the Office of Integrative Activities conducted an external panel review of the Science and Technology Policy Institute in August 2002. The panel examined the continuing need for STPI and alternatives to the FFRDC; operations and performance; support structure; and the balance of work for its primary client, the Office of Science and Technology Policy, with work for other Federal agencies. Based on the review, the panel recommended "NSF re-compete the Institute contact." The Office of the Director accepted the recommendation and a re-competition was held in 2003.

Evidence: NSF currently is conducting a full and open competition of the management of NCAR. A final decision among the bidders will be based on all the criteria listed in the question 2 CA1 and additional criteria pertinent to NCAR. The management of each of the other FFRDCs has similarly been recompeted during the past 5 years, or is scheduled for recompetition in the near future. Recent and planned management reviews and recompetitions include: NOAO/NSO - management recompeted in FY01-02, management review in FY05; NRAO - management review FY04; NAIC - management recompeted FY04-05; NCAR - management reviews FY01 and FY06, management recompeted FY06 - 08; and STPI - management recompeted FY03. See also the National Academy of Sciences study, "Strategic Guidance for NSF's Support of the Atmospheric Sciences" (http://www4.nas.edu/webcr.nsf/ProjectScopeDisplay/BASC-U-03-03-A?OpenDocument); Astronomy and Astrophysics Decadal Survey, "Astronomy and Physics in the New Millenium" (http://www.nap.edu/books/0309070317/html/); Advisory Committee for the Office of Integrative Activities, NSF Review Panel on the Science and Technology Policy Institute, Final Report, August 12, 2002 (internal report); and Astronomy and Astrophysics Decadal Survey, "Astronomy and Physics in the New Millenium" (http://www.nap.edu/books/0309070317/html/).

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Divisions that sponsor the FFRDCs frequently assess how best to meet the needs of their scientific communities. Prior to initiating new activities at the FFRDCs, workshops and external reviews are typically conducted to ensure that scientific opportunities justify the facility expenditure, and that supporting the acquisition and operation of infrastructure is the most efficient method of facilitating the science in question. In addition, research projects undertaken by the FFRDCs are examined to ensure that they could not be done in a university setting, but require a critical mass of expertise and/or facilities that is not available within the university environment. Through these efforts, NSF's FFRDCs promote coordination of basic research tools that benefits the entire scientific community. With respect to large and midsize facility projects undertaken at the FFRDCs, NSF senior management reviews and compares opportunities of competing projects and forwards recommendations to the NSB for subsequent review and approval. Interagency and international agreements and understandings are active and on file for most facilities projects, demonstrating the commitment of NSF to efficient and effective coordination of efforts. External and internal reviews of FFRDCs address the question of the benefits for the community of activities at the Centers. Regarding the potential benefits of user facilities and other resources the FFRDCs provide relative to others at NSF or other agencies, NSF resources are planned and coordinated as part of a portfolio of tools available to the scientific community. For example, NSF's ground-based telescopes are planned and operated in a way that complements and permits synthesis with NASA's astronomy resources. The Decadal Survey helps prioritize new astronomy facilities, and interagency coordinating committees help plan their construction and coordinate their use.

Evidence: Recent and planned management reviews and recompetitions which assess and compare the potential benefits of efforts within the program to other efforts that have similar goals include: NOAO/NSO - management recompeted in FY01-02, management review in FY05; NRAO - management review FY04; NAIC Management recompeted FY04-05; NCAR - management reviews FY01 and FY06, management recompeted FY06 - 08. Also relevant are: Review of the science, facilities, and management of NCAR and UCAR (www.ncar.ucar.edu/review01/); and Astronomy and Astrophysics Decadal Survey, "Astronomy and Physics in the New Millenium" (www.nap.edu/books/0309070317/html/).

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Budget requests and funding decisions are guided both by internal decision-making and periodic external reviews carried out by bodies such as the National Academy of Sciences. Although the FFRDCs are funded for five-year periods, annual program plans are submitted to and reviewed by NSF. These program plans are based on strategic plans for the Centers, a prioritization process within the FFRDCs which is guided by NSF program staff interactions, and priority guidance and budget allocations resulting from Congressional appropriation. Several examples of using a prioritization process specific to NSF's FFRDCs are cited in the Evidence section. With respect to major facilities that might be supported and managed by the FFRDCs under the Major Research Equipment and Facilities Construction account, NSF has a documented prioritization process. For example, the MREFC Guidelines have been updated over the past year, and the Guidelines will continue to be a living document. Prioritization is reflected in the FY 2006 Budget Request and internal funding decisions. Priorities for MREFC were explicitly provided in the FY 2006 Budget Request, as was a discussion of the process. Other facility investments are prioritized utilizing workshops, community-based planning efforts, and with advice from established Advisory Committees. In addition, external groups such as the National Academies provide prioritized recommendations. As an example, the Division of Atmospheric Sciences and NCAR increased funding for the HIAPER related activities in FY 2005 and beyond to support the transition from construction to full operation (FY 2006). Funds were directed away from other activities within the Division and at NCAR to support the mutually agreed to highest priority, i.e. the full operation of HIAPER.

Evidence: NSF's FFRDCs are part of the agency's prioritization process which guides budget requests and funding decisions. For the atmosphereic sciences evidence inlcudes the National Academy of Sciences study, "Strategic Guidance for NSF's Support of the Atmospheric Sciences" (www4.nas.edu/webcr.nsf/ProjectScopeDisplay/BASC-U-03-03-A?OpenDocument). The nation's astronomical community convenes panels of experts every ten years to establish priorities for the coming decade. These "decadal surveys" provide guidance to NSF in technology development and prioritization of new, moderate and large initiatives. The most recent is "Astronomy and Physics in the New Millenium" (www.nap.edu/books/0309070317/html/). Projects endorsed by this process in which NSF is making investment include the National Virtual Observatory, the Telescope System Instrumentation Program, and the Large Synoptic Survey Telescope. NCAR continues scientific, educational, and service-related efforts in many of the high-priority areas of the strategic plan. These include activities in weather and climate modeling, biogeosciences, the water cycle across scales, data assimilation, coronal magnetic fields and space weather modeling, climate and weather assessment science, and wildfire research. These also include new efforts in upper troposphere and lower stratosphere water vapor, ozone, clouds and their controlling processes using new airborne and spaceborne observational capabilities and the examination of the export of air pollutants from urban to regional and global environments. The newly released Community Climate System Model (CCSM-3) will continue to be used for production runs in support of the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment. NCAR will also make significant progress in FY 2005 in the research infrastructure development of NSF's new high-altitude aircraft, the High-Performance Instrumented Airborne Platform for Environmental Research (HIAPER) and begin instrumentation development for the new aircraft. See also the NSF Strategic Plan FY 2003 - FY 2008 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201), Congressional Justifications (www.nsf.gov/about/budget/).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: NSF programs regularly collect timely, high-quality performance data relating to key program goals through Committee of Visitors reports and other mechanisms, and use this information to make corrective actions, adjust program priorities, make decisions on resource allocations and make other adjustments in management actions. GPRA and PART performance data are verified and validated by an independent, external consulting firm. All FFRDCs provide progress reports on operations and significant science outcomes. Program Officers monitor and collect information through weekly to monthly scheduled meetings and teleconferences with FFRDC managers and appropriate financial, managerial, and scientific staff. In addition, Program Officers partner with the staff in the NSF Division of Contracts and Complex Agreements (DCCA) to provide monitoring and oversight to administrative functions of the FFRDCs. This oversight provides current and timely performance information that is meaningful to NSF program management. In agency construction programs, collection of performance data and monitoring can occur as frequently as daily. NSF collection is accomplished through formal channels of communication with interagency and/or international partners through weekly, quarterly, semiannual or annual reviews.

Evidence: Data and information showing that NSF regularly collects timely and credible performance information are included in: COV Reports (list of all reports and responses - http://www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - http://www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate http://www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); AC reports, including the AC/GPA report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); Performance Facilities Reporting System (https://www.fastlane.nsf.gov/documents/fprs/fprs_main_intro.jsp); and NCAR Annual Scientific Reports (http://www.asr.ucar.edu/); NOAO annual report (http://www.noao.edu/dir/reports_archive.html); NSO annual report (http://www.nso.edu/general/docs/).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The NSF program officers who provide oversight to the NSF FFRDCs hold the Centers accountable for performance, cost, and schedule with respect to major construction projects as well as the on-going operations. NSF's contracts and Cooperative Agreements specify expected business processes and practices that are consistent with FAR requirements (for contracts only) and OMB guidelines as well as cost, schedule and performance results with respect to specific projects. NSF Program Officers in partnership with DCCA monitor compliance with appropriate FAR & OMB regulations and guidelines, cost, schedule and technical performance and take corrective action when necessary. In the case of significant construction or procurement projects, NSF has established policies and procedures that require program managers to report to senior management all deviations on cost and schedule. Deviations on costs that are greater than 10% must also be reported to the National Science Board for its consideration. Agreements can be terminated or not renewed in cases where the awardee is unable to meet the terms of the award instrument. In the most recent example, in 2002 NSF conducted the second five-year review of the use and need for STPI, as required by the FAR. Of the four criteria employed to evaluate the Institute's management and operations, two of the criteria specifically addressed cost, schedule, and performance: (1) "the efficiency and effectiveness of the FFRDC in meeting the Government's needs, including an assessment of the quality and timeliness of work performed" and (2) "the adequacy of the FFRDC management in assuring a cost-effective operation." The Advisory Committee for the Office of Integrative Activities concluded that while the FFRDC was the most advantageous way to respond to requests for analytic support by OSTP and other Federal agencies, OSTP was not satisfied with the overall performance (e.g., technical quality, timeliness, responsiveness, and usefulness) of STPI. This finding led to the recompetition for STPI.

Evidence: Data and information regarding accountability for cost, schedule and performance results are included in cooperative agreements or contracts. Other evidence includes: Annual performance evaluation of NSF employees/program officers; COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); Internal report, "NSF Review Panel on the Science and Technology Policy - Final Report," Advisory Committee for the office of Integrative Activities, August 12, 2002; Awardee project reports; NSF Grant General Conditions (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf98gc1a); and Federal Cash Transaction Reports.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NSF routinely obligates its funds in a timely manner, and they are monitored to assure they are spent for the intended purposes; funds appropriated for FFRDCs are mostly two-year funds, but NSF obligates over 99% of this support within the first year it is appropriated. NSF also has pre- and post-award internal controls to reduce the risk of improper payments. Beginning in FY 2004 NSF has incorporated erroneous payments testing of awardees into its on-site monitoring program. This will provide NSF with more information about the usage of NSF funding by awardees.

Evidence: Data and information demonstrating that funds are obligated in a timely manner and spent for intended purpose are included in the PriceWaterhouse Coopers NSF FY 2001 Risk Assessment for Erroneous Payments; Data on NSF Carryover, presented in the NSF Budget Request to Congress (www.nsf.gov/about/budget/); Risk Assessment and Award Monitoring Guide; Clean opinion on financial statements for past 7 years; and Federal Cash Transaction Reports.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Funding allocations to the FFDRCs cover the cost of direct and indirect expenses and NSF works with the management of the FFRDCs to keep the overhead costs as low as possible through more effective a variety of techniques such as use of technology, outsourcing certain activities, and comparison of indirect expenses with those of universities and comparable institutions. The incentive for NSF and the FFDRCs is that for every dollar saved in overhead is a dollar applied to research, provision of facilities, and educational activities. In most cases, NSF's FFRDCs use the FAR regulations as guidance for acquisitions goods and services. These regulations provide a framework for obtaining the best value for the government with respect to acquisitions. In instances where facility capability may be commercially available, cost comparisons, including lease/purchase analysis per OMB A-94, are conducted to determine the most efficient and effective method of providing the required capability. As a result, NSF awardees employ a number of acquisition strategies, including direct purchase/construction, lease, tax-exempt bonds, and fixed-duration contract in providing facility services.

Evidence: Procedures to measure and achieve efficiencies and cost effectiveness in program execution are shown in the following documents: Measures tab; FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf; 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); UCAR business system review by NSF (internal document); NSF FY 2003-2008 Strategic Plan (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); and NSF Grant Proposal Guide (www.nsf.gov/publications/pub_summ.jsp?ods_key=gpg).

Does the program collaborate and coordinate effectively with related programs?

Explanation: NSF's FFRDCs are the agency's strategic partners, and as such, support the agency's strategic plan by embracing collaboration and coordination nationally and internationally to achieve our goals. The complex mix of science themes, cutting edge construction projects, and efficient and effective operations enabled through NSF's FFRDCs are coordinated with other federal agencies as well as with international partners. For example, the Atacama Large Millimeter Array (ALMA) is a collaborative international partnership under construction in the Atacama Desert in Chile. The U.S. ALMA effort is coordinated through the National Radio Astronomy Observatory, an FFRDC. NRAO International partners are the European Southern Observatory and Japan. Atmospheric Sciences is global in nature and requires global collaborations to achieve NSF strategic goals. Another example is STPI. While OSTP is STPI's primary client, STPI conducts analytic studies for other federal agencies through the Government Economy Act. The interagency agreement provides a mechanism for agencies to gain access to STPI's network of experts and request studies relevant to science and engineering policy. For example, STPI and Principal Investigators with projects supported by the NSF Computer and Information Science and Engineering (CISE) Directorate are involved in a study of Transformative Research commissioned by the National Science Board.

Evidence: Data and information demonstrating that NSF's FFRDCs collaborate and coordinate effectively with related programs are documented in each FFRDC website (www.naic.edu/, www.joss.ucar.edu/, and www.ctio.noao.edu/). Additional information is included in management plans; internal administrative manuals; program solicitations (www.nsf.gov/funding/); STPI Task and subtask orders, reports and public presentations. For example, NRAO has worked extensively with the European Southern Observatory (ESO) and with the Joint ALMA Office (JAO) in overseeing construction of up to 32 antennas for the Atacama Large Millimeter Array as documented in this press release (www.aui.edu/press/2005jul11.html).

Does the program use strong financial management practices?

Explanation: The strong financial management practices of this program are consistent with the strong financial management practices used at the agency level and across all of NSF's programs. NSF was the first federal agency to receive a "green light" for financial management on the President's Management Agenda scorecard. NSF has received a clean opinion on its financial audits for the last seven years. The NSF is committed to providing quality financial management to all its stakeholders. It honors that commitment by preparing annual financial statements in conformity with generally accepted accounting principles in the U.S. and then subjecting the statement to independent audits. As a federal agency, NSF prepares the following annual financial statements: Balance Sheet, Statement of Net Cost, Statement of Changes in Net Position, Statement of Budgetary Resources, and Statement of Financing. Supplementary statements are also prepared including Budgetary Resources by Major Accounts, Intra-governmental Balances, Deferred Maintenance, and Stewardship Investments.

Evidence: Data and information demonstrating strong financial management practices are included in the Executive Branch Management Scorecard (www.whitehouse.gov/results/agenda/scorecard.html); results of NSF financial audits; and Performance and management assessments (www.whitehouse.gov/omb/).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Reviews of the FFRDC programs by Committees of Visitors (COVs) have not revealed any major deficiencies, but have provided feedback on programmatic and management-related concerns with respect to the ongoing challenges facing the programs. In addition, the Foundation conducts an annual review to assess administrative and financial systems and procedures to ensure that effective management controls are in place and that any deficiencies are identified and addressed.

Evidence: Data and information regarding management of NSF's programs are included in the NSF FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); Booz Allen Hamilton analysis; COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); Annual report to senior management; and Inspector General (IG) reports including its annual letter on management challenges and reforms and NSF responses (www.nsf.gov/oig/pubs.jsp).

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: NSF requires that all construction projects are managed using annual cost and schedule goals as well as through "earned value". Construction and service acquisition projects that are managed by NSF's FFRDC and overseen by the Program Official follow all NSF guidelines as specified in Large Facility Projects Management and Oversight Plan. Prior to facilities becoming widely available (full operation), a transition plan is developed and approved by the Program Official. This plan contains expected performance of the facilities, the process for access by the science community, and estimated operation and maintenance costs.

Evidence: Evidence that NSF's FFRDCs are managed with clearly defined deliverables may be found in the FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); management plans; internal administrative manuals; STPI Task and subtask orders; Booz Allen Hamilton analysis; and COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc).

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: NSF FFRDC managing organizations are selected by a competitive process which uses merit review. The continuation of support is based on a merit-reviewed proposal. As a result of NSB guidance to periodically re-compete facility grants, NSF considers whether an expiring grant should be re-competed. An example of this is the recent STPI recompetition that resulted in new managing organization.

Evidence: Information demonstrating that awards to NSF's FFRDCs are based on a clear competitive process can be found in the following documents: NSF FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); NSB Policy on Recompetition (www.nsf.gov/nsb/documents/1997/nsb97224/nsb97224.txt); Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf); and NSF Merit Review Criteria (www.nsf.gov/pubs/1999/nsf99172/nsf99172.htm).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: All FFRDCs provide annual or more frequent progress reports on operations and significant science outcomes. Program Officers monitor and collect information through weekly to monthly scheduled meetings and teleconferences with FFRDC managers and appropriate financial, managerial, and scientific staff. In addition, Program Officers partner with the staff in the NSF Division of Contracts and Complex Agreements to provide monitoring and oversight to administrative functions of the FFRDCs. This oversight provides current and timely performance information that is meaningful to NSF program management. In construction programs, collection of performance data and monitoring can occur as frequently as daily.

Evidence: Evidence that NSF's FFRDCs have oversight practices that provide sufficient knowledge of grantee activities are found in: COV Reports (list of all reports and responses - http://www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - http://www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate http://www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); awardee project reports; Report to the NSB on the NSF Merit Review Process - FY 2004 (http://www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf); Risk Assessment and Award Monitoring Guide; Clean audit opinions; President's Management Agenda (PMA) Scorecard for Financial Management (http://www.whitehouse.gov/results/agenda/scorecard.html); Site visit reports; Workshops and grantee meetings as presented in the annual Performance and Accountability report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=par); NOAO annual report (http://www.noao.edu/dir/reports_archive.html); NSO annual report (http://www.nso.edu/general/docs/) AC reports, including the AC/GPA report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); Performance Facilities Reporting System (https://www.fastlane.nsf.gov/documents/fprs/fprs_main_intro.jsp); NCAR Annual Scientific Reports (http://www.asr.ucar.edu/); and UCAR/NCAR News reports (http://www.ucar.edu/news/)

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Annual performance data on FFRDC facilities construction and operations are made available through past GPRA Performance Reports and the Performance and Accountability Report. These reports are publicly available. In addition, the larger FFRDCs produce annual reports summarizing their science and facility accomplishments. These reports are posted on their web sites and are intended for the public.

Evidence: Evidence demonstrating that the NSF FFRDCs collect grantee performance data on and annual basis and make it available to the public include: FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); FY 2006 Budget Request (www.nsf.gov/about/budget/); NSF Grant General Conditions (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf98gc1a); Highlights of annual meetings/grantees meetings; workshops; AC/GPA annual reports www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216; awards database (www.nsf.gov/awardsearch/); news releases (www.nsf.gov/news/news_list.cfm?nt=2); NOAO annual report (www.noao.edu/dir/reports_archive.html); NSO annual report (www.nso.edu/general/docs/); News sites (www.noao.edu/news/, www.nso.edu/press/); NCAR Annual Scientific Reports www.asr.ucar.edu/; UCAR/NCAR News reports www.ucar.edu/news/

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The Federal Acquisition Regulations require the NSF to conduct comprehensive five-year reviews of its FFRDCs for renewal or re-competition. The review (1) assesses the Government's special technical needs and mission requirements to determine if and at what level an individual FFRDC continues to exist; (2) considers alternative sources to meet the Government's needs; (3) evaluates the efficiency and effectiveness of the FFRDC in meeting the Government's needs, including as assessment of the quality and timeliness of work performed; and (4) assesses the adequacy of the FFRDC management in assuring a cost-effective operation. For example, in 2002, NSF convened an external panel of experts to conduct the required five-year comprehensive review of STPI. The committee consisted of experts from government agencies and laboratories, international scientific societies, academic institutions, business and industry and representatives from OSTP and NSF as ex-officio members. Based upon the review of background materials and discussions with OSTP and NSF staff, the committee wrote an internal report of its findings to the Director of the NSF and recommended the re-competition of STPI. The Office of the Director provided funds and resources for the external review of STPI. FFRDCs are allocated funds competitively ("merit reviewed scientific research with limited competitive selection", per A-11).

Evidence: Evidence that NSF's FFRDCs allocate funds and use management processes that maintain program quality are found in: COV Reports (list of all reports and responses - http://www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - http://www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate http://www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); awardee project reports; Risk Assessment and Award Monitoring Guide; President's Management Agenda (PMA) Scorecard for Financial Management (http://www.whitehouse.gov/results/agenda/scorecard.html); Site visit reports; NOAO annual report (http://www.noao.edu/dir/reports_archive.html); NSO annual report (http://www.nso.edu/general/docs/) AC reports, including the AC/GPA report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); and Performance Facilities Reporting System (https://www.fastlane.nsf.gov/documents/fprs/fprs_main_intro.jsp).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The NSF FFRDCs have demonstrated good progress in achieving their long-term performance goals. These long-term goals are directly related to assisting NSF in achieving its strategic goals. NSF relies on external evaluation to determine whether it is achieving its long-term objectives. The external Advisory Committee for GPRA Performance Assessment found "significant achievement" under each of the following objectives: ??Expand opportunities for U.S. researchers, educators, and students at all levels to access state-of the-art S&E facilities, tools, databases, and other infrastructure. ??Provide leadership in the development, construction, and operation of major, next-generation facilities and other large research and education platforms. ??Provide for the collection and analysis of the scientific and technical resources of the U.S. and other nations to inform policy formulation and resource allocation. ??Support research that advances instrument technology and leads to the development of next-generation research and education tools. These evaluations provide further evidence that the programs are demonstrating adequate progress in achieving their long-term performance goals.

Evidence: Evidence showing that NSF's FFRDCs have demonstrated adequate progress in achieving its long term performance goals may be found in Measures Tab; AC/GPA report www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216; awardee project reports; and Performance and Accountability reports (www.nsf.gov/publications/pub_summ.jsp?ods_key=par)

Does the program (including program partners) achieve its annual performance goals?

Explanation: FFRDCs demonstrated achievement during the last two years using two metrics of evaluation for FFRDCs. These are: 1) Percent of construction acquisition and upgrade projects with negative cost and schedule variances of less that 10% of the approved project plan, 2) Percent of operational facilities that keep scheduled operating time lost to less than 10%. As the number of construction projects dropped to one in 2005, FFRDCs will be measured annually against operating time until and unless additional construction projects are started.

Evidence: Evidence demonstrating that NSF's FFRDCs achieve their annual performance goals may be found in Measures tab COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf, 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); NSF FY 2003-2008 Strategic Plan (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); and FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par).

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Facilities are improving efficiencies through development of instrumentation making use of state-of-the-art technology to provide greater data gathering capabilities, including more efficient use of equipment, improved transmission rates, and greater utilization (e.g., "plug compatible instrumentation for HIAPER). Upgrades to facilities provide improved technologies and enable more efficient operations. For example, scheduling of telescopes to carry out long-term observations is accomplished using queue-scheduling, a scheduling technique that significantly enhances efficiency of use of telescopes. Similarly, the development of scheduling software for community access to supercomputer resources has resulted in keeping the "turn-around" time for most computer jobs to tens of minutes.

Evidence: Evidence demonstrating improved efficiencies in achieving the goals of NSF's FFRDCs each year are shown in the following: Performance and Accountability reports (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); Measures tab; NSF Budget Justification (www.nsf.gov/about/budget/); and Committee of Visitor reports (www.nsf.gov/about/performance/advisory.jsp).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No other FFRDCs in the government can perform the services needed by the NSF-supported community as well as the NSF FFRDCs. NSF has chosen to support its strategic goals through a variety of modes of support including, but not limited to, individual grants, Science and Technology Centers, and FFRDCs. Through the peer review process, COVs and analysis of funding by strategic goals, NSF adjusts the balance of the modes of support to ensure that the agency is making the most effective investments to accomplish its strategic goals. The capabilities supported through the mix of funding modes, include, but are not limited to, provision of state-of-the-art tools for community use, contributions to a globally competitive workforce, and intellectual milieu for discoveries and the creation of new knowledge. With respect to the FFRDCs, statistics on the number of student, visitors, researchers, and policy makers served are tracked each year and publications connected with the FFRDCs are also used to monitor performance of the Centers.

Evidence: Evidence showing that NSF's FFRDCs compare favorably to other programs with similar purpose and goals can be found in: FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf; 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc); Master List of US Government FFRDCs (www.nsf.gov/statistics/nsf05306/); and AC/GPA report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Endemic to the oversight of NSF's FFRDCs are extensive independent evaluations. The FFRDCs are evaluated and guided by advisory committees, facility allocation committees, extensive peer review of the service to the community, quality and innovation of science, the effectiveness of the management and leadership role the Centers take with respect to the communities they serve, NSF COVs, AC/GPRA, and competition for the management of the Centers. NOTE: The weight of this question has been increased to reflect the importance NSF places on the conduct of independent evaluations to support program improvements and evaluate effectiveness.

Evidence: Evidence indicating that the independent evaluations of NSF's FFRDCs are of sufficient scope and quality indicate that the program is effective and achieving results may be found in: AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); awardee project reports; COV Reports (list of all reports and responses - www.nsf.gov/about/performance/advisory.jsp; 2005 COV report for the Division of Astronomical Sciences - www.nsf.gov/od/oia/activities/cov/mps/2005/ASTcov.pdf; 2003 COV report to the Advisory Committee of the Geosciences Directorate www.nsf.gov/geo/adgeo/advcomm/fy2003_cov/ATM_ULAFOS_2003_COV_report.doc).

Were program goals achieved within budgeted costs and established schedules?

Explanation: Cost and schedule information is collected for major NSF facilities. All of the FFRDC construction projects reported costs and schedules within 10% of the approved project plan.

Evidence: Evidence that the vast majority of the FFRDC program goals were achieved within budgeted costs and established schedules may be found in the FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par).