Detailed Information on the
Natural Resource Base & Environment (Grants) Assessment

Improving the focus of grant recipient reporting on outcomes.

Increasing planning and coordination with the Agricultural Research Service regarding the collection of stakeholder input.

Enhancing the tracking of measures in the budget justification, as well as the use of research and technologies.

Develop additional measures to show how much of the research is reaching users through extension activities.

Develop innovative ways of improving the efficiency of the grant awards process.

Develop a strategic plan for the portfolio in response to the panel evaluation and as guidance for the reallocation of resources.

Improve the efficiency in the grant process through the utiliztion of Grants.gov, the web-based Peer Review Syatem, and virtual panels where appropriate.

Measure: Cumulative number of Ecological-Economic models developed and used for management of invasive species

Explanation:Development and use of comprehensive interdisciplinary (ecological and economic) models critical to the assessment of management strategies related to priority invasive species on forest and range lands. No such integrated models currently exist, making it difficult to conduct meaningful cost-benefit analyses of either the threats of invasive species, or of the efficacy of prevention and mediation actions. It is anticipated that model development will occur in stages over four to five years.

Measure: Portfolio Review Score

Explanation:Goal performance is externally assessed on a 5 year basis to determine progress toward solving targeted national problems related to each Strategic Objective. The score reflects expert assessment on 3 PMA dimensions of relevance, quality and performance, along a continuum from exceeds expectations, to meets expectations, to needs improvement. The reported score is a mean of quantitative scores for the Goal''s objectives.

Measure: Cumulative Dollars Saved for Grant Review

Explanation:Dollars saved reflects the average salary saved by calculating the number of calendar days saved annually between receipt of proposal and date funding awarded for competitively reviewed proposals, then multiplied by the average daily salary for CSREES employees. The annual savings can be derived be comparing the change in cumulative savings between two consecutive years.

Measure: Internal Portfolio Review Score

Explanation:Goal performance is internally assessed annually by NPLs to determine progress toward solving targeted national problems related to the Objective. The score reflects expert assessment on 3 PMA dimensions of relevance, quality and performance, along a continuum from exceeds expectations, to meets expectations, to needs improvement. The reported score is a mean of quantitative scores for the Goal''s objectives.

Measure: Assessment and Control Technologies for Agricultural Emissions developed and used

Explanation:Number of assessments of priority and high consequence agriculture-related particulate, odor, and gaseous emissions control technologies for cost effective management approaches for regulators, commercial firms, and livestock and crop producers of varying scope and scale developed and used. Target and actual data is due in June 2006. Data will be provided at that time.

Measure: Proposal Review Time in Days

Explanation:Average number of calendar days between receipt of proposal and funding date funing awarded for competitively reviewed proposals. Average is calculated by summing the total number of days for all major competitively reviewed proposals and dividing by their total number.

Is the program purpose clear?

Explanation: The mission of the Department of Agriculture's programs for Goal 5 protect and enhance the nation's natural resource base and environment. The Cooperative State Research, Education & Extension Service (CSREES) funds grants for research, education and extension programs through coordinated work under 2 clearly defined portfolios: Management of Forest & Rangelands; Management of Soil, Air & Water to Enhance the Environment. Review panels found the education, research & extension with multiple funding authorities are appropriately integrated & well coordinated to improve management of forest & range lands & provide science-based knowledge & education; to improve the management of soil, air, & water to support production & enhance the environment.

Evidence: Responsibilities are authorized by: Food Security & Rural Investment Act of 2002; Research Facilities Act; Conference Report for Agricultural Risk Protection Act of 2000; Competitive, Special, & Facilities Research Grant Act; Agricultural Research, Extension, & Education Reform Act of 1998; Federal Agriculture Improvement & Reform Act of 1996; Food, Agriculture, Conservation, & Trade Act of 1990, Title XXV; National Nutrition Monitoring & Related Research Act of 1990; National Agricultural Research, Extension, & Teaching Policy Act Amendments of 1985; National Aquaculture Act of 1980; National Agricultural Research, Extension, & Teaching Policy Act of 1977; & the Rural Development Act of 1972. CSREES education, research & extension work for Goal 5, Protection & Safety of the Agricultural Food Supply, is clearly defined by the USDA Mission & Strategic Plan (2002-07), REE Mission Area & Strategic Plan (2003-08), CSREES Mission & Strategic Plan (2004-09).

Does the program address a specific and existing problem, interest, or need?

Explanation: CSREES grants for its goal 5 programs addresses the use and conservation of public and private natural resources & environment including soil, air, water, forests, rangeland, fish & wildlife through sustainable management. Through stakeholder involvement, issues have been identified: invasive species, drought preparedness & mitigation, water conservation, natural & human drivers of land-use & -cover change, evaluation of wildlife habitat, carbon sequestration, climate, eco/bio-diversity & fragmentation, fate/transport of atmospheric pollutants, new & alternate uses of ag. waste/byproducts, land conversion, decision support systems. See Healthy Forests Initiative for Wildfire Prevention & Stronger Communities www.whitehouse.gov/infocus/healthyforests/toc.html; Self-Review, 2005 Portfolio Review Natural Resources: Forest, Range, Fish, & Wildlife, 2005: II-19; National Drought Policy Act of 1998; EPA Regional Priority AFO Science Question Synthesis Document, 2004 manure.coafes.umn.edu/regulatory/1Introduction.pdf.

Evidence: Responsibilities are authorized by: Food Security & Rural Investment Act of 2002; Research Facilities Act; Conference Report for Agricultural Risk Protection Act of 2000; Competitive, Special, & Facilities Research Grant Act; Agricultural Research, Extension, & Education Reform Act of 1998; Federal Agriculture Improvement & Reform Act of 1996; Food, Agriculture, Conservation, & Trade Act of 1990, Title XXV; National Nutrition Monitoring & Related Research Act of 1990; National Agricultural Research, Extension, & Teaching Policy Act Amendments of 1985; National Aquaculture Act of 1980; National Agricultural Research, Extension, & Teaching Policy Act of 1977; & the Rural Development Act of 1972. CSREES education, research & extension work for Goal 5, Protection & Safety of the Agricultural Food Supply, is clearly defined by the USDA Mission & Strategic Plan (2002-07), REE Mission Area & Strategic Plan (2003-08), CSREES Mission & Strategic Plan (2004-09).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: CSREES' mission to integrate edu, research, & extension is coordinated with federal, state, local & private sector to avoid excessive overlap, redundancy or duplication. Program & funding diversity allow research & tech. transfer in response to emerging needs & long term work on causes of problems by adjusting mgmt. methods & technologies. Scope of work is integrated & complimentary within USDA (FS/NRCS/FSA/ARS/ERS), departments (DOI/EPA), states & private sector for inclusive, non-duplicative, coverage. Coordination adds value to multiple or complimentary solutions. External panels state some redundancy is needed- multiple solutions can solve problems quicker. National Program Leader (NPL) participation in review panels, federal interagency working groups, stakeholder workshops & NASULGC programs such as ECOP Forestry Task Force are important mechanisms for CSREES to identify emerging issues. AREERA formally requires that formula funded projects reflect stakeholder priorities.

Evidence: This is done through meetings with USDA research agencies (FS/ARS/ERS) seeking input for RFA's that complement & enhance the value of work, peer review panels, & participation & leadership in scientific societies. Including scientists from other agencies/departments with similar missions, & private sector/academy scientists complimentary to CSREES' focus helps ensure the highest quality priority proposals get funded & avoid inefficient, costly redundancy, duplication & improper focus. See Council for Ag. Sci. & Tech. www.cast-science.org/cast/src/cast_top.htm. Joint participation clarifies emerging issues.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Much of the funds provided are earmarks by Congress to fund special interests, & are not determined through a competitive review process. Optimally designed portfolios address needs of producers, growers, students, teachers, & professionals in natural resources & the environment. Funding is never awarded to inferior proposals & is deferred pending correction of deficiencies. This ensures that earmarks undergo the same process of aligning activities to strategic plan. Proposed budget adjustments include phased elimination of formula funded/earmarked research & exclusive focus on competitive grants, incl. NRI & a new competitive research program reasserting Agency discretion. To the extent possible, projects directed by Congress align with Strategic Goals & contribute to goal achievement. Program leaders work closely with earmark recipients to align in the proposal for & execution of projects.

Evidence: The 2006 budget request proposes no earmarks & replaces formula funds with more for the NRI (to $250 mil.) & $75 mil. competitive grants for regional, state, local priorities, with federal oversight. While earmarks are routinely eliminated in all budget requests, any subsequent earmarked projects are subjected to merit-based evaluations measured against standards established for all CSREES grants. NPL scientific expertise, independent review & stakeholder input address national needs. These reflect changes in the state of science & priorities for comprehensive, effective, efficient investment in protection of forest & range lands & natural resources. Peer review ensures relevance, quality & performance of activities. Integrated education, research & extension define science in monitoring, detection & mitigating natural resource & environmental threats.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: CSREES portfolios are targeted to concentrate research & education on enhancing the nation's natural resources & environment, including rangelands, forests, soil, air, water & biodiversity with focus on detecting & managing invasive species & mitigating agricultural based particulate emissions. A diverse & complicated combination of authorizations & funding streams target extramural education, research & extension at institutions with broad geographic coverage & includes resources for minority-serving & other underserved institutions & communities that target specific populations to reach intended beneficiaries. This ensures a comprehensive, continuous flow of science-based knowledge, technologies & management methods that are technically, scientifically & economically relevant to the diverse needs of multiple private & public constituencies including the workforce, researchers, educators, producers, consumers & private citizens.

Evidence: No other agency has the combined mission of education, research, & extension outreach spanning the classroom, laboratory & community. Recently completed Goal 5 portfolio reviews found a suitable process for determining the proper set of intended beneficiaries & the program is sufficiently flexible to adjust quickly as intended beneficiaries' needs & conditions change. Funding by CSREES through the IFAFS program led to the establishment of A Center for Subtropical Agroforestry. Its objectives are to provide relevant info. & tools regarding agroforestry practices, economics & funding opportunities for extension agents, natural resource professionals, educators & landowners. See McIntire-Stennins Cooperative Forestry Research Program, Accomplishments & Impact Statements 1999-2003.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: CSREES programs, including those for Goal 5, provide assistance entirely through grants related to research, education and extension. Grants provided through Goal 5 relate to enhancing the protection and safety of the nation's environment and natural resources. CSREES has a balanced strategy to deal with problems & topics to enhance rational & sustainable production & management practices for the nation's natural resource base & environment. Strategic Goal 5 has a limited number of quantifiable long term outcome based performance measures for the current decade that focus specifically on ensuring the long term viability of the nation's natural resource base & environment. Two of the measures, "assessment and control technologies for agricultural emissions," and "ecological-economic models for invasive species control and management" will measure the number of such technologies that are developed and used. The portfolio review score is a compilation of the scores of a number of programs that are not individually reflected in the PART.

Evidence: The overall long term performance measure established for Strategic Goal 5 is an aggregate 5-year Portfolio Review Score, with a goal of 80 for 2005. Additional specific long term outcome measures are (A) dealing with invasive species, & (B) air particulate pollution mitigation. These are consistent with priorities set by USDA, the President's Healthy Forests Initiative, see www.whitehouse.gov/infocus/healthyforests/, & Renewable Resources Extension Act, see laws.fws.gov/lawsdigest/renewre.html; www.csrees.usda.gov/business/awards/formula/renewableres.html. Both measures are baselined to 2002. See PART Measures tab; see CSREES Statement of Agency Goals & Objectives, Exhibit 4-1, 2005 Budget Submission.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The panel saw as ambitious the agency targets for Goal 5 over the next 5 years. The long term (2009) goal is to educate land owners & stakeholders knowledgeable about natural resource & environmental programs & management opportunities, especially regarding invasive species & agricultural particulate matter control. Identifying environmental & natural resource risks inherent in agricultural production strengthens agricultural competitiveness & maintains ecosystem & natural resource health. CSREES & its sibling agencies in the REE Mission Area, as well as Forest Service research agencies, aligned strategic plans with REE & USDA & coordinated similar long term objectives & problem areas to increase sustainability, value added, & returns to public education, research, & extension. The measures are considered to be ambitious due to the magnitude and complexity of the research necessary to achieve the desired results. These are mutliyear projects requiring participants from a range of scientific disciplines.

Evidence: The long term Strategic Goal 5 targets & time frames, through 2010, are (1) a 5% increase in the external expert portfolio review score; (2) development of comprehensive interdisciplinary (ecological & economic) models to assess management strategies related to invasive species on forest & range lands (there are currently no integrated models); & (3) assessments of agriculture-related particulate emissions control technologies (ranging from reduced tillage methodologies & techniques, to air management strategies for large Concentrated Animal Feeding Operations (CAFOs)) for environmentally & cost effective management approaches. The recently completed expert panel review judged long-term targets to be ambitious. See PART Measures tab; CSREES Exhibit 2-1, Statement of Agency Goals & Objectives; & FY 2005 Budget Submission.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: CSREES has specific annual performance & efficiency targets that measure progress toward achieving long term goals. Annual measures include output (quality, relevance & performance of agency funded education, research & extension work related to Strategic Goal 5, & efficiency (cost & time to review proposals & award funding to grantees). These measures & targets are consistent throughout the agency for all Strategic Goals & Objectives. Annually, the Agency conducts internal program assessments & the results will be provided to external panels to be convened every five years to demonstrate progress toward achieving the program's long-term goals.

Evidence: Annual performance of Strategic Goal 5 objectives is assessed to determine progress toward solving targeted national problems & achieving long term goals. Annual internal performance review scores for each objective reflect expert assessment of the PMA dimensions of relevance, quality & performance, on a continuum from exceeds expectations to meets expectations, to needs improvement on each of 14 categories. Efficiency measures monitor progress in reducing cost per proposal reviewed, & time per proposal processed, reviewed & awarded funding. Efficiency baselines for cost per proposal reviewed ($530) & mean time to review proposals & release funds (220) days were established in 2002 & revised in 2004 to reflect increased transportation & lodging costs & other variables beyond the control of the Agency, a 25% increase in honoraria, & the increased use of more effective but more costly on-site, rather than ad hoc, reviewers.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Following the completion of the USDA, REE & CSREES Strategic Plans, portfolios of work were established for each of five Strategic Objectives. Annual measures for performance & efficiency have been established. Baselines are established on recent performance (revised in 2004), & ambitious targets have been established to routinely & continuously improve performance, reduce costs, & save time. These targets are ambitious because they are being established for the first time ever & are expected to increase incrementally over the next 10 years (budget dependent). They seek to develop comprehensive interdisciplinary models critical to the assessment of management strategies related to priority invasive species on forest & rangelands. They also seek to develop control technologies for agricultural admissions. The recently completed expert panel review judged long-term targets to be ambitious. The measures are considered to be ambitious due to the magnitude and complexity of the research necessary to achieve the desired results. These are mutliyear projects requiring participants from a range of scientific disciplines.

Evidence: Annual performance scores for each objective are baselined at meets ambitious expectations. Annual performance targets call for a cumulative 5% improvement in exceeding expectations. Efficiency baselines for cost per proposal reviewed & mean time to review proposals & release funds days were established for 2002 & baselined in 2004. Cost per proposal reviewed reflects transportation & lodging costs & other variables beyond the control of the Agency, honoraria, & the increased use of more effective but more costly on-site, rather than ad hoc, reviewers; annual targets call for holding review & processing costs to a real increase of 2% & an average 2% reduction in review time. See PART Measures tab.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: CSREES enters into education, research, & extension partnerships only when mutual benefit is clearly demonstrated & it is consistent with CSREES mission, Strategic Plan & long term & annual goals. All submitted proposals & plans of work specify the scope of work, milestones, measures, outputs & outcomes, along with performance monitoring procedures. Submission of proposals & acceptance of awarded funds requires partners' contractual commitment to working to accomplish annual & long term goals.

Evidence: CSREES reviews >5000 proposals & funds >2000 proposals, cooperative agreements & other transactions yearly. All proposals/plans of work (including formula, special, earmarked, other mandated funding) must include measures relevant to long term & annual performance goals & targets. NPL review & monitoring of performance assures continued commitment of partners. Funds are withheld or deferred if the contracted commitment is not maintained. NPLs work together with peers in other agencies to develop RFA guidance that explicitly specifies long term & annual goals. Example of government partners working toward common goals: over the past 3 years Cooperative Extension Service partnering with Farm Service Agency built a mutually beneficial working relationship between government agencies in need of plant/animal training for new Farm Loan Officer Trainees.; www.joe.org/joe/2004april/ent.shtml#rb5. Also see PART Sec. 3 below for administrative procedures assuring contractual commitment.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Both Goal 5 portfolios are independently assessed & evaluated by panels of outside experts every 5 years, baselined in 2005. The assessments provide useful comments & recommendations to help NPLs & implementing partners generate meaningful outcomes over the long term. Impacts will be assessed both internally & externally on an annual basis. Other external reviews & evaluations of CSREES programs are encompassed in materials reviewed by the review panels in making their evaluations of the portfolios. The reviews have demonstrated that these portfolios were of high quality & of sufficient scope in generating results leading to solving critical needs of the nation.

Evidence: The results of independent expert portfolio review assessments for CSREES Goal 5, conducted in early 2005, are found in the Long Term Measures tab. Both Goal 5 portfolios were scored "exceeds expectations". Portfolio 5.1 received a score of 77, & Portfolio 5.2 was scored 81. An extensive evaluative research bibliography, largely in vetted, peer reviewed journals & government studies, verifies the effectiveness & topical relevance of publicly funded agricultural & natural resources R&D, including extension. See Bibliography of Agricultural Research & Extension Structure, Economic Impact, & Rate of Return Literature.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Budget & Performance Integration (BPI) process ensures a direct & continuous link between long range & annual goals, defined in the 3 Strategic Plans, & the annual Departmental Performance Plan, & REE Quarterly R&D performance reports submitted to the Under Secretary. Annual measures are in the process of being established but long term measures have already been established. See performance tab for LTM. Designation of explicit problem areas that define the portfolios are exclusive & non-duplicative, & make it possible to directly link budget lines with goals, objectives, portfolios, & problem areas in a transparent manner. However, the Budget justification does not clearly link the impact of funding decisions on performance measures.

Evidence: Annual & long-term Goal 5 performance goals & accomplishments were an explicit component of the FY 2006 President's Budget Request regarding natural resources & environment. See Agency budget requests, Explanatory Notes & accompanying information. New classification & reporting systems currently under development or in pilot testing will strengthen the budget & performance linkages & increase transparency in the future. The budget request eliminates earmarks & formula based research in favor of competitive programs to enhance capability & performance.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The portfolio review process identifies the opportunities for needed chnages on an annual basis through the use of core teams. Timetables are often based on the deadlines related to the issuances of RFA's (Requets For Applications) and are reviewed annually. The alignment of the CSREES programs in program areas that are consistent with the USDA research agencies and USDA's overall strategic plans is an example fo this process. Significant efforts were undertaken to fully align the new CSREES Strategic Plan with the REE & the USDA Strategic Plans. Comprehensive Problem Area codes have been developed for all agency work, & are being used to align areas of CSREES higher education, research, & extension work under the portfolios that support the strategic objectives. Budget Performance Integration (BPI) enhances agency budget request preparation, justification, & reporting based on need, strategic planning, & overall performance.

Evidence: CSREES demonstrates comprehensive linkages of both Goal 5 portfolios between congressional authorizations & appropriations, the Strategic Plans, the departmental & agency mission, the departmental Performance Plan, & specific areas of science & education. All 3 of the strategic plans are posted together on the agency web site at www.csrees.usda.gov/about/strat_plan.html. The CSREES OneSolution data management & reporting system comprehensively links strategic planning, Budget & Performance Integration (BPI) & output, outcome & impact reporting. The agency recently submitted legacy & major accomplishment expectations to the Secretary through the office of the Under Secretary.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Constituents, stakeholders & partners are involved in planning, development & review. NPLs consult with academic & industry scientists, USDA, advisory groups (National Ag. Research, Extension, Education & Economics Advisory Board; Council for Ag. Research, Extension & Teaching; Council for Ag. Science & Technology), & industry to assure complementarity with programs of others. Coordination with USDA, interagency work groups, other departments adds value & extends benefits of programs with similar goals. Portfolio review brought independent outside experts to review CSREES investments in research, extension & education progress toward national problems. Experts offered opinions & recommendations on new efforts to increase benefits. To receive funding from competitive sources, independent reviewers from outside rank high priority projects that are of high quality & expected to yield the most value. This allows for comparison of benefits among alternative proposals.

Evidence: CSREES designed & packaged Goal 5 programs in collaboration with other agencies (FS, CSRS, FSA, ARS, ERS) to complement work of others & supplement the work not covered in their agency mission areas to reach intended audiences & beneficiaries. R&D leaders confer quarterly with REE agencies. See REE Quarterly R&D Performance Management Plan & Quarterly Reports. CSREES' mission to support research, education, & extension is an asset in specifically utilizing & adding value to research efforts of other federal agencies & departments, as well as other public & private sector providers. Goal 5 portfolio reviews found there is a suitable process for determining the proper set of intended beneficiaries, & the program is sufficiently flexible to adjust quickly as the pool of intended beneficiaries, needs & conditions change. See Review Panel Report, Objective 5.1, 2005; NRE Objective 5.2 Portfolio Panel Review Report, 2005.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Agency develops annual budget & programmatic priorities based on existing & emerging needs, recent successes in research, the state of science, educational capacity, & direct input from partners, constituents & other providers, as well as other USDA agencies, the Secretary and the Office of Management and Budget. CSREES secures inputs from stakeholders through both formal & informal processes. CSREES held a series of workshops including the USDA-CSREES Stakeholder Priorities Workshop for Animal Agriculture; Fair 2002 Implementation Partnership - USDA-ARS & the USDA-CSREES; the 2002 Joint ARS/CSREES National Aquaculture Program Planning Workshop. An overall priority is to increase the funding for competitive reserarch grants and to propose not to continue funding provided through earmarks. The comprehensive BPI process makes the linkages & potential gaps & overlaps apparent as the budget request is being prepared, allowing for refinement of the budget request, strengthened justifications, & appropriate program changes, before the budget request is submitted. See Dodge Kelsey & Pense, "A Model for Gathering Stakeholder Input for Setting Research Priorities at the Land Grant University." CSREES will be publishing documentation of the review process on the Web by the end of September 2005.

Evidence: Comprehensive National Research Initiative RFA Planning Worksheets are prepared for each priority issue, clearly defining mission-relevant problems to address in short, intermediate & long term goals with stakeholders. AREERA requires recipients of formula funds to collect stakeholder inputs yearly. CSREES held a series of workshops with customers & constituents in 2003. This information with Portfolio Review recommendations is integrated in the BPI process in collaboration with OBPA & OMB to guide Agency budget requests & funding decisions. NPLs & peers collaborate on dev. of RFAs, incl. the prioritization, science & edu., & integration of portfolio work in the agency. In line with the President & Congress' desire for more accountability of public funds, CSREES redirected formula based funds for the Animal Health & Disease, Section 1433 Research Program & $75M to universities to competitively awarded grants. See FY 2006 Explanatory Notes.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Recipients of competitive, formula & earmarked grants are required to submit annual progress & final reports through CRIS or directly to NPLs as per Terms & Conditions of awards. This information is used to revamp the RFA's for future years, to make recommendaions to specific grantees, or to provide a different approach across a proram area. Since research and extension programs have different purposes, baseline and performance data would not be the same. NPLs monitor progress, make recommendations, measure impact & make changes in priorities, focus & program administration. CSREES collects information on progress of research formula awards through CRIS & on the total portfolio of agricultural research & extension through the plan of work reporting process & other proprietary databases, with NPL oversight. CSREES is implementing a reporting system feature to CRIS for extension work. This will integrate performance data across areas of education, research & extension. The OneSolution system will increase timeliness & quality of data & information, reduce the reporting burden on partners, & increase program performance management. OneSolution is estimated to save the Agency $20.2 mil. over 7 years, with a benefit cost ratio of 1.89.

Evidence: Terms & Conditions for competitive, formula & line item awards specify annual & final report requirements. Reports determine progress of each grant & if additional funds should be provided. NPLs use information to compile reports, measure program impacts & respond to inquiries from management, public, & Congress. See cris.csrees.usda.gov/. Research projects supported with formula funds are reported in CRIS as per Administrative Manuals for Hatch, Evans-Allen, McIntire-Stennis & Animal Health Research Programs. Some extension formula programs have proprietary database reporting systems. 1862 & 1890 Land-Grant Institutions are required to submit 5-Year Plans of Work & Annual Reports of Accomplishments/Results (See Guidelines for State Plans of Work & Annual Report Guidelines). Reports measure impacts of institutions addressing critical issues identified through stakeholder input. For OneSolution information, see OneSolution Initiative, Business Case, Final Draft March 3, 2005.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: CSREES managers are accountable at all levels of the agency. Office of Extramural Programs is fiscally accountable; program managers are subject to Financial Assistance Regulations. Partners submit Cash Transactions & Financial Status Reports & undergo audit per OMB Circular A-133. Formula grantees submit a Cash Transactions Report & annual Financial Status documentation to sustain the report. NPL's are accountable for schedule & performance results. NPL's monitor progress by evaluating & measuring program impacts. Performance standards relate to strategic goals & objectives. Project periods (3 to 5 years) ensure time to carry out work on schedule & as approved from peer- or merit- reviewed proposals & plans of work. NPL's consider budget adequacy; partners obtain prior approval for budget changes. NPL's review & approve plans of work from land grant institutions & arrange for & coordinate review of any programs not formally covered by plan of work reporting.

Evidence: Budget Office develops Control Tables with NPL assignments for the fiscal year. Office of Extramural Programs develops guidelines & training to assist NPLs in solicitation, review & award of grants. This is one effort of the PACE Committee, a CSREES workgroup of NPLs & administrative staff working to streamline & simplify the grants administration process & to solve problems of mutual interest. Grant recipients are provided an annual allocation letter that provides amounts of formula grant awards, & if applicable, matching requirements. Institutions with a match requirement submit a Certification of Offset form before formula funds are released. Guidelines for State Plans of Work, Annual Report Guidance, & Administrative Guidance for Integrated Research & Extension Activities provide formula grantees with compliance & reporting instructions for the administration of CSREES formula funds. See Administrative Guidelines, & Terms & Conditions.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are, in all cases (unless Congressionally mandated otherwise as No Year or Two Year), required to be obligated in the year appropriated. Audits, reviews & mandatory reporting, along with active post-award management, ensure that funds are spent for the intended & agreed upon purposes.

Evidence: CSREES prepares a schedule of obligations, monitors unobligated balances, compares obligations with scheduled estimates, & makes appropriate adjustments. NPLs can defer the dispersal of obligated funds if deficiencies or deviations are detected. Awardees are required to take proactive corrective action in order to secure the release of funds. In FY 2004, CSREES did not obligate about 85,000 in Integrated Water Quality Funds. The amount can be verified by checking the 06 Explanatory Notes. Since this occurance, CSREES has developed a report in CREEMS to prevent this from happening again. This is the only instance in quite a while for CSREES.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: CSREES has an internal grants management system, CREEMS, that assists management & staff by providing budget, expenditures & statistical data related to the administration of grants (e.g., time to process a proposal from receipt to award; time to process a post-award action). The system serves as a management tool for tracking action items & administrative compliance requirements. The system is CSREES' innovation in integrating all grant processes (programmatic, administrative, & fiscal) in one database. Annual efficiency measures & targets compare proposal review time & costs for continual improvement. The OneSolution data & reporting system procedures will increase the timeliness & quality of performance related data & information, reduce the reporting burden on key program partners, & increase program performance management. It is estimated to save CSREES $20.2 mil. over 7 years, with a benefit cost ratio of 1.89. Costs do not include the use of the internal grant management system. At the present time, CSREES does not know the impact that electronic grants management will have on program costs.

Evidence: See the Annual Efficiency Measures tab for this PART. For CREEMS process documentation, see www.csrees.usda.gov/business/other_links/egov/csrees/creems.html. See also CSREES Customer Service Satisfaction Standards. The Agency has a comprehensive IT Strategic Plan to increase the use of information technology to increase cost & time efficiency, see www.csrees.usda.gov/about/offices/pdfs/strategic_plan_03-08_Final.pdf. The Agency also has a specific Information Technology Strategic Plan, revised in July 2004, www.csrees.usda.gov/about/offices/pdfs/strategic_plan.pdf. For CSREES OneSolution documentation & analysis of projected benefits, see OneSolution Initiative, Business Case, Final Draft March 3, 2005.

Does the program collaborate and coordinate effectively with related programs?

Explanation: CSREES programs & portfolios are highly integrated & coordinated both within the agency & externally. There is a high degree of coordination between related areas of science, education & extension. Coordination through CSREES NPLs with complementary programs in USDA, other federal departments, universities, state agencies, industry, & with producers & other stakeholders leads to adjustments in program management & resource allocation. NPLs & their counterparts from other federal agencies & institutions collaborate on the development of RFAs & guidance for Plans of Work, including priorities, the direction of science & education, & the integration of multidisciplinary work within the agency & with the work of other providers. Program relevance to the departmental & REE missions & priorities is regularly evaluated & coordinated with other providers & partners with similar missions & program foci. CSREES has not shown that these collaborations have led to meaningful results.

Evidence: CSREES constituents, stakeholders & partners, along with others doing related work, are routinely involved in program planning, development & review. For example, NPLs collaborate with scientists & educators within & beyond the Department, & with advisory groups, industry, producers & citizens to maintain the relevancy of education, research & extension & to ensure complementarity & avoid redundancy with the programs of others. One result is jointly funded programs with NASA, NSF, DOE & others. Coordination takes place within the USDA, with interagency work groups, & with other departments to add value & extend the benefits of the efforts of CSREES programs & of other programs. R&D leaders meet at least quarterly; see REE Quarterly R&D Reports; Work Teams in CSREES prepare integrated National Research Initiative RFAs based on mission related problems, & 3, 5 & 10 year goals.

Does the program use strong financial management practices?

Explanation: All grantees must comply with Federal financial assistance regulations, specific USDA & CSREES implementing regulations. Audits, reviews & mandatory reporting, with active post-award management, ensure that funds are spent for agreed upon purposes. Grant recipients are required to report expenditures each quarter 45 days after close of the quarter via PSC-272 Federal Cash Transactions Report, via DHHS-PMS. Draw down requests are denied for failure to submit this report. If an institution fails to submit technical & financial reports by June 15th, the 4th quarter formula funds & subsequent formula allocations are withheld until all requirements have been satisfied. The program has had a clean audit opinion for the last three years.

Evidence: USDA received an unqualified audit opinion for its FY 2004 Financial Statements & certified their financial statements by November 15, 2004, the deadline imposed by OMB. CSREES has no material weakness. Financial management is conducted as per OMB Circulars for Federal financial assistance, USDA regulations & the Administrative Manual, CRIS Instructions, CREEMS instructions, & Agency records indicating the receipt & approval of financial & technical reports. CSREES implemented the Federal Financial Information System after FY 2002. Prompt action is taken to correct any material weaknesses, if identified by auditors. Under the CSREES eGovernment Executive Board, the CREEMS Funds Management Module is one of the projects supported in terms of resources by the Board. In the last 2 years, ISTM has supported the funds management component of the grants process, helping greatly with reporting & business processes & therefore avoiding material weakness with the CREEMS audit.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Agency has a number of improvement processes underway to upgrade its management efforts. CSREES also has taken action to improve general administrative, managerial, & fiscal functions; post-award management of its entire grants system, including formula, as well as line-item, & competitive grant mechanisms; & planning & accountability efforts. CSREES set up an oversight staff within the Office of Extramural Programs. They provide oversight to OIG audits, OIG hotline complaints, GAO evaluations, IPIA & FMFIA compliance requirements, liaison with the Planning & Accountability staff on the POW, site visits, & analysis of trends in areas for which better internal controls & business processes need to be established. This office works closely with other units in the agency to determine management deficiencies & develop processes & reports to determine these deficiencies. Deficiencies are corrected through recommendations made by managers, such as adding electronic review and reducing grant review time. As opportunities arise, committees are identified to make recommendations. New annual efficiency measures have been developed, with historical baselines & 2005 targets.

Evidence: e-Gov will result in paperless proposal/plan submission, review, approval & award to reduce processing time & cost. Guidelines for review/approval of formula funding (proposed for elimination) are in review. CSREES transitions to electronic submission this year. A Program Administration Coordination Effort addresses management, administrative & fiscal issues to increase grant award effectiveness. PACE completed Service Standards that set temporal guidelines for planning, awards management & time-critical administrative & management activities & is establishing guidelines for post-award management. See Customer Service Satisfaction Standards. Planning & Accountability provides guidance, analysis, policy recommendations for strategic & short-term planning, budget justification, performance measurement/eval., contributing to review & improvement of administrative functions & assisting NPLs assess accomplishments & determine changes to improve performance & efficiency.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The vast majority of grants are awarded on a competitive-based process. An extensive, rigorous review process for competitive programs includes a panel manager, qualified reviewers, strict guidelines & safeguards to avoid conflict of interest. Formula funds (proposed for phase out in the FY 2006 budget), research, special or administrative grants are subject to qualified expert assessment of merit prior to award of funding. In all cases, release of funds can be & is deferred pending submission & approval of an acceptable & merit review proposal of work against which performance is assessed. CSREES works with new & first-time recipients to ensure they understand Federal Financial Assistance regulations & polices & to ensure the proposed project is sound, relevant & meritorious. National Program Leaders monitor awards to ensure progress & the approved proposals. To ensure success & fulfill requirements, CSREES can place additional controls on new grant recipients.

Evidence: In accordance with statutes (c.f., Agricultural Research, Extension, & Education Reform Act of 1998) or other legislation & official policies, proposals & plans of work are subject to either peer review or rigorous merit assessment. The agency maintains an extensive data base of highly qualified grant proposal reviewers from government, academia, & industry, & provides strict guidelines, training, & NPL & administrative oversight to the proposal review & award recommendation processes. All reviews both peer & merit, are held to the same strict standards of relevance, quality & performance as per the PMA. See Administrative Guidelines & Terms & Conditions.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Agency's CRIS reporting system sufficiently monitors grantee activities. All grantees are required to submit proposals or plans of work fully describing the education, research & extension work for which all types of funding (including mandated formula and special grants) is requested. Grantees are required to submit Annual Reports of Accomplishments describing their progress in achieving their stated goals for each funded activity. Reports are entered in the CRIS data base cris.csrees.usda.gov/ & are available to the public. NPLs & Awards Management Branch provide review & oversight of all funded activities, including mandated projects. NPLs provide program leadership & oversight, including site visits; agency administrative & support staff assist in program management & support the review process for each program. Expert peer or merit review panels are used to provide assessment of the efforts of grantees.

Evidence: The Administrative Manual contains explicit program monitoring & reporting requirements. Grantees submit annual Progress Reports; a report is required for each funded project included in the institution's approved work plan for the Fiscal Year. Termination Reports must be submitted for each completed or terminated project. These must be submitted with progress reports on active projects & include a summary of outputs & accomplishments over the entire life of the funded project. CSREES is implementing an enhanced CRIS reporting system to include extension. See cris.csrees.usda.gov/; www.csrees.usda.gov/business/other_links. The agency conducts site visits to a substantial number of grantees on a regular basis & conducts annual audits of grantee performance. NPLs maintain close working relationships with grantees & their institutions.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: CSREES annually collects, compiles & disseminates grantee performance data down to the funded project level, & makes data publicly available in a transparent manner via the CRIS Web Site at www.cris.csrees.usda.gov/. Grantees are contractually required to submit an Annual Report of Accomplishments, & Termination Reports, to describe their performance in achieving their stated goals, & to describe their progress on funded projects are entered into the CRIS database.

Evidence: The widely used CRIS is a comprehensive repository of performance data for funded projects from CSREES, & for work of related agencies with similar missions & activities. These are identified by problem area, subject of investigation, & field of science, & include proposal summaries, annual performance reports, outcomes & impacts, & termination reports. This information is searchable, available to the public, & has the capacity to produce various types of reports according to customer needs. See www.csrees.usda.gov/business/other_links/egov/csrees/cris.html. Also see www.cris.csrees.usda.gov/. Another CSREES innovation, the Competitive Research, Education, & Extension management System (C-REEMS), makes comprehensive information on proposals, their review & award management available. See www.csrees.usda.gov/business/other_links/egov/csrees/creems.html. Also see OneSolution Initiative, Business Case, Final Draft March 3, 2005.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Partners are required to submit annual work plans subject to merit/peer review, by at least 1 NPL & 2 outside reviewers for approval prior to award. The same is used for non-competitive. CSREES ensures that non-competitive grant proposals submitted include all required administrative & programmatic elements. CSREES publishes guidelines for NPLs on how to request, review & fund proposals. Institutions are required to perform peer review Special Research Grants & Merit reviews for the Federal administration & to certify that they were done for both the research & extension grants. Financial management is explicitly conducted as per OMB Circulars for Federal financial assistance, USDA regulations & the Administrative Manual, CRIS Instructions, C-REEMS instructions, & Agency records indicating receipt & approval of financial & technical reports. Portfolio review assessment outcomes from panels provide quantitative & qualitative assessments that focus & guide future efforts. [Note: Of the total funding for goal 5 programs, 46% is provided through formula grants, 28% through competitive grants, nine percent through earmarks, and 17 percent through other grants.]

Evidence: Financial management is explicitly conducted as per OMB Circulars for Federal financial assistance, USDA regulations & the Administrative Manual, CRIS Instructions, C-REEMS instructions, & Agency records indicating receipt & approval of financial & technical reports. Portfolio review assessment outcomes from panels provide quantitative & qualitative assessments that focus & guide future efforts. Research projects supported with formula funds are reported in CRIS as per Administrative Manuals for Hatch, Evans-Allen, McIntire-Stennis & Animal Health Research Programs. For some extension formula programs there are proprietary database reporting systems. Portfolios of funded projects including formula & special grants are reviewed by independent panels of outside experts for relevance, quality & performance. See CSREES Customer Service Satisfaction Standards.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The question received a "no" because none of the measures has been in place long enough show progress toward reaching the long term goal.

Evidence: See measures tab.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program received a "small extent" because it only achieved its goals for the two efficiency measures. The external portfolio review is the 1st annual review and the results of the external reviews are being used as the baseline for internal reviews in years 2-5 of a 5 year cycle. Because this is the 1st year of the external review, no internal review has been conducted. The first internal review for portfolios under Goal 3 will be conducted in FY06. Nevertheless the Agency management has initiated portfolio modifications in response to the recommendations of the external review panel. These initial improvements support a score of small extent.

Evidence: Were an internal review to be conducted this fiscal year it would indicate that the program managers had made concrete portfolio adjustments in response to the recommendations of the FY05 review and result in an increase in the portfolio score. For example, the current draft of the NRI request for applications to be released in Oct. 05 will request additional research in ecosystem level management tools to address the critical issues in invasive species.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Agency estimates that at the end of FY05 that it will have met or exceeded the FY05 targets for its efficiency performance measures. The efficiency and cost-effectiveness of the Agency's programs are increasing through the reduction in time-to-completion of proposal solicitation, review and funding, and by minimizing peer vetting costs per proposal reviewed. Larger and longer term awards are reducing the relative cost of processing, days, and cost to grantees per award. Transition to full electronic solicitation, submission and processing of proposals has been initiated and will continue into 2006. This major change in the Agency's business process is anticipated to yield considerable efficiency and cost-effectiveness.

Evidence: Estimates at the end of the third quarter of FY05 indicate that the Agency has made improvements in both the reduction in time from submission to receipt of grant award and the associated cost. Efficiency benchmarks were reconfigured in 2004 to accurately reflect significant changes: increased airline transportation costs, a 25% increase in honoraria in order to remain competitive with other research agencies in attracting the best qualified peer reviewers, and increased use of more effective but more costly on-site, rather than ad hoc, reviewers. Even with these adjustments, CSREES cost per proposal reviewed is extremely efficient vs. comparable NSF costs of $650 per proposal. The continuing trend toward more focused, larger grant awards increases administrative efficiency. CSREES honoraria are low compared to other federal agencies with whom CSREES must compete for highly qualified reviewers.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The answer received a "small extent" no study was provided to show a direct comparison between this program and other similar ones. However, the Agency has solicited panel members from Universities, Agencies, and the private sector entities with similar purposes and goals to conduct independent external assessments of the Agency portfolios. The score from their independent and expert assessment as measured by the R&D criteria of relevance, quality and performance, is testimony to the strength of the CSREES programs relative to programs with similar goals.

Evidence: The Portfolio Review Expert Panels, comprised of entities with similar goals, reiterated their satisfaction with the relevance, quality, and performance of the programs. Individual portfolio scores were Forest and Rangelands, 77; and Soil, Air, and Water Quality 81. They emphasized the high quality of portfolio leadership. While they recommended a few refinements, the panels judged the portfolio to be of high quality and assigned them good scores. One can infer that because the reviewers come from organizations with similar goals and purposes, their relatively high scores imply that the Agency is performing well in comparison to the organizations from which they come.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Goal 5 portfolios were independently assessed by outside experts & recommendations help NPLs & partners. Research affects productivity & environmental health with a time lag; benefit streams typically extend beyond 30 yrs. Studies found rates of return of 40% to 60% for public R&D regardless of level of aggregation or geographical area. Extension reduces time lags; its major effect on productivity is the short to intermediate period, although major technology adoption throughout the sector is a LT accomplishment due to the dynamic & constantly changing technology & management methods available. Economists estimate rates of return to extension technology transfer & management & behavior change from 20% to over 100%. Number of citations reflects high quality research, as scientists most often cite important, seminal work in a field.

Evidence: The agricultural & food sector, including natural resources & environmental management, operates at high efficiency, using the knowledge, techniques & technologies developed at land grant universities through R&D programs. See National Academy Press, NRI: A Vital Competitive Grants Program in Food, Fiber & Natural Resources Research. An extensive evaluative research bibliography, largely in vetted, peer reviewed journals & government studies, verifies the effectiveness of publicly funded agricultural & natural resources R&D, including extension. See Bibliography of Agricultural Research & Extension Structure, Economic Impact, & Rate of Return Literature.