Detailed Information on the
State Energy Programs Assessment

Establish the current amount of energy saved by the program and set ambitious targets for the future.

Undertake an independent analysis of program benefits and effectiveness.

Measure: Energy savings

Explanation:

Measure: Ratio of program direction/contractor management funding to total program funding.

Explanation:

Is the program purpose clear?

Explanation: The statutory purpose of this program is to provide grants and related technical assistance to States to promote the conservation of energy and to reduce the rate of growth of energy demand.

Evidence: Program authority is contained in PL 94-163, Title III, Section C., as revised by PL 101-440. Governing regulations are contained in 10CFR 420.1.

Does the program address a specific and existing problem, interest or need?

Explanation: The program contributes to improved energy security and reliability by reducing energy consumption (and associated negative environmental impacts) and improving energy efficiency. The program provides the resources to enable States to plan and implement programs tailored to their individual energy situation and needs, to directly influence the energy use of their citizens.

Evidence: State Energy Program's Strategic Plan for the 21st Century, February 2000, page 2; and SEP Implementation Plan, November 2000, page v - vi.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: SEP provides resources to coordinate State, local, and Federal programs to reduce duplication and increase effectiveness. Some of these state programs, such as those run by the CEC in California, NYSERDA in New York, and Wisconsin's Focus on Energy, have similar objectives, but in these places, SEP serves the unique role to help plan for the use of these state resources and to integrate these programs into DOE objectives.

Evidence: April 2, 2004 testimony of the National Association of State Energy Officials (NASEO) before the House Appropriations Subcommittee. Annual SEP State Plans.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The SEP grant program is designed to allow the States maximum flexibility in developing programs that address their needs, thus achieving greater effectiveness. A more prescriptive categorical program design would be much less effective, since no single set of measures could effectively address the varying energy situations and needs of every State across the nation. Program managers regularly review SEP's effectiveness with its Regional Offices and the States and make policy and program changes as necessary.

Evidence: Two earlier Federal grant programs, the Energy Extension Service and the Institutional Conservation Program, have been consolidated into SEP in view of its flexible design and effectiveness. DOE Headquarters and Regional SEP managers meet semi-anually and conduct monthly conference calls to review policy and operational issues. Regional SEP managers meet at least annually with their States for the same purpose, and DOE holds a national "All-States" meeting every 18 months. In addition, State feedback on program operations and effectiveness is requested on the SEP Program Managers' website, The Corner Cafe.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: SEP's funds are provided to the States through a negotiated equitable formula that takes into account population, energy consumption and equal shares. Working through the States is the most effective way to closely target the program to the individual needs of all citizens.

Evidence: Allocation in 10CFR 420.11. NASEO April 2, 2004 testimony before the House Appropriations Subcommittee.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program is in the process of developing better and specific national programmatic long-term objectives for SEP through the update of the SEP Strategic Plan for the 21st Century. Ideally, long-term measures will be outcome-oriented (e.g., MBtu of energy saved, energy cost savings, tons of reduced pollutant emissions, oil imports displaced, all per annum). The program is in the process of discussing these measures with the States.

Evidence: The program is in the process of developing specific long-term performance measures for SEP. Current long-term goals for the program provided in the program's Strategic Plan for the 21st Century (February 2000) will be updated in FY 2004-2005 to include goals that are more quantifiable.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program is in the process of developing specific national annual and five-year objectives as part of updating the SEP Strategic Plan. Long-term and short-term measurement of these objectives is being developed as part of ongoing evaluation work.

Evidence: FY 2004 Annual SEP Team Work Plan.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program is in the process of developing better and specific national programmatic long-term objectives for SEP through the update of the SEP Strategic Plan for the 21st Century. Ideally, long-term measures will be outcome-oriented (e.g., MBtu of energy saved, energy cost savings, tons of reduced pollutant emissions, oil imports displaced, all per annum). The program is in the process of discussing these measures with the States.

Evidence: The program is in the process of developing specific long-term performance measures for SEP. Current long-term goals for the program provided in the program's Strategic Plan for the 21st Century (February 2000) will be updated in FY 2004-2005 to include goals that are more quantifiable.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Annual targets, and baselines for SEP are under development as part of long-term planning effort that is part of updating the SEP Strategic Plan. The States maintain their own baselines and establish individual targets for their annual measures.

Evidence: Program regulations in 10CFR 420. State Energy Program Operations Manual. Annual SEP State Plans. Headquarters and Regional Office SEP Work Plans.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Annual and long-term goals for SEP are under development with participation from States and their associations. The goals in the SEP Strategic Plan for the 21st Century were developed by a committee of State Energy Office directors and reviewed in draft by all the States and their associations. States use SEP funds to establish and carry out their own individual annual goals.

Evidence: SEP Strategic Plan for the 21st Century, pages 3-11, page 16. Information on State goals and activities is contained in the the Department's WinSAGA database. State Energy Advisory Board periodic and annual reports.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Although progress is being made, independent evaluations of sufficient scope and analysis are not yet in place. Oak Ridge National Laboratory has conducted a preliminary assessment. However, the States self-reported many of the facts for this assessment. To ensure quality data, every State should have auditable field measurement mechanisms. More importantly, Oak Ridge National Lab is not considered independent.

Evidence: The initial assessment of actual SEP performance was performed by Oak Ridge National Laboratory and published as ORNL/CON-487, "Estimating Energy And Cost Savings And Emissions Reductions For The State Energy Program Based On Enumeration Indicators Data," Schweitzer, et.al., January, 2003. In FY 2004 DOE has instructed ORNL to expand the number of activities areas assessed under the methodology and to seek responses of nonrespondent states to the metrics solicitation.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The 2005 Budget highlights the energy savings (Btus) and cost savings estimated for each dollar of grant funding provided under this program. It is noteworthy, however, that the estimates are based on extrapolation from figures in an Oak Ridge National Lab study that relies on incomplete data and methodologies that may require significant refinement.

Evidence: FY 2005 Budget Justifications. Annual SEP Program Guidance. Annual SEP State Plans.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In November of 2000 an implementation plan for the SEP Strategic Plan for the 21st Century was published and highlighted nine suggested activities that, if implemented within the States' plans, would enhance the achievement of three primary program goals. DOE, in cooperation with the States, plans to update the Strategic Plan in FY 2004-5. The new plan will contain specific long-term and annual objectives consistent with the program's statutory objectives and outcome measures. In FY 2002-3, DOE provided technical assistance to the States in strategic planning, to strengthen the process at the State level.

Evidence: State Energy Program's Strategic Plan for the 21st Century, February 2000 and SEP Implementation Plan, November 2000.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: States are required to report to DOE semi-annually on their progress toward meeting the goals and milestones included in their State Plans. These reports are captured in the WinSAGA database and are reviewed by DOE to identify operational and financial issues that need to be addressed. However, the State's choose their own goals and measures, which may not exactly reflect the measures used by DOE.

Evidence: Information on State activities and progress toward meeting their goals is contained in WinSAGA, the program's automated database system.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The program's headquarters (HQ) staff, the Regional Offices (ROs) and the States are all responsible and accountable for different aspects of SEP operations. HQ staff is responsible for budget formulation, policy development, annual SEP program guidance, resolution of programmatic issues, design and implementation of nationwide technical assistance, program evaluation and liaison with and annual monitoring of the ROs. The ROs are responsible for dealing directly with the States, reviewing annual grant applications and awarding grants, reviewing semi-annual programmatic and financial status reports, monitoring the States' implementation of SEP, and providing specific or region-focused technical assistance. States are responsible for developing and submitting their annual grant applications (which includes selecting the annual energy measures that best fit their needs); submitting semi-annual programmatic and financial status reports to the DOE ROs; and for monitoring their sub-grantees. In addition, States are subject to an annual independent audit of their activities.

Evidence: Responsibilities of program HQ staff, Region Offices, and States are spelled out in the SEP Operations Manual, pages 3-9 thru 3-13. Grant documentation including assurance of compliance is maintained by DOE's contracting officer at each Regional Office. Quarterly reports are filed on the WINSAGA grants management data system.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All Federal grant funds are obligated within the federal fiscal year to coincide with the States' fiscal year. In FY 2003-5 DOE is working with a number of States to reduce their high balances of unobligated funds carried forward at the end of the fiscal year. After a strategic program review identified this problem, Annual SEP Guidance for FY 2003-2004 required States with carryover balances over 25% to include in their annual State Plans a plan for reducing those balances. In FY 2004, a team of HQ/Regional SEP Managers is developing specific step-by-step procedures to be followed to resolve this problem with a State. In addition to reviewing the semi-annual reports, the Regional Offices monitor the States to ensure that SEP funds are spent for their intended purpose. HQ SEP staff, in turn, monitor Regional SEP operations annually.

Evidence: The WinSAGA system tracks grant awards and funds obligations. HQ and Regional Office monitoring schedules and reports are on file. State Energy Program Strategic Review, 2002

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: DOE has taken steps to improve the efficiency of SEP operations. Contracts awarded for technical assistance are awarded competitively. IT improvements have reduced time, travel and other costs associated with grants administration. Ultimately, the program will transition to Grants.gov, a cross-agency e-government initiative that will establish one system for all customers of federal grants to find, apply for, and manage grants.

Evidence: EERE Annual Broad-Based Solicitations; Annual Task Plans for WinSAGA database design and operation; State Energy Program Operations Manual; Grant.gov. Applications previously submitted by mail can be signed and uploaded electronically. With DOE HQ, Regional Offices and the States all part of the same electronic system, problems can be resolved and issues clarified in real time. Activities that used to require face-to-face meetings involving travel by large groups of people are now be carried out by web-cast. For example, HQ and the Regional Offices used to meet face-to-face quarterly, and now with the IT improvements such meetings are necessary only once per year. Program meetings with all of the States have been changed from annual to once per 18 months .

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Program has ongoing collaborative efforts with Environmental Protection Agency (EPA) to promote State Energy Office and State Air Quality Offices to develop Energy Efficiency & Renewable Energy (EERE) solutions to environmental problems. For example, Utah used SEP funds for a project jointly directed by staff from the State Office of Energy and Resource Planning, Office of Energy Services, and the Division of Air Quality, to design, develop and adopt ways to incorporate energy efficiency and renewable measures into Utah's State Implementation Plan under the EPA's Clean Air Act. States also work with EPA to promote consumer purchases of EnergySTAR energy efficient products.

Evidence: Through collaboration efforts between the EERE Technology Offices and the States, over 200 State Energy Program Special Projects are presently being implemented at the State level. Information on these projects is available in the program's WinSAGA database.

Does the program use strong financial management practices?

Explanation: The program complies with all DOE Financial Directives. Oversight is provided by the National Energy Technology Laboratory.

Evidence: No evidence of poor financial practices.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program continually reviews its management and operational procedures. DOE HQ and Regional Office SEP managers meet semi-annually and conduct monthly conference calls to review operational issues. Regional Office SEP managers meet at least annually with their States for the same purpose, and DOE holds a national "All-States" meeting every 18 months. In addition, State feedback on program operations and effectiveness is requested on the SEP Program Managers' website, The Corner Cafe. Recently, procedures for approving Regional SEP Annual Operating Plans were streamlined, so that funds are made available more quickly, and Regional Offices have more discretion in their use.

Evidence: Annual Program Guidance. SEP 21st Century Strategic Plan.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: HQ SEP staff monitors and provides technical assistance to the Regional Offices, which, in turn, oversee grantee activities, and are in regular contact with the grantees. Regional SEP Managers closely review annual State Plans, semi-annual financial and progress reports, track progress against milestones, and conduct site monitoring to ensure compliance with approved State Plans and soundness of State financial systems, so that federal funds are tracked and expended properly.

Evidence: State semi-annual reports are on file in the program's WinSAGA database system. Reports of on-site monitoring are on file at Regional Offices.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Oak Ridge National Lab (ORNL) developed an energy savings reporting system for SEP and used it to develop estimates of program benefits for various program activities. The program plans to expand the number of activity areas assessed under the methodology and to seek responses of nonrespondent states to the metrics solicitation. The program anticipates that annual reporting and publication of reporting results will be conducted from this point forward.

Evidence: ORNL/CON-487, "Estimating Energy And Cost Savings And Emissions Reductions For The State Energy Program Based On Enumeration Indicators Data," Schweitzer, et.al., January, 2003. ORNL SEP Support Task Plan for FY 2004

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program and States have not agreed to long-term performance goals. ORNL's 2003 study reported on SEP's annual progress related to its statutory intent of saving energy and energy costs, and reducing harmful emissions. The program has begun the collaborative process of expanding the number of activity areas assessed under the methodology and to seek responses of non-respondent states to the metrics solicitation. The data will inform development of meaningful long-term goals.

Evidence: Annual SEP Work Plans for HQ and Regonal Offices. Moitoring Reports on file at RO's. HQ Technical Assistance Plans and records. Oak Ridge National Laboratory , ORNL/CON-487, "Estimating Energy And Cost Savings And Emissions Reductions For The State Energy Program Based On Enumeration Indicators Data," Schweitzer, et.al., January, 2003. ORNL FY 2004 SEP Task Plan

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual goals for the program are still under development as part of SEP's strategic planning process.

Evidence: Annual SEP Work Plans for HQ and RO's. Moitoring Reports on file at RO's. HQ Technical Assistance Plans and records. Oak Ridge National Laboratory , ORNL/CON-487, "Estimating Energy And Cost Savings And Emissions Reductions For The State Energy Program Based On Enumeration Indicators Data," Schweitzer, et.al., January, 2003. ORNL FY 2004 SEP Task Plan

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Grants administration efficiencies are achieved each year through increased use of IT technology, consolidating/reducing meetings to reduce travel requirements and streamlined operational procedures. Required reports have been streamlined and the program has moved from quarterly to semi-annual reporting, to reduce the reporting burden on the States. Specific measurable efficiency goals are under development.

Evidence: WinSAGA Grants Administration Database. SEP Operations Manual. Annual Program Guidance. Applications previously submitted by mail can be signed and uploaded electronically. With DOE HQ, Regional Offices and the States all part of the same electronic system, problems can be resolved and issues clarified in real time. Activities that used to require face-to-face meetings involving travel by large groups of people are now be carried out by web-cast. For example, HQ and the Regional Offices used to meet face-to-face quarterly, and now with the IT improvements such meetings are necessary only once per year. Program meetings with all of the States have been changed from annual to once per 18 months .

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other block grant programs with similar purpose and goals. In some States, State funding augments Federal funding provided by this program. But State funding is not used to operate a separate, comparable progam.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Oak Ridge National Laboratory developed an evaluation methodology for the SEP program and reported positive results. However, the data were incomplete, and the methodology may need refinement, and ORNL is not considered independent. The long-term plan for the evaluation of the program calls for the gradual expansion of measured activities from the current 14 categories to 18 and of participation of almost all states compared to the previous 20. The nature of some SEP activities does not easily lend itself to specific measurement by the three criteria used in the ORNL methodology and the resulting metrics are therefore only a partial reflection of program performance. It is intended that metrics be refined and expanded on an annual basis to produce not only more accurate results but assessment of program activities over time. Data collection is performed on a voluntary basis in cooperation with National Association of State Energy Officials (NASEO).

Evidence: The initial assessment of actual SEP performance was performed by Oak Ridge National Laboratory and published as ORNL/CON-487, "Estimating Energy And Cost Savings And Emissions Reductions For The State Energy Program Based On Enumeration Indicators Data," Schweitzer, et.al., January, 2003. The program will expand the number of activities areas assessed under the methodology and seek responses of non-respondent states to the metrics solicitation.