Detailed Information on the
Building Technologies Assessment

Develop guidance that specifies a consistent framework for analyzing the costs and benefits of research and development investments, and use this information to guide budget decisions.

Provide funding consistent with meeting performance targets, including continued support for solid state lighting and reduced support for other technologies near commercialization.

Refine performance measures to better capture the breadth of activities supported by the program.

Accelerate analyses to support generation of new energy efficiency regulations prescribed by law.

Measure: Number of design technology packages for new residential buildings (and percent increase in energy efficiency relative to the 2004 Building America benchmark) with a payback period of five years or less. (There are 15 potential design packages: 3 building types in each of 5 climate zones. Design packages incorporating renewable energy technologies can lead to Zero Energy Homes.)

Explanation:Use of the design packages will reduce expected energy consumption of new residential buildings (single family homes, multi-family homes, and townhomes). The range in efficiency improvements reflects the range that can be expected in different climates.

Measure: Percent increase in energy efficiency of the International Energy Conservation Code for residential buildings based on cost effective proposals developed by the program.

Explanation:Code change proposals will increase the energy efficiency of all residential buildings constructed to the IEEC standard.

Measure: Number of design technology packages for new commercial buildings (and percent increase in energy efficiency relative to the ASHRAE 90.1-2004 benchmark) with five year or less payback.

Explanation:Use of the design packages will reduce expected energy consumption of new commercial buildings. The range in efficiency improvements reflects the range that can be expected in different climates. Over time, it is expected that the payback period for these packages will be reduced.

Measure: Percent increase in energy efficiency of the International Energy Conservation Code for commercial buildings based on cost effective proposals developed by the program.

Explanation:Code change proposals will increase the energy efficiency of all commercial buildings constructed to the IEEC standard.

Measure: Efficiency of "white light" solid-state lighting in a lab device, in lumens per Watt (LPW)

Explanation:Improving the efficiency, reducing the cost, and improving the quality of white light produced by LEDs can lead to increased commercial deployment and significant energy savings as a result. (Solid state lighting, also known as light emitting diodes [LEDs], can potentially be more than twice as efficient as fluorescent lighting [currently about 85 LPW] and may be able to last 10 times as long -- up to 100,000 hours.).

Measure: Number of proposals to update appliance standards and test procedures published in the Federal Register.

Explanation:These standards and test procedures lead to improved energy efficiency of appliances and equipment, which reduces energy use and greenhouse gas emissions. Includes ANOPRs, NOPRs, and final rules. For this measure "proposal" includes unique product inclusions in ANOPRS, NOPRS, and Final Rules. Multiple proposals (covering a number of product categories) could be bundled in Federal Register Notices.

Measure: Reduction in window energy use (percentage) compared to a 2003 Energy Star window, which represents the windows with energy performance in the top 25 percent of windows available in the market. The percentage reductions will only be considered complete if all other technical performance requirements are also met. These include visual transmittance, solar heat gain coefficient, durability (American Society for Testing and Materials tests), U-value, and incremental cost ($/sq.ft).

Explanation:Improving the efficiency, reducing the cost, and improving the performance of windows will help the program develop design for net zero energy buildings, which produce as much energy as they consume in a year.

Measure: Percentage reduction in annual energy consumption from heating, ventiliation, and air condiditoning (HVAC), water heating, and dehumidification, as measured against the Building America benchmark. The Building America benchmark roughly represents the energy performance of homes built to energy codes in place in the mid-1990s.

Explanation:HVAC, water heating, and dehumidification are major energy loads in buildings. Reducing energy consumption from these sources can support the development of net zero energy buildings, which produce as much energy as they use in a year.

Measure: Administrative costs as a percent of total program costs (%).

Explanation:This "overhead rate" measure is not a true efficiency measure but is a meaningful surrogate used for all DOE applied R&D and related programs. The objective is to maintain a reasonable overhead rate for effective operation while ensuring that the vast majority of funds address the program purpose. Administrative costs include all Program Direction and Program Support costs plus costs for supporting activities and analysis funded through programmatic appropriations. The targets and actuals represent corporate figures (i.e., for the entire Office of Energy Efficiency and Renewable Energy) because some EERE Program Direction costs are difficult to parse at the program level in a meaningful way. Appropriation levels for EERE programs and for EERE Program Direction directly affect whether the target is achieved. The baseline and targets for this measure are under development.

Is the program purpose clear?

Explanation: The mission of the Building Technologies Program is to develop technologies, techniques and tools for making residential and commercial buildings more energy efficient, productive, and affordable. This involves research, development, demonstration, and technology transfer activities in partnership with industry, government agencies, universities, and national laboratories. The program also develops building codes and appliance standards.

Evidence: FY 2004 Budget; P.L. 94-163, "Energy Policy and Conservation Act" (EPCA) (1975) and seven subsequent pieces of related authorizing legislation.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program aims to reduce energy use in buildings, which can help avoid emissions of pollutants and greenhouse gases. These potential benefits support the Administration's National Energy Policy, as well as the Administration's climate change goals.

Evidence: The program focuses R&D on activities that it considers too technologically risky for the private sector to undertake alone. Risk levels vary on a project-by project basis.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The program coordinates its efforts with other entities as appropriate. For example, many Buildings subprograms (windows, lighting, commercial buildings, building envelope, space conditioning) work closely with industry to identify pre-competitive R&D needs and prepare "roadmaps." The program coordinates with HUD and others in certain multi-agency efforts, such as the Partnership for Advanced Technology in Housing (PATH). Through the efforts of the Association of States Research and Technology Transfer Institute (ASERTTI), coordinated research agendas are developed with the counterpart State research entities.

Evidence: The program identified market barriers to private sector investment in energy efficient building technologies. For example, building construction is a fragmented industry comprised of thousands of builders and manufacturers, none of which has the capacity to sustain research and development activities over multi-year periods. Another factor is the compartmentalization of the building industry, in which architects and designers, developers, construction companies, engineering firms, and energy services providers do not typically apply integrated strategies for siting, construction, operations, and maintenance.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The majority of the program's activities are in the area of applied technology research and development to improve energy efficiency of buildings. The program also develops codes and standards and conducts technology transfer and information exchange to integrate R&D advances into new building construction and retrofits.

Evidence: The program found no studies to indicate that a more cost effective approach to improving energy efficiency in buildings exists.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: In support of the Administration's R&D Investment Criteria initiative, the program was asked to prepare "bubble charts" that plot key program variables (e.g., expected public benefits, funding levels, years to commercialization). Bubble charts can serve as an informational tool to help determine, along with other considerations, whether the program appropriately targets its R&D funding. While the program has made progress estimating public benefits, the Department has not yet developed a methodology to estimate benefits consistently within and across programs. Therefore, the program could not prepare meaningful bubble charts.

Evidence: In general, the program appears to target its resources wisely, but a lack of ability to provide appropriate evidence mandates a "no" response. EERE continues to work internally and with other DOE program offices to improve consistency and accuracy in estimating benefits.

Does the program effectively articulate potential public benefits?

Explanation:  

Evidence:  

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation:  

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has established reasonable long-term goals that cover the majority of its R&D activities. The goals relate to residential and commercial building R&D (including Zero Energy Building R&D), and development of building-related products (e.g., lighting, windows, etc.). The measures and targets have been modified for FY 2005.

Evidence: FY 2004 Budget. Building Technologies Program Multi-Year Plan (Draft in Progress).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program is in the process of identifying targets and off-ramps that would help it redirect, down-select, or terminate efforts in its main R&D activities, but its efforts are not yet complete.

Evidence:  

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The program has developed some acceptable annual measures for its activities related to development of buidling codes and appliance standards. However, the program has not developed annual performance measures for R&D activities that directly tie to the program's long-term R&D goals.

Evidence: Building Technologies Program Multi-Year Plan (Draft in Progress).

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: The program has not yet developed acceptable annual measures for its R&D activities.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program develops annual operating plans which reflect the activities in the draft Multi-Year Plan. Once the operating plans are approved, grantees, sub grantees, contractors, etc. work with the program to develop specific statements of work to reflect milestones and deliverables that ultimately support achievement of the long term goals.

Evidence: Building Technologies Program Multi-Year Plan (Draft in Progress); Draft FY 2004 Annual Operating Plans for the Commercial Buildings Team, Residential Building Team, Emerging Technologies Team, and Appliance Standards Team.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Historically, the program has not planned for peer reviews. However, in FY 2003, each of the four teams of the Building Technologies Program developed an evaluation plan, which includes quality and control activities, such as peer reviews and third party evaluation. The program is working with EERE to develop guidelines for peer reviews which will be adopted and implemented. In the Emerging Technologies Team, the space conditioning R&D activity conducted a peer review in May 2003. In addition to technology specific peer reviews, the program should consider expanding the scope of peer reviews to include overall program effectiveness and relevance.

Evidence: Space Conditioning Peer Review (May 2003). Evaluation Plans for Building Technology Teams.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Program funding requests are tied to specific activities that contribute to the program's long-term goals. However, budget documents do not clearly indicate the full costs of achieving the program goals. Salaries, benefits, and other admininstrative expenses to support the program are included in a separate budgetary line item ("Policy and Management"). EERE does not report the allocation of Policy and Management funding to the various programs it supports.

Evidence: FY 2004 Budget. Building Technologies Program Multi-Year Plan (Draft in Progress).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Among the deficiencies in strategic planning are inconsistencies and lack of clear links between goals and targets in budget submissions, program strategic plans, and annual performance plans. The program has begun a multi-year planning process that should address some of these deficiencies.

Evidence: Building Technologies Program Multi-Year Plan (Draft in Progress).

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: Each year, the program estimates the public benefits of its activities in support of the Government Performance and Results Act (GPRA) and the Administration's R&D Investment Criteria initiative. However, the program has not yet developed a consistent and reliable methodology for comparing potential benefits within and across programs with similar goals.

Evidence: FY 2004 Budget

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: For development of appliance standards, the program evaluates potential energy savings and prioritizes its proposals for rules accordingly. The program participated in EERE's zero-based budget exercise to help determine priorities for R&D activities. Priorities for the program were grouped and less clear than for other EERE programs, but the program did attempt to prioritize.

Evidence: EERE Priority Ranking Tool, Zero Based Budget Exercise.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The EERE Strategic Management System -- which establishes at the beginning of each fiscal year an 18-month schedule for key planning, budget formulation, budget execution, and analysis / evaluation functions -- requires that each EERE program establish and track long-term and near-term program performance goals and measures. Program results as evaluated through the goals and measures are used annually and throughout the year to assess partners' performance, adjust funding, and re-align R&D portfolios.

Evidence: SMS Implementation Letter for FY 2002 - 2005 (October 2001). The program also reports on quarterly milestones in the Department's Joule database. However, in general, milestones in the Joule system are not necessarily meaningful or fully reflective of program progress. Thus, the Department's Joule system provides little value-added. The new I-MANAGE system, currently under development, will better integrate budget and performance.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: The Annual Performance Appraisals of all EERE Program Managers include criteria directly related to cost, schedule, and performance results. EERE reviews these criteria monthly in the EERE Monthly Management Reviews. Most EERE contracts include award fee and other performance criteria to hold those partners accountable.

Evidence: Performance Plan and Performance Appraisal Form for Performance Management System Employees. EERE Award Fee and Performance Based contracts.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: EERE conducts an Annual Operating Plan (AOP) Review before each fiscal year to assure that new funding is obligated consistent with the appropriated purpose. EERE uses data from Departmental procurement and financial systems -- and similar data from National Laboratory partners -- to assure that actual expenditures occur for purposes and on a schedule consistent with the AOP. Unobligated balances brought forward to FY 2004 were $713,000, 1.2 percent of the program's FY 2003 appropriation of approximately $60 million.

Evidence: Annual Operating Plans for each of four Building Program teams. Monthly obligation and cost reports from the EERE Strategic Management System and Departmental financial systems. Building Technology Program FY 2003 Financial Status Report (June 2003), FY 2004 Apportionment

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: EERE's reorganization in 2002 clarified lines of responsibility and eliminated organizational "stovepipes" by consolidating planning, budgeting, and analysis into a single business administration office. The reorganization reduced management layers, although staff levels remained the same. EERE developed a new IT report to improve program managers' access to EERE cost, obligation, and procurement data. EERE plans to consolidate several legacy IT systems into a single program management system that is intended to track all required information on a project by project basis (cost share, type of contract according to A-11 definitions, etc.). EERE is also developing a measure to reduce uncosted balances, which means obligated funds will be put to use more quickly. These recent actions should achieve efficiencies and improve cost effectiveness, although it will be difficult in some cases to demonstrate definitively.

Evidence: EERE Reorganization "All Hands" presentation: www.eere.energy.gov/office_eere/pdfs/eere_reorg.pdf. EERE IT Business Case Number 019-20-01-12-01-1011-00-304-101. Building Technology Program FY 2003 Financial Status Report (June 2003).

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program coordinates with HUD and the private sector in the Partnership for Advanced Technology in Housing (PATH). Through the efforts of the Association of States Research and Technology Transfer Institute (ASERTTI), coordinated research agendas are developed with the counterpart State research entities. The program has collaborated with industry on the development of several Technology Roadmaps.

Evidence: Building Envelope Technology Roadmap. Windows Industry Technology Roadmap. Lighting Industry Technology Roadmap. High Performance Commercial Buildings Technology Roadmap. PATH website (www.pathnet.org)

Does the program use strong financial management practices?

Explanation: Each year, EERE develops and maintains a Spend Plan and a Measures spreadsheet that links the Spend Plan to annual and long-term goals and measures for each EERE program. The program reviews quarterly costing reports and weekly project status reports. There is no evidence of erroneous payments or statutory violations.

Evidence: FY2003 Spend Plan, Measures spreadsheet, and sample weekly project status report.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The National Association of Public Administrators (NAPA) found dozens of management deficiencies in the program's bureau (the Office of Energy Efficiency and Renewable Energy, or EERE) in a review published in 2000. EERE provided evidence that it addressed some of management deficiencies identified by NAPA, and has prepared a Management Action Plan that will address many of the remaining findings. While a few NAPA recommendations have not been addressed (e.g., that EERE conduct periodic audits to assure that cost-sharing partners actually provide funding they agree to), in general, EERE has taken meaningful steps to address most deficiencies.

Evidence: A Review of the Management in the Office of Energy Efficiency and Renewable Energy (NAPA, 2000). Letter Report from Assistant Secretary Garman to Chairman of the House Subcommittee on Interior and Related Agencies on implementation of NAPA recommendations (July 11, 2001). EERE Management Action Plan (August 2003)

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: Curently, the program provides much of its research funding to "lead labs," which represent centers of knowledge in particular subject areas. The program reports that it is moving away from the "lead lab" concept in favor of more competitively funded research, including having national labs compete against each other. In addition, in 2003, the program developed a Quality, Control and Evaluation plan for each of its four teams, which is used to maintain quality and timeliness of the program's R&D. Despite these advances, the program could not document the conduct of its R&D activities in accordance with OMB Circular A-11 definitions (e.g., merit-reviewed with limited competitive selection, Congressionally directed, etc.). The program could also not demonstrate that research stage (basic, applied, development, demonstration) correlated with statutory and Administration guidelines for cost sharing.

Evidence: The program estimates that less than half of its FY 2003 funding was competitively awarded or supported work at the national labs.

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation:  

Evidence:  

Does the program adequately define appropriate termination points and other decision points?

Explanation:  

Evidence:  

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: The program's long term measures and targets have been modified for FY 2005. The program has not developed good indicators, so progress toward achieving the new long-term goals is difficult to assess. However, in FY 2003, the Residential Building Integration Team reportedly passed one milestone, the 30 percent energy savings in building design packages, on the path to designing net Zero Energy Homes, a key long-term goal. In addition, the National Academy of Sciences reviewed a small subset of historic program activities and concluded that the program has produced several technological successes in energy efficiency that have saved energy for the nation and energy costs for consumers.

Evidence: EERE FY 2005 OMB Budget Submission. Energy Research at DOE: Was It Worth It?: Energy Efficiency and Fossil Energy Research 1978 to 2000 (NAS, 2001).

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program's annual performance measures for R&D activities are under development. The program reports that it has met recent targets for development appliance standards, although it's not clear that these targets were ever explicitly identified in budget documents, GPRA performance plans, or other materials.

Evidence:  

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The program largely supports R&D, and could not demonstrate improved efficiencies in achieving its long-term goals, which have been modified this year. For development of codes, the program reports that it has instituted a process improvement initiative to better collaborate with industry. The reported result is that the time to create a standard has been reduced from five plus years to three years or less. The program did not provide evidence of an accelerated timeline.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: The program coordinates with, but does not compete with, other Federal, state, and private activities.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The National Academy of Sciences "found very positive returns on a relatively modest federal investment for all but one of the projects reviewed." NAS reviewed only seven Buildings projects out of hundreds that DOE has pursued since the 1970s. The program also reports that it has received nine R&D 100 awards, 15 Energy 100 awards, seven Popular Science "Best of..." awards, 10 Excellence in Technology Transfer awards, and over 90 patents. The awards indicate external recognition for program accomplishments, but it's difficult to assess whether the numbers are impressive given the investment of more than $7 billion since the program began. It may be useful to benchmark awards/patents per dollar invested against similar applied R&D programs.

Evidence: Energy Research at DOE: Was It Worth It?: Energy Efficiency and Fossil Energy Research 1978 to 2000 (NAS, 2001).

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation:  

Evidence: