Detailed Information on the
Energy Information Administration Assessment

Continue increased outreach and collaboration with the energy research community.

Establish baseline data for performance measures and set targets.

Charter an external team for an independent external review of the agency.

Obtain principal economic indicator status for EIA's Weekly Natural Gas Storage Report.

Expand collaboration with broader energy research community by expanding our annual public meeting and developing a new web product.

Measure: Quality of EIA Information Products: 90 percent or more of customers are satisfied or very satisfied with the quality of EIA information.

Explanation:EIA regularly polls our customers, either by conducting our own customer satisfaction surveys, or by participating in surveys administered by others, such as the American Customer Satisfaction Index.

Measure: Cost savings realized from a subset of surveys, released on schedule, without any decrease in accuracy.

Explanation:EIA has baseline cost estimates for conducting three of our surveys. Cost savings will be realized if future costs are less than the baseline adjusted for inflation.

Measure: Percent of outside experts rating EIA's analysis and forecasting program as relevant and reliable; consistent with changing industry markets; and the quality of EIA's analysis and forecasting products high.

Explanation:EIA conducted surveys of two of our more knowledgeable user groups, those registered to attend our National Energy Modeling System conference, and those on our email notification list for our Short Term Energy Outlook. Their responses will be our measure of satisfaction for these more expert customers.

Measure: Percent of key EIA survey frames with sufficient industry coverage to produce reliable supply, demand and price statistics.

Explanation:An internal EIA team evaluated 34 survey frames, and found that 5 were out-of-scope, and 25 of the remaining 29 frames were judged to be sufficient, which is 86%. A complete re-evaluation is not planned for several years.

Measure: Percent of EIA surveys meeting quality targets.

Explanation:As part of our internal quality assessments, each survey manager was asked to select a specific annual goal for their survey. We do not yet have the percent of goals that were met, but expect to have this and future year targets by December 2005.

Measure: Timeliness of EIA Information Products: 95 percent of selected EIA recurring products meet their release date targets (all product types).

Explanation:EIA establishes a list of which products to measure and a calendar of their scheduled release dates at the start of the calendar year. The frequencies of the products being tracked range from weekly to multi-year.

Is the program purpose clear?

Explanation: EIA collects, evaluates, assembles, analyzes, and disseminates to the public and private sectors unbiased and comprehensive energy data, information, and analysis on energy resource reserves, production, demand, technology, related economic and statistical information, and adequacy of the resource base to meet near and long term demands. EIA's mission is to promote sound policymaking, efficient markets, and public understanding.

Evidence: 1977 Department of Energy Organization Act 42 USC 7135: requires that EIA collect, evaluate, assemble, analyze, and disseminate to the public and private sectors unbiased and comprehensive energy data, information, and analysis on energy resource reserves, production, demand, technology, related economic and statistical information, and adequacy of the resource base to meet near and long term demands.

Does the program address a specific and existing problem, interest or need?

Explanation: EIA provides comprehensive and unbiased national and regional energy information that the private sector will not or can not provide because: 1) revenues from disseminating the information would not cover the costs of collecting it; and 2) much of the information is proprietary. EIA provision of data ensures bias is avoided, helping to inhibit market manipulation. EIA also provides unbiased policy analysis and, uniquely among federal statistical agencies, forecasts (in EIA's case of future energy scenarios).

Evidence: The 1977 Department of Energy Organization Act 42 USC 7135 created EIA to provide unbiased and comprehensive data as such information was unavailable to the government and public. The Oil Daily on market impact of Weekly Petroleum Supply Report and Weekly Natural Gas Storage Report: this trade journal indicates that EIA's provision of certain data is crucial in ensuring that oil and gas markets are efficient. Barclays Capital Research April 15, 2004 (proprietary): this analysis provides expert opinion that energy data provided by a source other than EIA is of such poor quality as to be useless.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: EIA is the only organization that collects, analyzes, and disseminates comprehensive energy information. In limited cases, private, state or other federal agencies collect energy information. However it isn't comprehensive, lacks sufficient detail, and may be confidential. EIA works to ensure that it doesn't duplicate such efforts, but complements them, minimizing burden on industry. While a few private firms do proprietary forecasts, they do not place their models in the public domain. Thus biases are unknowable to the general public, and access may be limited due to cost.

Evidence: Electric Power Surveys; lack of other comprehensive energy data.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence that any other method for data collection would be more effective.

Evidence: Participation on Interagency Council on Statistical Policy and Federal Committee on Statistical Methodology; American Statistical Association Committee on Energy Statistics. Through participation in these organizations, EIA ensures that design flaws do not exist.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: EIA focuses resources on critical industry sectors, while maintaining a comprehensive balance. EIA reallocates resources to address gaps, such as coverage of the electricity and natural gas sectors, as those industries have changed with restructuring and deregulation. EIA reaches a broad audience, pioneered usage of a website in 1995 for web dissemination, reaching a web audience of 1.5 million in 2004. Customer satisfaction surveys consistently rate the program as very strong/excellent.

Evidence: Annual Customer Satisfaction Surveys, 2001 American Customer Satisfaction Index survey, budget documents, NEMS/AEO Conference satisfaction survey, website statistics.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: EIA has established longterm performance measures, which support EIA's mission to promote sound policymaking, efficient markets, and public understanding. They include: Percentage of EIA survey frames with sufficient industry coverage; Percentage of outside experts rating EIA's analysis and forecasting program as relevant and reliable; Percentage of EIA surveys meeting quality targets; Percentage of customers satisfied with the quality of EIA information; Percentage of recurring products meeting release date targets; and Sub-Set of surveys released on schedule at a cost-savings, with no decrease in accuracy. These are ouput measures as opposed to outcome measures because it is inherently difficult to measure sound policy making, efficient markets, and public understanding. These measures address quality, timeliness, and relevancy of EIA's products, all necessary attributes of information which will contribute to sound policy making, efficient markets, and public understanding.

Evidence: See measures tab.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: While EIA has ambitious targets for customer satisfaction (maintaining rating of over 90% satisfied or highly satisfied), targets have not been set yet for the remaining four measures which are new measures.

Evidence:

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: EIA is developing annual performance measures.

Evidence:  

Does the program have baselines and ambitious targets for its annual measures?

Explanation: EIA is developing baselines and targets for its annual measures.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: EIA will award its first full performance-based contract on September 1, 2004. This award will be the first full performance-based award that links the contractor's performance directly to EIA's annual and long-term goals.

Evidence:  

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: EIA actively seeks specific advice from experts on projects of current interest, particularly methodology, twice per year from the American Statistical Association (ASA) Energy Committee. EIA staff develop information concerning projects of current interest, and seek specific advice, particularly concerning methodology. Meetings are open to the public. EIA posts meeting transcripts as well as EIA's follow-up actions to recommendations, on the EIA website. EIA also has an Independent Expert Review (IER) process. This review program contracts, on an as needed basis, with academics or consultants to examine a particular process or report. Additionally, EIA's forecasting model is in the public domain, and therefore available for public review and comment. However, no recurring independent evaluations of sufficient scope (program-wide and addressing achievement of performance targets), effectiveness, or independence occur.

Evidence:  

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: EIA's budget request makes clear the impact of funding and legislative decisions on expected performance and explains why the requested performance/resource mix is appropriate

Evidence: FY 2005 Budget Documents.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: EIA routinely seeks input from customers, stakeholders, and staff. EIA is making good progress in setting new long-term performance measures and is developing annual performance measures, baselines, and targets. EIA is also seeking to initiate independent, broad program evaluations.

Evidence: EIA Strategic Plan Stewardship Information; conversations with OMB.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: While EIA is in the process of setting new performance measures, historically EIA has collected timely and credible performance informations. EIA survey managers monitor response rates, release times, revisions, and other quality measures of their surveys on an ongoing basis. Surveys conducted under contract report basic quality measures. For both managers and supervisors, EIA explicitly incorporates quality measures into its performance evaluations. EIA has some performance-based contracts and is encouraging more. Customer satisfaction surveys are monitored, as are comments received and website usage by product.

Evidence: EIA 2002 Standards Manual; Survey Response Rates; Data quality summary for the 2001 Residential Energy Consumption Survey; Website usage statistics. The Standards Manual sets detailed, specific quality guidance, which managers apply.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: EIA's Federal employees are held accountable through individual performance plans. EIA's first full performance-based contract will be awarded in September 2004. The award will have a base period of sixteen months and three one-year options. EIA will evaluate the contractor's performance/contributions to EIA's goals annually and discontinue the contractor's services after any one-year interval inwhich the contractor fails to meet performance requirements. I

Evidence: Performance Plans.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: EIA assures obligated funds are used for their intended purpose through the use of several fiscal oversight processes. These include: Monitoring the obligation of funds against a program specific Annual Operating Plan; Monitoring the use of uncosteds against an annual utilization plan;Small Business goals, monitored and reported monthly;Monthly review of EIA financial activities with DOE financial reports; Monthly status of obligations, uncosted and deobligated balances reported to EIA Administrator.

Evidence: Annual Operating Plan; Obligation, Uncosted and Small Business monthly report; Example of Individual Office report on Obligations, Uncosteds, and Small Business goals; Example of the monthly DOE Financial Data Warehouse (FDW) report on 1) small business, 2) Uncosteds.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: With the recent award of the EIA Omnibus Procurement instrument (EOP II), EIA has a series of contractors for each functional area (survey, IT, Administrative support, etc.), which are pre-approved to compete for tasks and EIA mission support activities. With virtually all EIA work (labor hours and services) competed among these pre-approved contractors, value and quality for each dollar expended is enhanced. Additionally, EIA has an efficiency long-term measure to produce a set of surveys in Petroleum or Natural Gas, and Coal or Electricity at a cost-savings with no decrease in accuracy.

Evidence: Strategic Plan; EIA Omnibus Procurement.

Does the program collaborate and coordinate effectively with related programs?

Explanation: EIA actively collaborates with state statistical agencies and professional organizations, sharing research, methods, and software. This is evidenced by its relationship with the IOGCC. Specific examples of EIA's cooperation with Federal agencies include: working with Census as the data collection agent for its Manufacturing Energy Consumption Survey; contracting with Census to evaluate five of our survey frames; and incorporating data collected by the U.S. Department of Health and Human Services into its Residential Energy Consumption Survey. EIA also particpates monthly in the OMB-led Interagency Council on Statistiacl Policy (ICSP).

Evidence: www.eia.doe.gov/pub/oil_gas/natural_gas/analysis_publications/ngprod/ngprod.pdf. Report on "How EIA Estimates Natural Gas Production" showing EIA uses data from MMS and State of Texas describes how EIA collaborates and coordinates with States and the Department of the Interior's Mineral Management Service on natural gas production data. While the data could be more rigourous, this is an example of how EIA prioritzes scarce resources, by using data collected elsewhere. tonto.eia.doe.gov/FTPROOT/Consumption/063297.pdf from RECS 1997 page 224 for LIHEAP data showing coordination with the Department of Health and Human Services - EIA collected data for DHHS.

Does the program use strong financial management practices?

Explanation: EIA's efforts to ensure its fiscal and oversight rsponsibilities include: Monthly and annual close-out reconcilations between DOE financial management systems and EIA's Contracts Management System (CMS); EIA senior management have real-time access to status of funds status; Continuous monitoring of obligations, uncosteds reductions, and small business obligations and their tracking with approved plans; and mandatory verification of fund availability prior to obligations.

Evidence: Intranet Financial Status Report (FSR); Finance Division Monthly Contracts Management System and DOE financial management system reconcile report.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: EIA actively seeks to improve its efficiency as an organization. Its goals in this regard are outlined in the Strategic Plan. Specific efforts to improve efficiency include, making office directors responsible for tracking their uncosteds, holding bi-annual management meetings, and making performance measures a key portion of all levels of management's performance evaluations. In addition, EIA's quality council has undertaken several initiatives to improve the organization.

Evidence: Obligations report; List of Current Activities of EIA Quality Council; EIA Quality Council Charter

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: EIA is revising its performance measures to better focus on managing for results. EIA believes these new goals will strengthen future performance. EIA does have historical data for two of its long-term measures which indicate EIA would have met the targets. Furthermore, in regards to the previous performance measures scheduled for retirement, EIA did meet or surpass the targets for these measures through 2003.

Evidence: Consolidated Quarterly Performance Report Second Quarter 2004 - EIA green rating; Customer Survey Results - 2001 Amercian Customer Satisfaction Survey found that EIA is delivering service equal to the private sector, and better than the government-wide average.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Prior year peformance goals were met. For the measures currently under development, performance results are not available. In the area of outreach, two of EIA's previous goals and annual targets were met.

Evidence: EIA performance measures are in the process of being revised. EIA, its contractors, and partner agencies collect basic data quality measures like response rates and timeliness. However, EIA's new aggregate measures and targets are not at the stage where performance mesures for these areas are available. DOE quarterly reports show EIA has met performance goals for measures in use during 2003.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Despite declining real budgets (EIA's FY 2005 Budget request represented a 54% real decrease over the 1980 level) and steadily falling numbers of full-time employees (between FY 1990 and FY 2004 EIA reduced its Federal staff by 100 FTEs and its contract staff by over 160 positions), EIA has increased its output. Illustrative examples of increased output include EPACT 1992 requirements for Alternative Transportation Fueled Vehicles Survey (EIA-886) and Voluntary Greenhouse Gas Reductions Registry; and the Weekly Natural Gas Storage Report.

Evidence: Budget documents; GAO Report "Electricity Resturcturing" www.gao.gov/new.items/d03586.pdf, found that regarding electricity data, EIA had increased the number of entities it reports on and the amount of information it collects.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: EIA compares favorably with other federal statistical agencies. EIA participated in the American Customer Satisfaction Index (ACSI) in January 2001. The overall index score for EIA's website users was 73 on a 0-100 point scale, which compares favorably with other statistical agencies. It was approximately 4 points higher than the 2000 Federal government index of 68.6. EIA plans to participate in the ACSI in 2004 and every 3-4 years thereafter. EIA is the preferred source of data for market participants and analysts."Time" cited EIA's Web site as one of the Best Web site on the Internet. Results of EIA's customer satisfaction surveys show the majority of customers have favorable opinions of EIA. EIA is unique among statistical agencies because of the amount and transparency of its energy analysis and modeling programs.

Evidence: Barclay's Capital Research, April 15, 2004; Customer survey results. The American Customer Satisfaction Index 2001 which allows comparison to other organizations doing similar work, found that the quality of EIA's information was very strong or excellent (83). EIA intends to participate in the ACSI every three years, with the next survey in 2004.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: While EIA appears to be effective and achieving results, based in large part on universal reliance on EIA's data and analyses, no independent evaluations of the entire program have been conducted.

Evidence: ACSI Customer Satisfaction Survey; Barclay's Capital Research April 15, 2004.