Detailed Information on the
FAA Facilities and Equipment Assessment

Continue efforts to develop better internal financial management standards and controls to validate the basis for estimating capital program costs and benefits.

Improve contract management discipline by increasing the use of performance-based contracts and employing an incremental lifecycle approach.

Revise the Acquisition Management System to ensure it aligns with the government standards for justifying capital investments in FY 2005.

Measure: Rate of fatalities per 100 million persons on board.

Explanation:The aviation safety measure formerly was the U.S. commercial fatal aviation accidents per 100,000 departures (last 3-years' average). Beginning in FY 2008 this measure has been changed from the number of accidents to the number of fatalities. Since all fatal accidents, as defined by the National Transportation Safety Board (NTSB) criteria, are weighed equally, the result is that an accident with a single fatality is viewed in the same way as an accident involving hundreds of passengers. The new metric is more relevant to the flying public, as it is a better measure of individual risk.

Measure: Fatal general aviation accidents

Explanation:

Measure: Flights arriving on time (within 15 minutes of schedule) at 35 Operational Evolution Plan airports

Explanation:This measure tracks flight arrivals at the 35 largest airports

Measure: For major systems acquisitions, percentage of schedule goals that are met

Explanation:this measure tracks the achievement of annual milestones within a major system acquisition

Measure: For major systems acquisitions, percentage of cost goals within 10% of target

Explanation:this measure tracks the achievement of annual cost goals within a major system acquisition

Measure: Capital Investment projects that exceed either cost or schedule by 10%. Baseline and target under development.

Explanation:Tthis measure tracks projects' status against baseline cost and schedule goals for the duration of development and deployment

Measure: Air Traffic Organization (ATO) overhead rate

Explanation:This measure tracks ATO overhead obligations, to include Service Area, Service Unit and HQ Overhead, as a percent of total ATO labor obligations. It replaces the old efficiency measure - Reduce the number of baseline modifications.

Measure: Percentage of JRC approved acquisition programs projected to complete within 10% of total cost and schedule baseline.

Explanation:This metric will report, as a percentage, the number of active, JRC approved acquisitioin programs forecast to be within 10% of their cost and schedule baseline at completion. Active acquisition programs are defined as those with F&E funding in the reporting year.

Is the program purpose clear?

Explanation: The mission of Facilities and Equipment program is to provide development and acquisition for products and services that enable the FAA to enhance the safety of the NAS and satisfy current and future operational needs of the U.S. civil aerospace system for national and international operations.

Evidence: Federal Aviation Act (P.L. 85-726). Part A of subtitle VII of Title 49, United States Code. Omnibus Bill 1999. FAA Order 2500.8A: Funding Criteria for Operations, Facilities and Equipment (F&E), and Research, Engineering and Development (R, E&D) Accounts. FY03 Performance Accountability Report.

Does the program address a specific and existing problem, interest or need?

Explanation: The F&E Program targets shortfalls (mission needs) in the operational capabilities the FAA needs to perform its mission. The program provides the necessary equipment and facilities for the FAA to fulfill its mission for a safe, secure, and efficient National Airspace System.

Evidence: Federal Aviation Act (P.L. 85-726). Part A of subtitle VII of Title 49, United States Code. Flight Plan 2004-2008. FY03 Performance Accountability Report, pages 6 and 83.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The FAA has sole responsibility for the national airspace system. If the program did not exist, no other public or private organization could take its place. FAA's Facilities and Equipment (F&E) program provides improvements to the National Airspace System (NAS) to accommodate demands for aviation services, to maximize operational efficiency and to replace or modernize aging equipment and facilities.

Evidence: Capital Investment Plan. NAS Architecture. Operational Evolution Plan.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program focuses on acquisition, instead of changes to operating outcome. The program should use spiral development techniques to validate operating outcome before deployment commitments. Capital funding processes have driven management priorities towards discrete priorities rather than comprehensive, integrated, and on-going refresh of operating capabilities. In addition, contract management disciplines should be more widely use, defined, and measured.

Evidence: Acquisition Management System. Capital Investment Plan. NAS Architecture. F&E Management and Control briefing.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: FAA uses a decision tool during the annual budget review process to support annual resource allocations. The tool weighs each project's contribution within a program office to agency goals and civil aviation needs relative to other. However, often projects consistently experience large cost/schedule overruns and the relative benefit is unclear, but FAA rarely terminates or discontinues programs. Furthermore, since the F&E program does not focus on achieving outcomes, it is unclear if resources reach the highest percentage of target beneficiaries.

Evidence: Capital Investment Plan. NAS Architecture. FAA FY 2004-2015 Aerospace Forecast. Expert choice, Budget formulation, Baseline Variance (IBEAM)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The F&E Program's long-term performance measures tie to the FAA Flight Plan 2004-2008. These measures focus on increasing aviation safety, creating greater capacity, and ensuring organizational excellence.

Evidence: Flight Plan 2004-2008. F&E Performance Goal Chart.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The F&E Program's long-term targets are challenging, ambitious and that align to the FAA corporate-level strategic plan.

Evidence: Flight Plan 2004-2008.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The F&E program only had one goal in FY 2005--percent of major system acquisitions that are on schedule and within budget. During the PART review, FAA developed additional annual measures to more comprehensively measure program performance, which will be included in the FY 2006 Congressional Justification.

Evidence: FY 2005 FAA Congressional Justification and draft FY 2006 Performance Goals.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and ambitious targets are only established for one annual measure. FAA will establish baselines and targets the new annual measures in the FY 2006 Budget.

Evidence: FY 2005 FAA Congressional Justification

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All partners are committed to achieving F&E program element goals. The FAA's NAS Architecture and Operational Evolution Plan are developed in close collaboration with the aviation community, DoD, and NASA. RFPs reflect required contractor performance in support of the overall F&E mission and contractual incentives are included as appropriate. It is not feasible to hold contractors accountable for long-term goals (i.e. reduction in fatal accident rate) over which they have no control. Contractors play no role other than providing equipment to achieving those objectives.

Evidence: NAS Architecture. Target System Description. Operational Evolution Plan. Acquisition Management System. Capital Investment Plan. RTCA stakeholders. Joint Planning and Development Office charter. Memorandums of Understanding [National Weather Service, NASA, DOD]. Incentive-type contract vehicles [ERAM sample].

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The F&E Projects are reviewed on a regular basis by the GAO, IG and other independent entities. These studies are sufficient in scope and quality; they typically focus on specific projects. Council or other organizations.

Evidence: Evaluation of FAA Acquisition Reform-The First Three Years: April 1996-March 1999, (recommendations page iv and 6). GAO reports: FAA's Modernization Efforts (10/2003); Current Efforts and Proposed Changes to ATC System (5/2003). GAOReducing Delays and Congestions (5/2001). FAA IG: Status of FAA's Major Acquisitions (6/2003).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The draft FY 2006 Congressional Justification request ties resource requirements to accomplishment of annual and long-term goals.

Evidence: Draft FY 2006 FAA Congressional Justification

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The FAA is working to implement a well-defined, standard process for developing operations and maintenance cost assumptions, estimating life cycle costs, and determining program outcomes. It will apply lifecycle techniques that allow usable segments of the development effort to be addressed to the programs that are to be rebaselined.

Evidence: FAA FY2003 Performance and Accountability Report. Flight Plan 2004-2008. Administrator's Testimony, March 17, 2004. SPIRE tracking and reporting tool [pbViews]. Brief to OMB on the Air Traffic Organization.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: All projects of the F&E Program collect and analyze cost, schedule, and technical performance data from contractors on a monthly basis. ANSI Standard Earned Value Management is applied against 66 percent of FY 04 major acquisitions contract dollars. This data is used 1) to manage the project, identify areas needing improvement, and target corrective action; 2) brief the ATS Board on program performance (quarterly); 3) brief the Administrator on performance (monthly); 4) track and report goals (quarterly); and 5) to support semi-annual JRC acquisition reviews.

Evidence: ARA FY03 Performance Plan. ARA Quarterly Performance Status Reports. ATS Performance Plan FY 2001-2003. Air Traffic Services Subcommittee metrics. Joint Resources Council (JRC) presentations. SPIRE status formats (sample smart sheets). Acquisition Baseline Management Process. Performance Dashboard (ATOP).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Managers are held responsible for achieving cost, schedule, and performance targets through the Performance Management System Plan, which is evaluated semiannually. The FAA's Core Compensation Plan, the Organizational Success Increase reward system, and the performance standards in the Executive's Short-Term Incentives program. For program partners, incentives for good performance are embedded in contract documents, when appropriate.

Evidence: Performance Management System documentation. ARA and ATS Senior Executive Service Short-Term Incentives. Organizational Success Increase documentation. Core Compensation Plan documentation. Acquisition Management System incentive clauses (ERAM and ATOP examples). Service contract (TAC2 example). Memorandum of Understanding (National Weather Service example).

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The F&E account is a three year appropriation. 84% is obligated the first year, 10% the second year, and 6% the third year. Each fiscal year the program offices develop commitment and obligation plans (spend plans) for the F&E Program. These plans are tracked monthly and analyzed to determine if targets will be met. Deviations from plan are noted and corrective actions taken as required to improve performance.

Evidence: Monthly Financial Commitments and Obligations Reports. ATOP Program Obligation Plan. Overall Obligation Plan. End of the year F&E Obligation Report.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program's procurement planning processes includes 1) a market analysis to determine the availability of commercial items and their cost; 2) vendor selection and award based on a determination of best value. However, these procedures to not measure efficiencies and cost effectiveness. FAA has two efficiency measures: cost and schedule variance from basline and reduce contract administration costs.

Evidence: Acquisition Management System (samples of source selection documentation, pre- and post-award audits, market analysis, SIR1 and SIR2, investment analysis report). OMB Circular A-76 documentation. Draft FY 2006 Efficiency Goals.

Does the program collaborate and coordinate effectively with related programs?

Explanation: During the annual budget formulation process, FAA's F&E and Operations collaborate to ensure that sufficient F&E funds are allocated to support new systems to be fielded. However, while there are procedures in place to account for operations and maintenance costs, the operations account can not demonstrate savings through operational efficiency and effectiveness as a result of the deployed technology. The acquisitions and operations programs have been managed separately; therefore, program success was driven by acquisition completion rather than measured change in operating outcome.

Evidence: Memorandum of Agreement for ASR-11. North American Aerospace Surveillance Council documentation. Memorandum of Understanding and working group minutes (National Weather Service, National Aeronautics and Space Administration). OEP, Integrated Program Plan, IOT&E Overview

Does the program use strong financial management practices?

Explanation: FAA is currently implementing a new financial system--Delphi. FAA has developed a scorecard and is tracking its efforts to address all transition issues and FAA has made great strides in this area. However, because it is still reconciling data, the F&E program does not yet have financial information that is accurate and timely. FAA is confident that these outstanding issues will be resolved in early FY 2005.

Evidence: FAA FY2003 Performance and Accountability Report (which includes the independent KPMG audit report and the DOT OIG letter dated December 19, 2003). Status Reports on Delphi.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The F&E Program continually reassesses its management processes using the FAA-integrated Capability Maturity Model (iCMM). This system identifies and helps correct program management deficiencies and uses the system to make corrections.

Evidence: Office of the Inspector General, Report AV-2003-045, Status of FAA's Major Acquisitions (June 27, 2003). FAA-integrated Capability Maturity Model (iCMM) v2. ECG IPP, F&E Performance Goal Chart, iCMM description for PART, iCMMv2 Quick Reference Guide.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The F&E Program is making significant progress in achieving its long-term goals.

Evidence: Capital Investment Plan. Flight Plan 2004-2008. FAA FY2003 Performance and Accountability Report, p. 11. Operational Evolution Plan.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The F&E program has demonstrated results on only one of its annual goals: for Acquisition programs (80% of acquisition resources in FY 2003) 88% of the goal to meet annual cost goals within 80%; 77% met the goal on achieving annual schedule milestones within 80%. This goal was only partially achieved.

Evidence: DOT Performance & Accountability Report FY 2003. FAA Year-End Report. FAA FY2003 Performance & Accountability Report. (The F&E Program supports seven of the twelve goals referenced in this report.) Operational Evolution Plan.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: During FY 2003, FAA is currently conducting an A-76 competition for operating automated flight service stations, which requires a mandatory savings of 22% (operations and capital). FAA is also in the process of developing an effeciency measure that will focus on increasing the number of performance-based contracts.

Evidence: A-76 competition on Flight Service Stations. FAA FY2003 Performance & Accountability Report (pp. 29-30 on A-76).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: GAO began an evaluation of FAA's investment management practices in FY 2003, using GAO's information technology investment management (ITIM) maturity model. The FAA's independent program evaluation staff (ACM) conducted an evaluation of FAA's Acquisition Management System (AMS) compared to the FAA's pre-AMS acquisition system, which was based on DoD's implementation of Circular A-109. The ACM study concluded that the FAA's AMS process saved time over the DoD-based system. However, FAA's ATC modernization program has been on the GAO High Risk List since 1995.

Evidence: Information Technology Investment Management (ITIM) self-assessment submitted to GAO. ITIM Guidance. ACM Evaluation of FAA Acquisition Reform: The First Three Years. OMB Circular A-76 documentation. GAO High Risk Series.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The F&E Program is subject to independent evaluations by FAA stakeholders, the ATS Subcommittee, and independent internal and external organizations. Although some of these evaluations have identified shortcomings, the F&E Program as a whole is effective and achieves results.

Evidence: FY05 F&E Program Budget Formulation Process Paper. Expert Choice Briefing. Air Traffic Services Subcommittee minutes. Monthly Administrator status reviews. List of ACM-10 evaluations. Acquisition reviews - minutes. OIG Report, FAA Oversight of Cost-Reimbursable Contracts, FI-2002-92, May 2002. FAA FY2003 Performance and Accountability Report.