Detailed Information on the
NASA Astronomy and Astrophysics Research Assessment

Report for major missions on: estimated mission lifecycle cost upon entering development; key schedule milestones associated with each mission phase for those missions formally approved for formulation; mission cost and schedule progress achieved in each phase before entering the next; and any plans to re-baseline lifecycle cost and schedule.

Improving flight project cost and schedule performance by changing mission plans, scope, partners, and management where appropriate.

Improving performance of partners (including grantees, contractors, cost-sharing partners, and other government partners) towards achieving cost and schedule goals.

Establishing means to maximize return on available resources for flight and research projects as well as metrics to measure efficiencies gained.

Making grantee annual performance data available on the NASA web site.

Measure: Cumulative percentage baseline cost overrun for spacecraft under development.

Explanation:Astrophysics projects exceeding $250M lifecycle cost (LCC) will not exceed their baseline costs, on average, by more than 10% cumulatively.

Measure: Cumulative percentage baseline schedule slip for spacecraft under development.

Explanation:Astrophysics projects exceeding $250M lifecycle cost (LCC) will not exceed their baseline schedules, on average, by more than 10% cumulatively. Schedule is measured in months from Confirmation Review to launch.

Measure: Percentage of research budget allocated through open, peer-reviewed competition.

Explanation:Performance measure changed effective FY 2006 to focus on research budget (rather than entire Astrophysics budget) and to increase threshold to 95%.

Measure: Percentage of scheduled operating hours delivered for operations and research facilities.

Explanation:

Measure: Accomplishment of key technology and development activities in support of the Astrophysics Program. FY 2007: Complete GLAST Operations Readiness Review, Complete HST Servicing Mission 4 (SM4) Test Readiness Review, Complete JWST Integrated Science Instrument Module (ISIM) Preliminary Design Review, and Begin Kepler assembly, test, and launch operations (ATLO). FY 2008: Launch GLAST, Complete the JWST Preliminary Design Review, Complete the HST SM4 Pre-ship Review, and Complete the Kepler Spacecraft Integration and Test (I&T) phase. FY 2009: Release the Joint Dark Energy Mission (JDEM) Announcement of Opportunity (AO), Complete the James Webb Space Telescope (JWST) Integrated Science Instrument Module (ISIM) Critical Design Review (CDR), Begin Stratospheric Observatory for Infrared Astronomy (SOFIA) open-door testing, and Complete the Kepler Launch Readiness Review (LRR).

Explanation:Key program milestones are identified for upcoming fiscal years.

Measure: Demonstrate progress in understanding the origin and destiny of the universe, phenomena near black holes, and the nature of gravity. Progress will be evaluated by external expert review and be based on advances beyond the prior year's body of knowledge.

Explanation:

Measure: Demonstrate progress in understanding how the first stars and galaxies formed, and how they changed over time into the objects we recognize in the present universe. Progress will be evaluated by external expert review and be based on advances beyond the prior year's body of knowledge.

Explanation:

Measure: Demonstrate progress in understanding how individual stars form and how those processes ultimately affect the formation of planetary systems. Progress will be evaluated by external expert review and be based on advances beyond the prior year's body of knowledge.

Explanation:

Measure: Demonstrate progress in creating a census of extra-solar planets and measuring their properties. Progress will be evaluated by external expert review and be based on advances beyond the prior year's body of knowledge.

Explanation:

Measure: Reduce number of days to make research award selections with constant level of budgetary and personnel resources.

Explanation:

Is the program purpose clear?

Explanation: The program has a discretely defined purpose that relates directly to the NASA vision and mission statements. Its goals and objectives are linked to specific elements of the Science Mission Directorate, the NASA Strategic Plans, and the President's Vision for Space Exploration. The Vision for Space Exploration instructs NASA to "Implement a sustained and affordable human and robotic program to explore the solar system and beyond". The Astrophysics Division is responsible for certain elements of solar system exploration, but solely responsible for exploration beyond the bounds of our solar system. Accordingly, the NASA Science Plan for NASA's Science Mission Directorate (hereafter, the NASA Science Plan) establishes the goals for the Astrophysics Division to "Discover the origin, structure, evolution and destiny of the universe and search for Earth-like planets". The purpose of the program is to conduct observations, research and development in support of these goals. The Astrophysics Division exploration strategy further refines these goals by detailing four objectives: (1) further our understanding of the origin and destiny of the universe, phenomena near black holes, and the nature of gravity; (2) Learn how the first stars and galaxies formed and how they changed over time into the objects recognized in the present universe; (3) Advance our understanding of how individual stars form and how those processes ultimately affect the formation of planetary systems; (4) Create a census of extra-solar planets and measure their properties. These goals represent a balanced exploration of the universe beyond our solar system as directed in the Vision for Space Exploration.

Evidence: The Vision for Space Exploration (http://www.nasa.gov/mission_pages/exploration/main/index.html); the NASA Science Plan (http://www.nasa.gov/pdf/142303main_2006_NASA_Strategic_Plan_sm.pdf); the Astrophysics Roadmap (http://science.hq.nasa.gov/strategy/AstrophysicsScienceProgram.pdf).

Does the program address a specific and existing problem, interest, or need?

Explanation: The program allows NASA to explore the universe in order to understand "the origin, structure, evolution and destiny of the universe" and to search for Earth-like planets. These topics are not only of interest to the scientific community, but also to the general public (and have been for centuries).

Evidence: The program addresses an existing interest and is documented in the National Research Council's Astronomy and Astrophysics in the New Millennium (http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=9839) and Connecting Quarks with the Cosmos (http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=10079) reports, as well as the NASA Strategic Plan (http://www.nasa.gov/pdf/142303main_2006_NASA_Strategic_Plan_sm.pdf).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: No other US agency currently launches scientific missions to explore the universe. While the National Science Foundation (NSF) conducts astronomical research, it is performed exclusively from ground-based observatories. Some universities also conduct limited studies, including those funded in part or in total by NASA. The goals and objectives of the NASA astrophysics program can primarily be realized through space-based missions. There are no efforts by any federal, state, local, or private entity in the U.S. of the magnitude and scope of NASA's astrophysics program. Astrophysics is a unique, one-of-a-kind program that seeks to achieve both near- and long-term science goals by studying objects and phenomena in the solar system and beyond.

Evidence: The program utilizes multiple space missions to collect a broad spectrum of scientific data. Data from these missions are archived and available for addressing a wide variety of scientific questions. [e.g., Spitzer Space Telescope data archive (http://ssc.spitzer.caltech.edu/archanaly/), Chandra X-ray Observatory data archive (http://cxc.harvard.edu/cda/), Hubble Space Telescope data archive (http://archive.stsci.edu/)] The program also pursues and develops both enabling and enhancing technologies to provide new capabilities to collect data and achieve unique scientific advances. No other program directed at astrophysics supports such a broad range of published goals and objectives. There is generally no redundancy with the National Science Foundation's (NSF) astronomical endeavors: NSF focuses on ground-based astronomy, while NASA's astrophysics program science objectives are generally pursued via space-based investigations.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The scientific design of the program and its ability to effectively and efficiently achieve its goals is continually assessed, adjusted, and optimized based on the advice and counsel of a broad community of experts who have been intimately involved for a number of years. These experts are from NASA and other federal agencies, universities, industry and our international partners. Their input is codified in a roadmap, which outlines the program's missions and objectives as endorsed by the National Academy of Sciences, NASA advisory committees, and the Astrophysics Subcommittee under the NASA Advisory Council.

Evidence: The NASA Science Plan (http://www.nasa.gov/pdf/142303main_2006_NASA_Strategic_Plan_sm.pdf) and the Astrophysics Roadmap (http://science.hq.nasa.gov/strategy/AstrophysicsScienceProgram.pdf) lay out direction for ten years to ensure the program's optimal design. The science community advises NASA on the appropriateness of the roadmap to ensure the Program adequately addresses the goals of the scientific community, and that it will enable NASA to effectively achieve these goals. The recently concluded National Research Council's Performance Assessment of NASA's Astrophysics Program (http://books.nap.edu/catalog.php?record_id=11828) recommended that the Astrophysics Program could be better balanced with respect to mission size (number of large vs. small missions). The Astrophysics Division will use this information as input to future planning cycles and when developing future budgets to increase the program's effectiveness and efficiency. In accordance with the PART Program Improvement Plans, the Science Mission Directorate has also undertaken a commitment to regularly report lifecycle costs (LCC) not only for missions in development, but for those under formulation and operations, as well.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The rigor with which the program is designed, structured, managed and funded ensures that resources will reach the intended beneficiaries and will address the program's purpose directly. The science objectives outlined in the Astrophysics Roadmap guide the activities of the program and provide the context through which specific research objectives are formulated, science investigations are defined, and missions that address them are planned. Only the activities included in the roadmap are funded, thereby ensuring that all resources are spent to achieve the Astrophysics science goals.

Evidence: The scientific purpose of each mission is well documented [see the Integrated Budget and Performance Document (under "FY 2008 Budget Request" at http://www.nasa.gov/about/budget/index.html), the Strategic Plan (http://www.nasa.gov/pdf/142303main_2006_NASA_Strategic_Plan_sm.pdf), and the Astrophysics roadmap (http://science.hq.nasa.gov/strategy/AstrophysicsScienceProgram.pdf)] and is linked to specific goals and objectives. Funds are issued to the appropriate entity at the mission level or below. Above a certain level, Federal law prohibits the redirection of resources issued for one program to another program without Congressional approval.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program's long-term measures focus on outcomes and meaningfully reflect the program's purpose. The program has four long-term performance measures that direct progress toward the overall purpose of the Astrophysics program, which is to advance scientific knowledge of the universe by: (1) Furthering our understanding of the origin and destiny of the universe, phenomena near black holes, and the nature of gravity; (2) Learning how the first stars and galaxies formed and how they changed over time into the objects recognized in the present universe; (3) Advancing our understanding of how individual stars form and how those processes ultimately affect the formation of planetary systems; (4) Creating a census of extra-solar planets and measure their properties. Both the long-term and annual measures call for a panel of external experts to review the growth of knowledge resulting from NASA and NASA-funded efforts. This annual external review details the advances being made in each of these four areas.

Evidence: The long-term performance measures, or outcomes, flow directly from the Strategic Plan goals and subgoals, which can be viewed at http://www.nasa.gov/pdf/142303main_2006_NASA_Strategic_Plan_sm.pdf.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Astrophysics program seeks to answer what are among the most profound and long-standing questions posed by humankind. The program's targets and timeframes are also ambitious in that their achievement depends on NASA making significant strides in the development of new technologies for one-of-a-kind spacecraft. The timeframe for the long-term measures, as outlined in the Astrophysics Roadmap (http://science.hq.nasa.gov/strategy/AstrophysicsScienceProgram.pdf) and the Integrated Budget and Performance Document (http://www.nasa.gov/about/budget/index.html) is decadal. Progress against these goals is measured by annual targets calling for growth in knowledge in each of the four Astrophysics science focus areas, or outcomes. Progress is measured against the baseline of the previously existing body of knowledge, which is well-known by the external expert panel and reflected in the existing body of scientific literature. The material reviewed and produced by the panel details and evaluates the advances being made each year.

Evidence: The last decadal survey for Astronomy and Astrophysics in 2000 (http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=9839) recommended an extremely ambitious set of goals. As an example, NASA is making progress toward the top goal in this survey, the largest space telescope ever constructed, the James Webb Space Telescope.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals. The program's annual performance measures support and indicate progress toward addressing its long-term measures. The program long-term measures are supported by annual measures that can serve as indicators of program performance and management. The program's long-term goals are to advance understanding various facets of astronomical and astrophysical knowledge; the program's annual measures support progress toward achieving the long-term goals by assessing yearly increases in scientific knowledge; the degree to which the program reaches programmatic milestones in the development and operation of observatories used to acquire necessary data; and the degree to which the program adheres to cost and schedule goals in the development of observatories, thereby ensuring that resources will be available for future observatories to be developed in a manner that allows for the most expeditious return possible of new science.

Evidence: The program employs different types of annual performance measures to assess progress in mission development, science, and the efficiency and effectiveness with which various portions of the program are executed. One agency outcome involves understanding the origin and destiny of the universe, phenomena near black holes, and the nature of gravity. Progress will be demonstrated through performance on annual performance goals that call for completion of specific mission milestones for the GLAST (Gamma-ray Large Area Space Telescope), Hubble Space Telescope, and JWST (James Webb Space Telescope) missions, as well as one which calls for a panel of external experts to evaluate the given fiscal year's science results in this area. The annual performance goal that evaluates delivery of scheduled hours for operating spacecraft also serves to indicate progress toward program goals. These APGs are part of the FY 2007 Performance Plan, which can be viewed at http://www.nasa.gov/about/budget/index.html under "FY 2008 Budget Request."

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has baselines and ambitious targets for its annual measures. The program management annual measures have baselines and ambitious targets for cost and schedule growth, availability of operating spacecraft, and the percentage of funds awarded competitively. As an example, the measure related to cost and schedule growth establishes a 10% threshold, which is quite stringent when applied to programs charged with building one-of-a-kind spacecraft. Each mission involves a unique configuration of hardware, and, very often, new inventions of hardware, which makes cost and schedule estimation and performance very challenging. Other annual measures call for advances in the understanding of various facets of astrophysical and astronomics knowledge to result from NASA and NASA-funded efforts. These advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of scientific literature.

Evidence: Annual performance goals (APGs) can be viewed at http://www.nasa.gov/about/budget/index.html, under "FY 2008 Budget Request."

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: NASA's Astrophysics Program partners (NASA Centers, JPL, contractors and subcontractors, universities, international organizations and other Federal agencies) are directly involved in planning and establishing the program's goals and objectives. Consequently, they fully support and are committed to the achievement of both the annual and the long-term goals of the program. Both regularly scheduled and ad hoc reviews provide management insight into whether partners are adhering to and supporting the program's goals and objectives. Program goals are made clear to partners. Program missions must document their goals, objectives, and technical deliverables (as they relate to Theme goals) in program plans and commitments, signed agreements between NASA HQ, Program Offices, and the lead NASA center. These documents are available to all partners. The program uses instruments available to government agencies to enter agreements with other entities. The purpose of such instruments is to obtain commitments to working toward and reporting on progress in achieving the annual and/or long-term goals of the program. Letters of Agreement, Memoranda of Understanding, and Technical Assistance Agreements have been signed with major international partners and other Federal agencies. Contracts and grants with industry and universities have been signed and task-level agreements between the Astrophysics division and the Jet Propulsion Laboratory and other NASA centers have also been reached. The program conducts award fee reviews, mid-year performance reviews and ad hoc reviews to determine and verify partner commitment. Independent contract and programmatic reviews are conducted routinely.

Evidence: Signed agreements and other program documents are available to all partners. The GLAST (Gamma-ray Large Area Space Telescope) Memorandum of Understanding with the Italian Space Agency for their hardware contribution to the mission is an example of an agreement that details such plans and commitments.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program effectiveness and program relevance are subjected to regular reviews and evaluations. Annually, an external expert panel evaluates program performance as part of the development of the GPRA-mandated Agency performance report. The panel debates advances in various facets of astrophysics knowledge against the baseline of the existing body of knowledge, which is well-known by the panel's expert members. The panels' assessments, which focus on science results, but also include considerations of progress, or lack of progress, in the entire program portfolio, have included judgements that the program has not demonstrated sufficient results in some areas. Peer review of research proposals ensures that future funding will support high-quality research, as demonstrated by oversubscription rates that range from one in three to one in eight. The Congress has also mandated that the National Academy of Sciences review the performance of each division of NASA's Science Mission Directorate every five years. The Astrophysics Subcommittee of the NASA Advisory Council's Science Committee provides inputs each year on community opinions and recommendations. In addition, every three years, a broad community of experts from NASA, other federal agencies, universities, industry and international partners evaluates the program and offers strategic advice and counsel that leads to a revision of the NASA Science Strategic Plan. At the mission level, Non-advocate reviews and independent cost reviews are conducted to qualify a project to move from formulation to implementation. The Agency implements a quarterly performance review, entitled the "State of the Agency" process to independently evaluate programs and projects. On a monthly basis, an independent assessment team (consisting of PA&E, OCE, OSMA) provides the Agency with a performance assessment and rating for all key programs and projects. The results of the independent assessment (technical, schedule, cost, & programmatic) are reported at the Agency Program Management Council (PMC) and informs decisions at the Agency and the Mission Directorate (program) level on the programs and project directions. For operating missions, external peer-review panels weigh the scientific merits of continued operation of missions that have fulfilled their designed lifetimes.

Evidence: The most recent comprehensive review of the Astrophysics program was conducted in 2007 by the National Research Council. (A Performance Assessment of NASA's Astrophysics Program, http://books.nap.edu/catalog.php?record id=11828) The review was conducted by approximately a dozen external experts, and covered the program's entire portfolio. The National Academy of Sciences also reviews the program as part of its Decadal Survey to help NASA prioritize missions and science objectives for the next ten years. The most recent survey, Astronomy and Astrophysics in the New Millennium, was published in 2001. The Astrophysics Roadmap was created to lay out an effective path to achieve the vision set forth in the Decadal Survey. External expert reviews by subcommittees of the NASA Advisory Council are conducted annually to evaluate progress toward meeting scientific outcomes. Minutes and findings of NASA Advisory Council Science Subcommittees are posted at: http://science.hq.nasa.gov/strategy/comm.html. In addition, the NASA Advisory Council, its Science Committee and the Astrophysics Subcommittee meet three to four times per year to conduct reviews of science and program implementation strategies. Independent program assessment reviews are conducted at critical junctures for all missions that are in development or in space operations. Examples include mission confirmation review, critical design reviews and launch readiness reviews.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: In order to be approved for funding, each mission must demonstrate how it supports the program's long term performance measures, which are assessed and validated by external reviews. Budget requests for each mission are dependent upon the completion of the current year's planned activities and anticipated future requirements. In accordance with recommendations from previous PART reviews, the life-cycle cost requirement for each mission (in full cost) is included in the budget request (Integrated Budget and Performance Plan) in a complete and transparent manner.

Evidence: The Integrated Budget and Performance Document (http://www.nasa.gov/about/budget/index.html, under "FY 2008 Budget Request") displays important status data for each mission, lists the budget requirements for life cycle cost, and identifies the specific long-term outcomes and annual performance goals supported by that mission. Performance against each annual measure is reported to Congress each year in the Performance and Accountability Report (http://www.nasa.gov/about/budget/index.html, under "Performance and Accountability") and budget overruns are reported as required by Congress per the NASA Authorization Act of 2005.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program regularly reviews its strategic planning and utilizes a number of different mechanisms to determine and correct any deficiencies. Experts review program progress, leading to revision every three years of the Science Mission Directorate Strategic Plan, which the National Academy of Sciences then reviews. Astrophysics strategies, missions, and objectives are also reviewed by the Science Committee of the NASA Advisory Council and its Astrophysics Subcommittee. Changes in strategic planning are incorporated into the Astrophysics Roadmap and the NASA Science Strategic Plan.

Evidence: In the Performance Assessment of NASA's Astrophysics Program, the NRC offered three recommendations for improving the strategic planning process. NASA has acted on these recommendations. One deficiency noted was the need for improved cost modeling for use in the Decadal Survey. The improvements NASA is making to provide more reliable mission costs will greatly enhance the Decadal Survey process, from which the strategic planning process flows. Another recommendation called for better balance in the Astrophysics Program with respect to mission size (number of large vs. small missions). The new Associate Administrator for the Science Mission Directorate instructed division directors to examine ways of restructuring their programs to address this recommendation, and they have done so as part of the FY 2009 budget process. The NRC's Performance Assessment will be available at http://books.nap.edu/catalog.php?record_id=11828. Also in response to recommendations, the Astrophysics Program created the Primordial Polarization Program Definition Team (PPPDT) in April 2007. One of the planning deficiencies cited by the NRC is the circuitous path advice had to take to get from the community to NASA managers. The PPPDT is a community committee addressing a specific science area (cosmic inflation) about which the Astrophysics Division needs more information. Rather than follow the formal advisory path for such a specific, but important item (in the Beyond Einstein program), NASA has created this group and will do likewise in similar situations in the future.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: When a program/project is formulated, its concepts, technology requirements, operations concepts, internal management controls, budget and institutional requirements are evaluated by independent bodies. During this period of formulation, design trade studies are conducted in order to reconcile trade-offs between competing performance factors. Programs/projects are subjected to independent reviews throughout their life-cycle to evaluate their ability to meet commitments. Included in these reviews are recommendations for proceeding with, modifying or terminating the program or project, or for enhancing overall technical and programmatic performance.

Evidence: The Astrophysics program evaluated multiple lifecycle investment strategies for the James Webb Space Telescope (JWST). Based on analysis of past internal and external programs, Astrophysics management identified a trend involving advanced technologies on mission critical paths not maturing fast enough for nominal systems integration. This phasing disconnect created disproportionate increased costs and schedule risks late in the program lifecycle. For JWST, Astrophysics rebalanced the traditional budget allocation and investment strategy for advanced spacecraft development by moving more resources (funds & schedule margin) into the earlier phases of the program lifecycle in order to buy down advanced technology risks. Astrophysics will compare cost, technical, and schedule performance of the advanced technology investments in JWST (particularly during system integration) to determine the ROI (return on investment) on this approach vs. traditional budget phasing strategies.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: This question is not applicable. NASA's is a basic research program that is unique in the type of scientific discovery it enables. Its benefit is the generation of scientific knowledge via the use of space-based assets.

Evidence: N/A

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Independent outside organizations review the program and help set scientific priorities in line with Agency goals and objectives. Missions are then chosen to accomplish these scientific priorities. The Astrophysics roadmap presents these missions in a time-ordered sequence subject to future science and technology developments. The NASA Science Plan provides the current set of prioritized missions through 2016. Budget requests consider the relative importance and placement of missions and research based on this strategic plan, along with resource availability. During the budget formulation process, missions provide life cycle cost data and must show relevance to annual and long term goals. The NASA budget process, Planning Program Budget and Execution (PPBE) ensures that prioritizations and strategic goals tie directly to the funding decision.

Evidence: The National Research Council Decadal Survey was delivered in 2001 and is available at http://www.nationalacademies.org/bpa/aanm.html. The next Decadal Survey is scheduled for 2010. The Astrophysics Roadmap is available at http://science.hq.nasa.gov/strategy/AstrophysicsScienceProgram.pdf. Additionally, NASA has implemented Planning Program Budget and Execution (PPBE), a budget process that NASA used to implement and guide the FY2008 budget decisions (http://www.nasa.gov/about/budget/index.html, under "FY 2008 Budget Request"). This process allows NASA to focus additional attention on the translation of strategy into actionable programs, allows for trade-offs, and ensures that NASA, the Science Mission Directorate, and Astrophysics priorities and strategic outcomes are aligned.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects relevant technical and programmatic performance data on a monthly basis. This information is used to assess monthly progress and annual progress toward meeting long-range outcomes, and can be used to develop risk mitigation strategies, adjust priorities, alter resource allocation or take other appropriate management actions. Key data includes the tracking of lifecycle costs (LCC) for missions in formulation, development, and operations.

Evidence: The Science Mission Directorate reviews performance data monthly. Programs over a certain dollar value must exercise a contractor-owned, Agency-approved earned value system and are reported on a monthly basis at all levels within NASA (from the center to Mission Directorate levels). Recently, the Science Mission Directorate reviewed risk mitigation and cost reduction strategies to determine whether and where to make strategic changes. The Astrophysics Division decided to reinstate the SOFIA program following a review in 2006 that found no significant technical problems existed in the program that would prevent the successful completion of the aircraft. This input, combined with a clear statement of the importance of the SOFIA program to NASA's international partner (Germany), helped inform the decision to reinstate the program. The review outlined actions that could improve the management of the SOFIA Program, such as moving the center of operations from one NASA Center to another.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and program partners are held accountable for cost, schedule and performance results through a series of formal procedures and requirements. NASA's annual budget requests clearly identify the federal manager who is responsible for each program and project. Federal managers who fail to demonstrate the required level of performance are subject to a variety of disciplinary actions, including reassignment or termination. Partners who likewise fail to demonstrate the required level of performance may find their level of participation in the program either diminished or terminated. Every manager is required to develop a formal personal performance plan with his or her supervisor. This plan consists entirely of critical elements, at least one of which must be linked to the Agency's Strategic Plan or the organization's operating plan or goals. Although the program's performance may be evaluated on a more frequent basis, the program manager's performance is formally evaluated twice yearly. Bonuses and promotions are dependent upon the manager making positive progress toward meeting the goals of the program. Should he or she fail to do so, corrective actions ranging from counseling, reassignment or, in extreme cases, termination may result. Implementing Centers are held accountable for the successful implementation of their programs. International and federal government partners work in accordance with applicable Memoranda of Understanding (MoUs) and agreements, which detail schedule and performance commitments. Contractors and subcontractors are held accountable for the timely delivery and quality of products. NASA uses award fees to incentivize performance where appropriate. Partners who fail to perform as required may find their participation reduced or terminated. The Astrophysics Program has taken such actions, but is currently examining ways in which more can be done to address contractor performance issues swiftly.

Evidence: NASA Procedural Requirements (NPR) 7120.5D (http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D_&page_name=main&search_term=7120%2E5) establishes the requirements by which NASA will formulate and implement space flight programs and projects, consistent with the governance model contained in the NASA Strategic Management and Governance Handbook (NPD 1000.0). The NPR applies to NASA Headquarters and NASA Centers, including Component Facilities and the Jet Propulsion Laboratory, and contractors/service providers to the extent specified in their contracts with NASA. Recent examples of actions taken to hold contractors accountable for performance issues include: Based on poor performance, management of the SOFIA airplane refurbishment and maintenance contract was removed from the contractor's control and placed under direct government management. The fee was withheld from the contractor for the Kepler project in fall 2006 due to unacceptable cost and schedule performance. Management changes at the contractor were implemented, as well. Due to workmanship and quality issues on contractor instruments (SM4 instrument), the Hubble Space Telescope project pulled hardware development activities back into the government and began ramping down contract staff earlier than originally planned. NASA has also opted to install additional on-site personnel at the contractor worksite for various missions, including JWST, GLAST, and Kepler, in order to improve performance.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Annual NASA funds are available for obligation for two years and are fully obligated by the end of that period. NASA's Program/Project work breakdown structure implemented in FY06 provides more visibility into the expenditure of funds, allowing NASA to verify that funds are being spent for the intended purpose. Operating plans for the program year are submitted to Congress and revised as needed over the two year time period if changes in program expenditures need to be made. Internally, obligation and cost plans are developed, compared to actual spending, and reviewed monthly by all levels of the program. Contractor and government accounting systems are audited periodically to ensure compliance with government standards.

Evidence: Through September 30, 2006, 72% of FY06 program funds had been obligated. Individual program obligations ranged from 41% to 100%, with most programs in the low to mid-eighties range. At the close of the first quarter of FY07, more than 92% of program funds had been obligated. The remainder of FY06 funds will be obligated during FY07. Obligations for the Beyond Einstein program (41%) will increase with the determination by the Beyond Einstein Program Assessment Committee (BEPAC) of the order in which the missions will proceed. Grants programs also typically maintain a larger uncosted and/or unobligated carryover into the next year in order to guard against the possibility of a continuing resolution. It should be noted that in FY06 program obligations were impacted by the extended downtime required for transition of Agency financial systems prior to the close of the fiscal year. Specific reports that record and track the obligation and expenditure of program funds are as follows: contractor monthly and quarterly 533 reports, SF133 reports on budget execution and budgetary resources, FMS2108 year-end closing statement, and the annual Performance and Accountability Report (http://www.nasa.gov/about/budget/index.html, under "Performance and Accountability").

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has achieved efficiencies in some areas and has procedures in place to address issues with cost effectiveness. However, the program can greatly improve such efforts to ensure that efficiencies and cost effectiveness are consistently sought and measured in all areas, without sacrificing quality. The program has a single efficiency measure and would benefit from the addition of others. The program has policies to incentivize competitive outsourcing, best value procurement practices, and employee performance and productivity improvements. Information Technology is used extensively. The implementation of full cost accounting has given program managers a more detailed view of all the costs associated with program activity. During the budget formulation process, Centers provide program managers with the estimated costs of doing business at that Center. Contractors (including the Jet Propulsion Laboratory) are motivated to achieve cost effectiveness and efficiency via fee review. The program has a single efficiency measure that assesses the annual reduction from a baseline level in the number of days required to fund grants assuming a constant level of effort by program personnel.

Evidence: The following operating missions offer examples of improved efficiency and cost effectiveness achieved through IT improvements, cost comparisons, competitive outsourcing, and other means: The Chandra program shows improved efficiency each year. They reduced their labor by approximately 25% from 2002 (mission year 3) to 2006 (year 6) by taking a series of steps including re-engineering operations processes (e.g., automating science data verification and validation, reducing number of on-console shifts), consolidating ground software maintenance (e.g., descoping or ending subcontracts and moving maintenance in-house to SAO), and migrating operational ground systems to more cost effective platforms (SGI to Linux). They were able to achieve these efficiencies while continuing to meet or exceed all mission requirements, e.g., observing efficiency, time for data delivery to observers, on time completion of annual observing cycles, etc. As they proceed to future years with the baseline 2006 labor level, their plan requires continued efficiency improvement while maintaining the mission performance levels. This is driven by the increasing demands of an aging spacecraft: increased numbers and complexity of flight software patches, increased mission planning complexity due to increasing thermal constraints, and more stringent modeling needs, among others. In order to apply the necessary resources to address these issues within approximately constant labor resources, the Chandra team is restructuring in other program areas; examples include further consolidating ground software systems (e.g., science and operations), consolidating CM and administrative processes and systems, and automating science monitoring and trends. As a second example, at the other end of the scale, the Astrophysics Program's "cheapest" mission, FUSE, has achieved even further improvements in efficiency by adopting new software tools for operations planning, including automated short-term scheduling software (SOVA, developed by Wallops with NASA HQ support).

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program collaborates and cooperates, where reasonable and practicable, with other NASA programs and/or Federal agencies where shared or similar goals and objectives might permit a more efficient use of resources while increasing the scientific and/or technological return. In addition, NASA maintains a willingness to collaborate with other nations in exploring the universe where there is evidence of a genuine intersection of interests.

Evidence: The program coordinates and collaborates with the Department of Energy's Office of Science to facilitate studies related to the origin and destiny of the universe as well as the technologies associated with relevant missions to explore those areas that span the boundary between traditional astrophysics and high energy physics. The program also maintains ongoing collaborative relationships with various international partners at the program and project level for planning and coordination. The Astrophysics Division works with the Space Operations Mission Directorate within NASA to insure the successful completion of the final Hubble Space Telescope servicing mission. The Science Mission Directorate Astrophysics Division is working with the Exploration Systems Mission Directorate to identify relevant science enabled by NASA's return to the moon. The Astrophysics Division also closely works with the scientific community through the Great Observatory science centers, smaller mission operations centers, and the Research and Analysis program.

Does the program use strong financial management practices?

Explanation: The most recent Independent Auditor report for NASA identified two (2) material weaknesses, both of which are repeat items, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: NASA's FY 2006 Performance and Accountability Report (www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the Independent Auditor. In addition, the GAO has published numerous reports identifying shortcomings in NASA's new financial management system as well as its financial management processes (example is GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Science Mission Directorate, which manages the program, has a well-structured process by which each of its program is required to present management, technical, and financial status information at monthly reviews held at HQ. The program director has frequent contact with directors of implementing organizations for program projects to discuss and mitigate any management deficiencies, and there is a long tradition of inviting independent bodies to review projects for various deficiencies, including management, and propose solutions to any problems. Lessons-learned workshops are conducted to alert management to the kinds of mistakes that have been made under similar circumstances in the past so as to avoid repeating them in the future. While the program has taken these positive steps to address management deficiencies, the program needs to go further in doing so. In particular, the program has a poor recent record of controlling costs for its flight missions; if adequate steps are not taken to address this problem, resources for new flight missions will be extremely constrained if not unavailable altogether. In addition, the program has only a single efficiency measure and therefore while it can claim successes in achieving efficiencies sporadically, program managers lack a program-wide, coordinated mechanism for measuring efficiencies and cost effectiveness gained across the program.

Evidence: While fundamental program management deficiencies exist, projects within the Astrophysics program's purview, including the James Webb Space Telescope (JWST) and SOFIA, are examples of programs or projects that have recently undergone corrective management actions. These actions included adjusting the levels of programmatic contingency and accelerating technology development for JWST at the recommendation of an external review body and movement of lead center responsibilities for SOFIA aircraft development from Ames Research Center to the Dryden Research Center, again at the recommendation of an external review. SOFIA, an aircraft-based observatory, was to have originally flown first by 2001 and begun operations in 2008. This long delay accompanied by associated cost growth led NASA to initiate several reviews of the science, operations and management of the program. As a result of recommendations flowing from these reviews NASA made major changes to the program both in the nature of the management (privatized to government run) and the location of further major development (Ames to Dryden). These major revisions were deemed optimal for NASA following cost overruns and schedule delays encountered by the program in its previous state.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: Each Astrophysics program project is defined by a plan, which contains a clearly defined list of deliverables, along with the required performance characteristics, costs and schedule goals. Progress is measured by traditional methods such as earned value, schedule accomplishment, independent assessments, etc., in order to determine whether the project's performance requirements and limited window for launch can be met for the anticipated cost or whether the cost is exceeding predetermined limits. The results of these assessments and reviews impact program management decisions. While maintenance of clearly defined deliverables and capability/performance characteristics has been very successful, weak performance on cost and schedule goals has led NASA to make changes in its approach to developing credible estimates. As the program is focused on acquiring one-of-a-kind capital assets that carry a substantial amount of risk to contractors, it therefore does not often use fixed-price and performance-based contracts.

Evidence: Each program project has a clearly defined list of hardware and software deliverables, along with required performance characteristics and costs and schedules that must be developed, documented, maintained and managed. Documentation includes the Program Commitment Agreement, the program plans, and the project plans. The program manages carefully to the information contained within these documents. There is always a hardware descope plan in case the mission is projected to be over budget or has been losing schedule and must take an action to get back on schedule and budget. Any mission that projects cost increases in excess of 15% of total life cycle cost established at confirmation will be subject to a termination review. The NASA Authorization Act of 2005 mandates Congressional notification if projected costs exceed this threshold. Should the projection exceed 30%, funding is halted by Congress and NASA must request reinstatement, with credible plans for keeping cost under control. Recent changes have been made in an effort to improve cost estimating, the most significant of which is the requirement for independent cost estimating to the 70% confidence level for missions with a lifecycle cost exceeding $250M. The Agency has also developed a "State of the Agency" process to independently evaluate programs. On a monthly basis, an independent assessment team (consisting of PA&E, OCE, OSMA) provides the Agency with an assessment. The results of the independent assessment (technical, schedule, cost, & programmatic) are reported at the Agency PMC and the Mission Directorate (program) responds to the independent results quarterly.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The program awards the great majority of its Research and Analysis (R&A) grants (> 97%) based on clear competitive processes that include a qualified assessment of merit. This assessment of merit is based on the following primary evaluation factors: 1) relevance to NASA objectives, 2) intrinsic scientific and/or technical merit, and 3) value and costs. All grants have a specified period of performance, which is typically three years. The grantee must submit annual reports demonstrating satisfactory progress before each year's funding is released. When the specified period of performance concludes, the grantee must again go through the same competitive process as other applicants.

Evidence: The great majority of grants selected for funding by the Science Mission Directorate, including the Astrophysics program, are broadly competed through the NASA Research Announcement process. Grant proposals must relate directly to both Agency and Mission Directorate goals and objectives. All proposals are peer-reviewed by a mix of scientific disciplines and are selected on merit. NASA also utilizes an electronic mailing list as part of its outreach efforts.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: NASA has an oversight practice that provides sufficient insight into and knowledge of grantees' activities. Discipline scientists take the results of the grant peer reviews and select investigation proposals for funding according to their scientific merit, relevance to NASA missions and programs, and availability of funds. These scientists then monitor the progress of the grant toward meeting its stated goals for the duration. Formal annual reports are provided by the grantee, and expenditures are tracked at a cumulative level. This gives the discipline scientists who work with the grantee sufficient insight into the performance to understand what the grantees do with the resources that are allocated to them.

Evidence: The formal annual reports are the primary method through which oversight and management control are exerted on the grantees. All grantees are required to submit annual progress report before the next increment of funding is released to them. These procedures are outlined in Appendix D of the 2007 NRA Proposers Guidebook at http://www.hq.nasa.gov/office/procurement/nraguidebook/proposer2007.pdf.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NASA collects grantee performance data on an annual basis. The agency, however, does not yet make grantee performance data available publicly in full.

Evidence: Formal progress reports, which are a required output of each research and analysis activity, are submitted on an annual basis. The NASA lead scientist, together with appropriate discipline scientists review the progress reports before recommending continuation of the research activity. The results of grants-based research are broadly disseminated to the public through the use of science forums, publications, NASA press releases and news conferences, museum displays, educational materials, and NASA's web site; the agency, however, does not yet make grantee performance data available publicly in full. Due to high-priority requirements on NASA's web-based systems to accommodate e-Gov initiatives, including Grants.gov, and absence of additional funding, NASA has currently suspended work on a publicly accessible database to post grantees' annual reports. Given resources constraints and competing demands, the program has chosen at this time to focus on improving its ability to make selections and initiate funding of grants. The provision of post-award performance data will be revisited later this year when OMB issues instructions regarding standard reporting forms.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The program selects missions using a broadly competitive process and will only sole-source projects on the basis of a demonstrated unique expertise or capability and to maintain technical competencies at NASA field centers. In this way, the program can ensure that its funds are spent on only the highest quality missions. Funding of external sole-source projects is relatively rare; such projects, which are almost exclusively the result of Congressional direction, go through a technical review prior to funding. Funds are allocated to each mission as directed by Congress, and the financial status is tracked closely and reported regularly to program management. The program has the flexibility to re-allocate funds as needed to maintain program quality, and reports these changes to Congress in the Operating Plan. In addition, the peer review process is used extensively and is essential for a high quality, relevant program. The use of external peer review enhances the quality of NASA's investigations and activities because it brings the best and most critical national and international experts to the evaluation process. External peer review ensures that fresh viewpoints, alternative perspectives and state-of-the-art understanding are included in the evaluation process.

Evidence: Regular financial and management status reports allow program directors to make decisions regarding the balance between ongoing and future missions as appropriate. Furthermore, most new missions include extended Phase A and B stages in order to retire technical risk and ensure program quality before going into full development. During development, review boards comprising contractor and NASA personnel conduct Preliminary Design Review, Critical Design Review, and Design Certification Review. This verifies that the "design-to" baseline is established and meets requirements, the detailed design is suitable and the "build-to" baseline is established, and each "as-built" system satisfies the final performance requirements.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: With the reorganization of the Science Mission Directorate (which took place since the last PART assessment in 2004), the Astrophysics theme has created new long-term performance goals that reflect the goals of the new organization. There is much carryover from the long-term goals of the previous themes, in which significant progress has been made.

Evidence: NASA's progress in pursuit of its long-term Astrophysics goals is reflected in the favorable results of annual evaluations performed by panels of experts external to the agency. (See NASA FY04-FY06 Performance and Accountability Reports at http://www.nasa.gov/about/budget/index.html, under "Performance and Accountability.") Remarkable scientific results have led to a new understanding of the fundamental questions about our universe that the Astrophysics program seeks to answer. As an example, results from the Wilkinson Microwave Anisotropy Probe (WMAP) mission have significantly advanced our understanding of the origin of the universe by defining its age far more accurately than ever before. The program has implemented improvements in cost control and schedule performance management for missions in development to minimize impacts to future progress.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Astrophysics has achieved its annual performance goals to a small extent. The program exceeded its goals for delivering scheduled operating hours and competing research funding, but did not achieve its cost or schedule goals. Although progress was made, the program missed its annual efficiency goal for reducing the time required for selection of research grants. Minor delays to the GLAST and Kepler missions and more significant delays to JWST and SOFIA caused the program to miss the annual performance goals for these missions. The program achieved its four annual performance goals calling for advances in: understanding the origin and destiny of the universe, phenomena near black holes, and the nature of gravity; understanding how the first stars and galaxies formed, and how they changed over time into the objects we recognize in the present universe; understanding how individual stars form and how those processes ultimately affect the formation of planetary systems; and creating a census of extra-solar planets and measuring their properties.

Evidence: See FY 2006 data for annual measures in performance measures section.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program does demonstrate, to some extent, improved efficiencies or cost effectiveness in achieving program goals each year. All key elements are in place for future missions to continue returning science data with increased efficiencies and cost effectiveness.

Evidence: The program tries to restrain unwarranted growth in cost and schedule (thereby increasing efficiency) by additional testing, extending initial Phase A and B development, and maturing critical technology off-line until it reaches a mission-acceptable level. Maturing technologies prior to development, as recommended by Government Accountability Office reports, costs more up front, but is expected to save the expenditure of larger amounts later on in the development cycle.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other programs, either government or private, with similar purpose and goals and of similar size and scope, with which to compare the solar system exploration program.

Evidence: N/A

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Rigorous, high-quality, independent evaluations of the Astrophysics program conducted by the National Research Council and external expert panels confirm that the program is achieving results, but express concerns regarding future results.

Evidence: External expert panels conduct independent reviews of the program's annual performance, and reported for FY 2006 that Astrophysics made good progress on its science performance goals, but concerns exist regarding the impact of current cost and schedule overruns on future science results (FY06 Performance and Accountability Report, http://www.nasa.gov/about/budget/index.html). The National Research Council's "An Assessment of Balance in NASA's Science Programs" (2006) (http://books.nap.edu/catalog.php?record_id=11644) cautions that the NASA Science program "is neither sustainable nor capable of making adequate progress toward the goals recommended in...the decadal surveys." For the future, the Congress has mandated that each SMD division's entire portfolio be assessed by the NRC every five years.

Were program goals achieved within budgeted costs and established schedules?

Explanation: The Astrophysics Program continues to produce technologically groundbreaking spacecraft that bring spectacular scientific breakthroughs; however, the program has experienced significant cost and schedule growth in each of its major missions, largely because of their unique requirements.

Evidence: See performance measures section. To improve cost and schedule performance, the Astrophysics Program has undertaken management of lifecycle costs (LCC) for missions in formulation and operations, as well as for those in development. In addition, in the case of its most expensive mission, the James Webb Space Telescope (JWST), the program is adopting a more conservative budget posture by projecting JWST costs to the 70% confidence level.