Detailed Information on the
NASA Earth-Sun System Research Assessment

Report for major missions on: estimated mission life cycle cost upon entering development; key schedule milestones associated with each mission phase for those missions formally approved for formulation; mission cost and schedule progress achieved in each phase before entering the next; and any plans to re-baseline life-cycle cost and schedule.

Assess the obstacles to improving the hand-off of NASA's research and development to other federal agencies and implement to the extent possible organizational and system fixes to ensure results.

Assure that the priorities developed in the National Research Council's forthcoming Earth science decadal survey are reflected to the extent feasible in the program's portfolio.

Measure: Progress in enabling improved predictive capability for weather and extreme weather events.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in quantifying global land cover change and terrestrial and marine productivity, and in improving carbon cycle and ecosystem models.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in quantifying the key reservoirs and fluxes in the global water cycle and in improving models of water cycle change and fresh water availability.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in understanding the role of oceans, atmosphere, and ice in the climate system and in improving predictive capability for its future evolution.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating). However, for the first year, FY 2005, the rating represents a self-assessment.

Measure: Progress in characterizing and understanding Earth surface changes and variability of the Earth's gravitational and magnetic fields.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating). However, for the first year, FY 2005, the rating represents a self-assessment.

Measure: Level of customer satisfaction with ESS information systems, as measured by a baselined index obtained through the use of externally administered annual surveys.

Explanation:ESS is committed to achieving at least 70% customer satisfaction with its information systems.

Measure: Number of distinct users of ESS data and services.

Explanation:ESS is committed to establishing a baseline for and then expanding the reach and use of its data and services. FY 2005: 2,742,462 accesses, with 192,077 unique users placing orders

Measure: Progress in understanding the structure and dynamics of the Sun and solar wind and the origins of solar variability.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in specifying and enabling the prediction of changes to the Earth's radiation environment, ionosphere, and upper atmosphere.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in understanding the response of magnetospheres and atmospheres to external and internal drivers.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Average number of days to research award selections for 80% of awards.

Explanation:Measures the time from deadline for receipt of proposals to mailing of awards notification letters

Measure: Cumulative percentage baseline cost overrun on spacecraft under development.

Explanation:On average, spacecraft projects in development will not exceed their baseline costs by more than 10% cumulatively.

Measure: Progress in developing the capability to predict solar activity and the evolution of solar disturbances as they propagate in the heliosphere and affect the Earth.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in understanding the role of solar variability in driving space climate and global change in the Earth's atmosphere.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Accomplishment of key development activities in support of advanced observing capabilities to help resolve key Earth Sun System science questions.

Explanation:*For 2006, successfully complete Solar Dynamics Observatory (SDO) structure and begin Integration and Test, complete Operational Readiness Review for NPOESS Prepatory Program, successfully launch STEREO, and complete Global Precipitation Mission (GPM) Confirmation Review. * FY 2007: Deliver Solar Dynamics Observatory (SDO) instruments to spacecraft for integration, Complete Magnetospheric Multiscale (MMS) instrument suite Preliminary Design Review (PDR), Award Geospace Missions Radiation Belt Mapper Phase A instrument contracts, Successfully launch Time History of Events and Macroscale Interactions during Substorms (THEMIS) spacecraft, Release Explorer Program Announcement of Opportunity, Complete Landsat Data Continuity Mission (LDCM) Confirmation Review, Complete Orbiting Carbon Observatory (OCO) Assembly, Test and Launch Operations (ATLO) Readiness Review, Complete Glory mission Pre-Ship Review, and Complete Ocean Surface Topography Mission (OSTM) Critical Design Review (CDR). FY 2008: Complete Magnetospheric Multiscale (MMS) System Design Review (SDR), Complete Phase A for the Geospace Radiation Belt Storm Probes mission, Complete Solar Dynamics Observatory (SDO) Integrated Observatory Performance Test. FY 2009: Complete the Magnetospheric Multiscale (MMS) Spacecraft Preliminary Design Review (PDR), Complete the Geospace Radiation Belt Storm Probes Confirmation Review, Complete the Explorer down-select.

Measure: Accomplishment of key technology activities in support of the ESS Program.

Explanation:* Advance at least two technologies to the point (Technology Readiness Level 6 or greater) that they could be demonstrated in space or in an operational environment, and advance 25% of all funded technology developments at least one Technology Readiness Level.

Measure: Cumulative percentage schedule slip on spacecraft under development.

Explanation:On average, spacecraft development projects will not slip from their baseline schedules by more than 10% cumulatively

Measure: Progress in determining the evolution of the heliosphere and its interaction with the galaxy.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Progress in discovering how magnetic fields are created and evolve and how charged particles are accelerated.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Percentage of budget for research projects allocated through open, peer-reviewed competition.

Explanation:On average, ESS will allocate the targeted level of funding competitively, consistent with the target held by other NASA science programs.

Measure: Percentage of scheduled operating hours delivered for all on-orbit instruments.

Explanation:

Measure: Progress in understanding the coupling across multiple scale lengths and its generality in plasma systems.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Measure: Number of reports issued with partnering organizations that validate that using NASA research capabilities (e.g., observations and/or forecast products) could improve their operational decision support systems.

Explanation:NASA Applied Sciences uses a iterative systems engineering process in collaboration with our federal partners to prototype versions of their decision support systems (e.g., agriculture, aviation, ?? water management) using NASA research results as additional, value-added inputs. The ultimate goal of the activity is to "Benchmark", or measure, the change in performance of the decision support system when it uses NASA research results. If there is a measured improvement in performance, then it would be up to the partner agency to adopt or adapt the capability for their operational use. NASA documents the results of systems engineering activities in reports, i.e. Evaluation Reports, Verification and Validation Reports, Benchmark Reports, which are published with the partners for access by the community at large. The steps from evaluation to verification and validation to benchmarking are a progression in the process of meeting the NASA objective "to improve future operational systems."

Measure: Number of studies completed on plans to transition the results of NASA research and development, including scientific spacecraft and instruments, models, and research results with potential to improve future operational systems of partner agencies.

Explanation:Through capacity implementation plans describing research to operations transition activities, NASA identifies with its partners (e.g. NOAA, USGS, EPA, et. al.) what can and needs to be done as a significant and necessary step in performing a transition from research to operations. The plans include the necessary science, systems engineering, information management, infrastructure, trade study, operations, administrative, and business considerations of a transition, From these plans, all stakeholders to the transition decision are informed about the implementation, execution, and performance ramifications of the transition.

Measure: Progress in understanding and improving predictive capability for changes in the ozone layer, climate forcing, and air quality associated with changes in atmospheric composition.

Explanation:External advisory committees will rate the program's progress toward this measure on an annual basis using a green-yellow-red stoplight scale, where "green" signifies excellent progress (and "blue" is reserved for superior progress, even above the green rating).

Is the program purpose clear?

Explanation: The Earth-Sun System (ESS) Program follows the purpose dictated by the NASA vision and mission statements and the NASA Strategic Plan to better understand our planet, our sun, and the connection between them. The rationale for this multi-faceted purpose can be found in the Earth and Space Science Enterprises Strategic plans. To achieve its mission, NASA pursues ESS science questions for which NASA technology and space-based observations can make a defining contribution. NASA works with the science and engineering communities, other Federal agencies, and international programs to develop and demonstrate new obserivng capabilities from space, pioneer the use of these data to further science goals, and transition mature capabilities and results to operational users.

Evidence: NASA's authorzing legislation (Space Act of 1958), its amendments, and the Clean Air act mandate NASA's activity to conduct space activities to explain knowledge of the Earth and monitor the Earth's upper atmosphere and report to Congress on stratospheric ozone depletion. NASA ESS fulfills its overarching purposes by posing a suite of scientific questions, which are then addressed by numerous scientific roadmaps. These roadmaps break down the overarching purposes of ESS into tractable sections and call for transparent evidence of how the ESS Program will go about providing answers to our science questions as well as enhancing observing capability, and how this information, technology, and capaibility can lead to the realization of societal and economic benefits. NASA's Integrated Budget and Performance Document (IBPD) articulates the rationale for the program.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses specific, identified national needs in several areas including: the causes and consequences of climate change; improvements in the reliability and extension of weather forecasts including space weather; and the monitoring and eventual prediction of natural hazards such as floods, volcanic eruptions, and earthquakes. NASA's role is to develop and make first use of new observing and research capabilities to understand the underlying processes, provide objective scientific information to researchers and decision-makers, and transition mature capabilities and results to operational users. NASA contributes, for example, to reducing space weather uncertainties, thereby improving the performance of future scientific and exploration spacecraft, technological systems on Earth, and the ability to maximize protection to aircraft crews and astronauts in the radiation environment.

Evidence: National needs in the areas of climate, weather, and natural hazards, along with the need for NASA participation, are summarized in the following documents: Strategic Plan for the U.S. Climate Change Science Program (CCSP) (www.climatescience.gov); U.S. Weather Research Program (USWRP) Vision Document (box.mmm.ucar.edu/uswrp/program_overview/overview.html); Reducing Disaster Vulnerability Through Science & Technology; report of the Subcommittee on Disaster Reduction of the National Science and Technology Council (sdr.gov/).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The ESS Program is not duplicative of other programs or efforts, and is in fact complementary to other Federal programs. NASA, as directed by Congress in NASA's authorizing legislation, has the unique responsibility to conduct activities in space to expand Earth system science. Only the ESS program develops and flies new experimental instruments in space to resolve scientific questions about the Earth and Sun-Earth systems that cannot be answered by existing data, ground-based measurements, or measurements from operational instruments. NASA ensures coordination with other agency efforts through the U.S. Global Change Research Program (USGCRP), Climate Change Science Program (CCSP), U.S. Weather Research Program (USWRP), and the National Disaster Reduction Committee. The ESS program is the Nation's only program designed to partner with operational agencies to systematically benchmark uses of NASA's remote sensing data and research results (observations and model predictive capabilities) in their decision support systems.

Evidence: Through interagency and international working groups, NASA conducts joint planning to ensure that observing capabilities are fully leveraged. The largest component of ESS is coordinated via the congressionally mandated U.S. Global Change Research Program (USGCRP) and the Climate Change Science Program (CCSP). ESS research and observations are an integral part of the interagency CCSP strategic plan released in the summer of 2003. As a specific example, NASA is working with other agencies to implement the U.S. Carbon Cycle Science plan (www.carboncyclescience.gov/planning.html#plan). NASA is also key participant in the recent efforts to coordinate Earth observation both nationally and internationally. Memoranda of Agreement are executed with partners to ensure that duplication of efforts does not occur. Partnering organizations include federal agencies (FAA, USDA, USGS, DHS, FEMA, EPA, CDC, NIH, DOE, DOD, DOI), state organizations (NSGIC, ASA, AAGS) and other national and international organizations. A detailed listing and status of Earth Science Applications Program Memoranda of Agreement (MOAs) are maintained by ESE. (www.earth.nasa.gov/eseapps/).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program design is built on a systematic, end-to-end approach that seeks to develop new technologies and data sets that contribute to improved predictive capability in research and operational forecast models. Program design encompasses interdisciplinary research, advanced technology development, development and deployment of global observing capabilities, scientific information product generation, and data assimilation and modeling. The program design, and its ability to effectively and efficiently achieve its goals has been optimized by considering and incorporating the advice and counsel of a very broad community of experts who have been intimately involved for a number of years. The complexity and scope of the program and the scientific questions ESS addresses require that the program draws on the expertise and talents of a diverse pool of investigators, including those from NASA centers, academia, other government agencies, and the private sector. The use of open, competitive programs provides a good approach for obtaining the best available talent to address our nation's scientific needs. The implementation of this approach requires significant effort in the writing and review of proposals, as well as the need to assure the absence of duplication, avoidance of gaps, and coordination of related activities among the cadre of investigators. The hardware/software development element of the program is subjected to a series of formal design reviews to ensure that the "design-to," "build-to," and "as-built" baselined requirements are properly established and met. In addition, lessons-learned workshops are conducted to prevent any previous mistakes from being repeated. NASA has had successes in transitioning research results for operational utilization. There is an opportunity for Federal agencies to use systematic approaches to improve our collective ability to identify opportunities and develop plans to transition research results to operational utilization as well as to use operational capacity for research activities. NASA and NOAA are developing a systematic approach to facilitate these transitions between the two agencies.

Evidence: The approach of the program is defined in several strategy documents. Different parts of the program have been reviewed in several different ways. The National Academy of Sciences review of the Sun-Earth Connection (SEC) program (now part of ESS) as part of its 2002 decadal survey found no major flaws. The Sun-Earth Connection Advisory Subcommittee (SECAS) 2003 review of SEC performance found no major flaws in the program and gave "green," or top, ratings to SEC's performance against its annual goals. Independent review teams assessed the readiness of several missions and found no major scientific, technical, or managerial flaws. These same missions were subsequently confirmed by Agency Program Management Councils. A good testimonial is that transition of essential measurements from Terra and Aqua to the National Polar-Orbiting Enviornmental Satellite System (NPOESS) will occur via the NPOESS Preparatory Project. Also, NASA, NOAA, and the European space program have formed a partnership for the transition of ocean altimetry measurements, and NASA is working with the NPOESS program office and the U.S. Geological Survey on the transition of Landsat-type data collection. Several external and NASA studies on socio-economic benefits resulting from the Earth Science Applications projects estimate potential annual benefits totaling approximately $20 billion. Specific studies include: (a) "An Estimate of NASA/ESE Power Program Benefits to the U.S. from 2002 through 2017;" and (b) "The Socio-economic Benefits of Earth Science and Applications Research: Reducing the Risks and Costs of Natural Disasters in the United States."

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program is designed to target key ESS program questions, and to provide data sets and model results to researchers, operational users and decision-makers. The rigor by which the ESS program is designed, structured, managed and funded ensures that resources reach only the intended beneficiaries and address the program's purpose directly. The ESS Roadmaps guide the activities of the ESS and provide the context through which specific research objectives are formulated, science investigations are solicited, and missions that address them are planned. Missions and all other ESS tasks are broken down into discrete work breakdown structure-style activities, and funds are issued at that level. These funds may not be spent on anything other than the purpose for which they were issued.

Evidence: A selection criterion for each science investigation, whether it is a research and analysis (R&A) grant or a science investigation in a mission, is "relevance to ESS objectives" as defined in the ESS Roadmaps. These roadmap can be found at www.earth.nasa.gov/visions/index.html. Thus, funding is targeted effectively from the start. Funding for multi-year R&A grants is performed yearly, dependent upon the successful completion of the past year's objectives and continuity in plans for follow-on research to maintain relevance to ESS objectives. The scientific purpose of each mission is well documented (see the IBPD and the Strategic Plan) and is linked to specific Enterprise and Agency goals and objectives. Funds are issued to the appropriate entity at the mission level or below. Federal law prohibits the redirection of resources issued for one program to another program without express Congressional approval. The Agency's full-cost management approach instills additional rigor in properly targeting and managing funds, ensuring that each program dollar is properly directed and expended. In terms of disseminating ESS observations and research results, data sets from NASA ESS missions are made available in standard formats to researchers and other users at gcmd.gsfc.nasa.gov/. Selected higher level data products are posted at www.esipfed.org/. Unintended subsidies are avoided by clarifying roles and responsibilities in Memoranda of Agreement with partners.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The ESS program's long-term PART measures focus on outcomes and meaningfully reflect the program's purpose. The long-term measures reflect the program's goal of developing and demonstrating new space-based observing capabilities and providing global Earth-Sun System data and information to answer priority science questions and improve understanding and prediction of the Earth-Sun system. ESS has 15 long-term performance measures,which address scientific outcomes that embody the purpose of the ESS program. Performance in support of long-term measures is evaluated on an annual basis by external scientific advisory committees.

Evidence: See ESS's long-term performance measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The ESS program is ambitious by nature, pursuing missions that have never been attempted before. The ESS program has ambitious targets and timeframes for its long-term measures. ESS's scientific measures aim for an annual rating of "green" on a green-yellow-red stoplight scale, signifying excellent progress as evaluated by an external advisory committee. These measures will be assessed annually for the program's duration.

Evidence: See the ESS long-term performance measures, timeframes, and targets.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Each long-term measure is also essentially an annual measure, in that progress toward accomplishing each long-term goal is evaluated annually, as noted in the targets and timeframes contained within each long-term measure. The ESS program also is committed to several annual measures that contribute to program goals. Such measures indicate the program's success in satisfying customer information needs, taking steps to transition assets from research to operations purposes, operating its spacecraft without interruption, meeting development milestones for new projects, competing research awards to ensure value for the dollar, and adhering to baseline cost, baseline schedule, and a competitive awards regime.

Evidence: See the program's long-term and annual performance measures, timeframes, and targets.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The ESS program has baselines and ambitious targets for its annual measures. The long-term/annual scientific measures all aim for ratings of "green" on a green-yellow-red stoplight scale, signifying excellent progress, by an external advisory committee ("blue" signifies superior performance, above even "green"). Targets for the development and technology measures stem from the baseline schedules and milestones established for the projects when they are confirmed to proceed. Targets are ambitious: for example, the ESS program commits to a high level of technical functionality in keeping its spacecraft operational 90 percent of the time. Several of the annual measures are new; thus, the program selected targets to be ambitious yet realistic, considering they are associated with new activities.

Evidence: See ESS's performance measures.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ESS partners (NASA Centers, JPL, contractors, universities, international organizations, and other federal agencies) are directly involved in planning and establishing the program's goals and objectives. Consequently, they fully support and are committed to the achievement of both the annual and the long-term goals of the program. ESS engages partners through the solicitation process and through interagency agreements that are geared towards the annual and long-term goals of the program. Both regularly scheduled and ad hoc reviews provide management insight into whether ESS partners are adhering to and supporting the program's goals and objectives. Partners who fail to exhibit proper support can be terminated from the program.

Evidence: The evalutation criteria in solicitations reference focus area and science questions. For basic research grants, the solicitations explicitly includes the program goals. Grant applications and progress reports provide sufficient means for program managerd to assess performance and continuing relevance (e.g. http://lcluc.gsfc.nasa.gov/products/results.asp?type=3). For all other contracted activities, the contracts specify the performance required and means of assessing performance. Through regularly scheduled science focus area reviews the ESS program assesses how well focus areas are adhering to program goals. Mission reviews are conducted to assess not only performance but progress towards annual and long-term program goals. ESS conducts ad hoc reviews to determine/verify partner commitment. Also, independent contract and programmatic reviews are conducted routinely.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The ESS program's effectiveness and program relevance are subjected to regular reviews and evaluation by the National Academy of Sciences (NAS), NASA advisory committees, and the ESS advisory committee. Annual performance toward achieving stated outcomes is both determined and validated by annual external reviews. The National Academy reviews meet the quality, scope and independence requirements of the PART and are used to assess many of the Governments R&D programs. International programs we contribute to are (often) assessed, usually by science steering groups or the NAS, for their performance as well as the quality and appropriateness of NASA contribution to the program. In addition, an Earth Science Decadal Survey is underway, and will improve program performance and relevance. A preliminary Decadal Survey Report was delivered in the spring of 2005, and a final report is expected by the end of calendar year 2006.

Evidence: Scientific and programmatic progress and performance for the ESS program are presented to the FACA-chartered Advisory Committee on a semiannual basis (e.g. www.earth.nasa.gov/visions/ESSAAC_minutes.html), and by the NAS on a periodic basis (e.g. www4.nas.edu/webcr.nsf/ProjectScopeDisplay/SSBX-L-03-05-A?OpenDocument (www.nap.edu/books/NI000352/html.index.html). Results of these reviews are the basis for revisions and updates of all ESS program activities and plans. For discrete missions, independent reviews are held that include but are not limited to: Non-Advocate reviews, Flight readiness and launch readiness reviews, operational readiness reviews. National Academy Reviews of the CCSP include Implementing Climate and Global Change Research (2004), books.nap.edu/catalog/10635.html. The findings from the latest NASA advsiory council maybe found in the annual Performance and Accountability Report.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The ESS budget request is the outcome of a rigorous bottoms up formulation process whereby the requirements are aligned to the science focus areas. Budget requests for each mission and project are dependent upon the successful completion of the current year's planned activities and the future requirements. NASA's full cost budgeting aligns the institutional and program resource needs and reflects a direct relationship between resources and Program strategic plans. The life-cycle cost requirements for each mission, in full cost, are included in NASA's Integrated Budget and Performance Document.

Evidence: Budget requests for mission formulation, development and mission operations are derived from assessments of annual performance and estimates of resources required to complete the mission and produce the scientific measurement. The resource requirements are clearly stated in full cost. NASA's Integrated Budget and Performance Document (IBPD) displays important status data for each mission, lists budget requirements for life cylce cost, and identifies the specfiic long-term outcomes and annual performance goals enabled by that mission. The ESS has a crosswalk from science focus area budget to Agency budget structure and is currently in process of transitioning to management structure based upon the focus areas. In addition NASA is one of the few agencies to receive a "green" rating from OMB for budget and performance integration.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Strategic planning deficiencies are identified and addressed through a series of regular updates and reviews to the long term plans. The strategic planning process engages the full range of participants of the Agency and involves sequenced generation of an overall strategic plan and several component strategies (research, applications, technology, education, data management). NASA has taken numerious steps to correct those deficiences identified both through external review as well as those identified internally.

Evidence: The strategic documents (e.g. plans, questions, and roadmaps) that guide the program are vetted by various external advisory groups. In the past both the science advisory groups to NASA and the National Research Council's (NRC) Space Studies Board (SSB) have performed these reviews. In some cases shortcomings in plans, both in content and presentation, were identified and corrected prior to the final version of the document. Other shortcomings have been addressed by managing the Program by focus area as well as by merging previously existing science programs to explore areas of scientific overlap between the programs. Budget cross-cuts by focus area are prepared for their use in trade-off assessments. Missions have also been reviewed and in cases when these missions were not performing to expectation (either scientifically or fiscally) these missions have been suspended or descoped accordingly (e.g. GIFTS). NASA engages the NRC to develop decadal surveys with recommended priorities for science questions and measurements.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: The analysis of alternatives to implement NASA's goals in ESS science begins with the development and broad vetting of science goals, questions, and focus areas. Concepts and alternatives for achieving the goals of these focus areas are analyzed and summarized in roadmaps, which identify the measurement and modeling capabilities needed, and provide a framework for assessing technical requirements and readiness, budget profile options and tradeoffs, internal management controls, operations concepts, and other implementation considerations (e.g. launch vehicle requirements, implied cumulative data rate). The broad science and aerospace industry community is invited to propose mission alternatives to meet the goals of the roadmaps through open and competitive solicitations. Selections themselves are based on analyses of trade-offs between cost, schedule, risk, and performance goals. After selection, programs and project are subjected to independent reviews throughout their life-cycle to evaluate their ability to meet commitments.

Evidence: Mission proposals are analyzed for both their scientific merit and their technical, cost and management approaches by scientific and technical peers from inside and outside of NASA. Implementation of the capability after selection follows a gated product development process as descrbied in NASA prodcedural Requiremtns NPR 7120.5C, which includes major external reviews at signficant stages of the development process. Through implementation, the ESS program regularly assesses program performance and the risk associated with different courses of actions. A recent example is the SDO mission; though three instruments were selected to begin phase A (formulation), only two meet the criteria to move forward to phase B (formulation refinement), while the third was terminated due to a projected cost cap exceedance of over $40 million.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The ESS program continually evaluates its efforts with respect to the relative potential benefits of alternatives in the process of implementing the the program's roadmaps, in part to determine if roadmap progress can be achieved using results produced outside of NASA. It is recognized that in some cases NASA has unique goals that no other agency has or is capable of obtaining. In some cases, missions are selected to work with in situ networks to contribute to a global observations strategy as called for under the global Earth Observation Summit.

Evidence: Examples of NASA coordination with other programs include the interagency Committee on Earth Observation Satellites (CEOS, www.ceos.org/pages/overview.html) and the interagency working groups coordination U.S. government research in each of the focus areas of the CCSP (www.usgcrp.gov/usgcrp/ProgramElements/default.htm). Documents that state requirements of satellite remote sensing in an integrated surface/in situ/satellite context include those found in the global climate observation system (www.wmu.ch/web/geos/groshome.html), and the integrated Global Observing Strategy (pc.unesco.org.igosparnters).

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Independent outside organizations help set scientific priorities in line with the Agency's and Program's goals and objectives. Within each science focus area are activities that indicate the specifc scientific advances to be pursued in the near and mid-term. These questions form the framework for identification of specific missions, technology tasks, and science NASA Research Announcements. The ESS program uses the priorities for scientific advances in the roadmaps, estimates of costs of these activities, and the availablity of funding to guide budget requests and funding decisions. Repeated management and scientific peer reviews ensure that each mission and scientific grant provides data and knowledge in a cost effective manner.

Evidence: The National Academy of Sciences reviewed the Sun-Solar System Connection element of ESS as part of its Decadal Survey to help NASA prioritize missions and science objectives for the next ten years. The ESS program focus area roadmaps and research strategies provide desired long-term outcomes and identify the critical inputs that are needed in order to accomplish the long-term goals (e.g. earth.nasa.gov/roadmaps and www.earth.nasa.gov/visions/researchstrat/Research_Strategy.htm). The FY2006 budget was formulated and will be managed within the context of the science focus areas. This will enable clear communication of the prioritization both within and between science focus areas. In addition, outside elements help to review, refine and, if necessary, set in a larger context: NASA plans, activities, and implementation strategies (e.g. the National Academy of Sciences for the Sun-Earth Connection, the Climate Change Science Program strategic plan for the climate change identified portions of the ESS program)

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The ESS Program collects relevant technical and programmatic performance data on a monthly basis. Additional program performance data is collected weekly, semi-annually, and/or annually, from both agency and partner organizations, depending on individual program requirements. This information is used to assess progress toward meeting long-term outcomes and goals, and is used to develop risk mitigation strategies, adjust priorities, and/or make resource allocations.

Evidence: The program conducts monthly and quarterly reviews to gather and analyze performance data from all participating organizations and compare against expected baseline performance data. Each flight mission and data program activity is reviewed monthly for cost versus plan, schedule movement, reserves, mass and power margins, and estimate to complete assessments often with the aid of earned value assessment. More detailed quarterly reviews are conducted for all ESS flight projects. The weekly, monthly, and quarterly reports are archived and available online to ESS management. The ESS Program has descoped, restructured, or cancelled programs and projects based on the evaluation of programmatic performance data showing negative or non-recoverable trends. Independent science and program technical/management groups review progress toward achieving long-term performance outcomes and efficiencies at major program milestones in development and bi-annually during implementation. The research program managers review the annual grantee reports and assess the accomplishments of the grantees against their original proposals. The results of the program managers' assessments are used to determine if continued funding is warranted. For example, the Applied Sciences Program considers resource adjustments at the semi-annual program reviews.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and partners are required to meet annual performance standards, including schedule and cost goals, and are rated through a number of formal mechanisms and requirements. Depending on their performance as measured by these efforts, partners and managers are subject to awards, such as mission extensions, or penalties, such as diminished program participation or funding cuts. A program or project manager is clearly designated for each mission per NPD 7120.4, Program/Project Management. Federal managers who fail to demonstrate the required level of performance are subject to a variety of disciplinary actions.

Evidence: All employee performance plans for managers include elements tied to the program's strategic plans. Managers can receive a fail grade on their performance ratings if they do not satisfactorily meet cost, schedule or performance metrics. Award fee reviews are performed on contracts and past performance evaluations are integral in Announcement of Opportunity (AO) criteria. All grants and cooperative agreements are subject to deliverables and milestones that must be met in order to receive funding renewal. Partners who fail to perform as required may likewise find their participation reduced or terminated--grantees, for example, might not have their project funds renewed if they mis-manage the funding. Cooperative Agreements have defined milestones that must be met by the Principal Investigators (PI) in order for the PI's to receive payment. These milestones are measured against performance metrics that are part of the Cooperative Agreements and approved by the NASA Program Managers in order to release further funding.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The ESS program obligates its funding in a timely manner and spends it for the purpose as appropriated by Congress. The ESS Program has had no significant erroneous payments and has not been in violation of the Anti-Deficiency Act.

Evidence: On average the ESS Program obligates approximately 85% of its authorized annual budget within the fiscal year for which funds are provided (NASA's funds are appropriated for a two-year period). One hundred percent of the ESS budget appropriated and authorized by the U.S. Congress is obligated over the two years available and is spent for its intended purpose as described in the IBPD and updated through the operating plan to Congress. Directorate and Agency-wide controls ensure that funds are spent for the intended purpose.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The ESS Program has the appropriate incentives and procedures in place to assure efficiency and effectiveness in program execution. Tracking earned value, baseline plans versus actuals, and reserves as a fraction of cost-to-go are typical tools used to track cost effectiveness. The program is certified ISO 9000 compliant, uses NPD 7120 for program and project management, and IFMP for financial management. The program has an Integrated Budget and Performance Document which guides the program execution. These measures are designed to give improved insight into use of resources so that timely decisions can be made to optimize performance and efficiency. An efficiency measure assesses time to research award from selection.

Evidence: IT improvements are used throughout the ESS Program to improve the flow of data and make information more available to the public and scientific community. NASA's move to full cost management is providing the ESS Program with a complete understanding of the cost of doing business across the NASA institution, the first step in reducing and controlling costs as well as understanding how efficiently the institution is supporting the program. All activity carried out in the program is either openly competed or selected after comprehensive peer review, and performance is monitored on a monthly basis (grants on an annual basis). Competition and regular contract review cycles assure that vendors selected perform in a cost-effective way, and experienced program/project management assures continued performance. The most effective method by which contractors/grantees are motivated to achieve cost effectiveness and efficiencies is through the fee review (contracts) and peer review (grants) processes. Several times a year, depending upon contract stipulations, a panel reviews the contractor's progress and assigns it a rating which determines how much fee the contractor will earn for that review period.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The ESS Program collaborates and cooperates, where reasonable and practicable, with other NASA programs and/or Federal agencies where shared or similar goals and objectives might permit a more efficient use of resources while increasing scientific or technological return. NASA's space-based assets are a critical part of an effectively coordinated observing capability. To promote the utilization of these observations in accomplishing shared goals, NASA routinely participates in joint planning and implementation efforts, joint initiatives, and jointly funded center and grant announcements with many other NASA programs and/or Federal agencies. In addition, NASA maintains a willingness to collaborate with other nations where there is evidence of a genuine intersection of interests.

Evidence: NASA program officials serve as Climate Change Science Program (CCSP) principals and participate in the interagency working groups tasked with coordinating efforts and developing joint interagency plans, milestones and deliverables (e.g., synthesis and assessment reports). NASA participates in the United States Weather Research Program (USWRP) developing common goals and coordinating efforts. The ESS Program participates in the National Space Weather Program, a program coordinated by the Office of the Federal Coordinator for Meteorology. There are also numerous NASA-NOAA joint projects, such as GOES, POES, NPOESS, which support both the research and operational communities. NOAA and the Air Force Weather Agency use data products from ESS missions to provide daily space weather predictions for operational and national defense applications. In addition, close international coordination is carried out both bilaterally (e.g., Ocean Topography with France, Global Precipitation Mission with Japan) and through the Committee on Earth Observation Satellites (www.ceos.org/pages/overview.html). The Earth-Sun System Division Applied Sciences Program is an integral component of Administration and interagency collaborations including the Climate Change Science Program, the Climate Change Technology Program, Commercial Remote Sensing Space Policy, CENR Subcommittee on Earth Observation, and the CENR Subcommittee on Disaster Reduction. The Program Plans, National Application Success Stories, and reports documenting the systems engineering process to deliver integrated system solutions to our partners in the twelve National Applications (aiwg.gsfc.nasa.gov).

Does the program use strong financial management practices?

Explanation: The most recent NASA independent auditor report identified four material weaknesses for the agency, all of which are repeats, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: NASA's FY 2004 Performance and Accountability Report (available at www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the independent auditor. In addition, the GAO has published numerous reports identifying shortcoming in NASA's new financial management system as well as its financial management processes (example is GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The ESS Program has a well-structured process in place to conduct both monthly and annual performance reviews. Any management deficiences that are uncovered during these reviews are noted and subsequently remedied. ESS management also has frequent contact with directors of implementing organizations for ESS projects to discuss and mitigate any management deficiencies. Finally, independent review of programs and projects are conducted at major milestones and periodically during implementation to provide an unbiased quality check of the engineering and management efforts. Corrective action plans are prepared, reviewed, and implemented for any deficiencies that are found. The program's financial management challenges are largely a product of agency-wide financial management difficulties and are likely to improve with improvements at the agency-wide level.

Evidence: The ESS Program conducts program management in accordance with the NASA Program and Project Management Processes and Requirements (NPR 7120.5C) procedure. A hierarchy of Program Management Councils (PMCs) exists to ensure appropriate levels of management oversight for programs. PMCs are established at the Agency level, Directorate level, at the assigned NASA Centers, and at lower levels within each Center as required. These councils evaluate monthly and quarterly the cost, schedule, and technical performance of ESS programs to ensure commitments are being met. When deficiencies are identified, corrective actions are assigned and tracked to completion. For example, the THEMIS project was required to present a recovery plan when cost reserves fell below acceptable levels.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: The program clearly defines and documents the capabilities or characteristics that are expected from projects, including specific milestones to demonstrate progress towards completion, and clearly identifies who is responsible and authorized to make management decisions based on whether milestones are being met. Responsibilities for oversight and execution of programs/projects are specifically assigned to officials at various levels of Agency management.

Evidence: Every flight mission is required to develop Level 1 Requirements, which is a key component of formal program implementation and documents the scientific, technical, schedule, and cost committments for the program or project. In addition, flight projects are required to develop Mission Success Criteria which define the minimum capabilities that must be achieved in order to declare mission success. The Level 1 Requirements and the Mission Success Criteria are established and controlled by the responsible Program Management Council. The newly issued NASA Procedural Requirements NPR 7120.5C defines the deliverables and performance expectations and defines the requirements for key management documentation that include deliverables, cost, schedule, and technical performance characteristics. The key management documents required to plan and control programs and projects are defined in NPR 7120.5C (http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005C_&page_name=main).

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: ESS awards a great majority of its Research and Analysis (R&A) grants (> 85%) based on clear competitive processes that include a qualified assessment of merit. This assessment of merit is based on the following primary evaluation factors: 1) relevance to NASA objectives, 2) intrinsic scientific and/or technical merit, and 3) value and costs. About 15% of grants, however are awarded non-competitively as a result of congressional direction (total of $56M in FY2005). Non-competitive grants are still subjected to a merit assessment per the NASA Grant Information Circular prior to being funded.

Evidence: Most activity is selected in response to full and open competition through Announcements of Opportunity, NASA Research Announcements, and Cooperative Agreement Notices (http://research.hq.nasa.gov/Formats.cfm). Mail and/or panel review is used for solicited as well as the small fraction of unsolicited research. Regular program reviews and, for centers, extensive use of visiting committees and review panels provide assessment and feedback. Solicitation vehicles provide a "level playing field" for all entrants, new and old (e.g. http://research.hq.nasa.gov/code_y/nra/current/NRA-01-OES-04/index.html). NASA Federal Acquisition Regulations supplement 1835.016 and 1872.403 dictate that peer review will be the method used to evaluate and select research for funding.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The ESS Program maintains effective oversight of grantee activities and performance. Grant renewals require annual reports and demonstrated performance levels consistent with the milestones and quality of performance established by the peer reviewed proposals. The agency has improved its oversight of grantees with a revised Grants Information Circular that addresses unsolicited proposals and congressional earmarks.

Evidence: Discipline scientists take the results of the grant peer reviews and make selections as to whom grants are awarded. These scientists then monitor the progress of the grant toward meeting its stated goals for the duration. Formal annual reports are provided by the grantee, and cumulative expenditures are closely tracked. This gives the discipline scientist who work with the grantee sufficient insight into his/her performance to understand what the grantees are accomplishing with the resources that are allocated to them. The formal annual reports are the primary method through which oversight and management control are exerted on the grantees. Close contact with funding recipients is maintained through site visits, Principle Investigator meetings, and discipline-oriented meetings. According to the Agency's Grant and Co-operative Agreement section 1260.22, an annual progress report describing the accomplishments during the reporting period is due 60 days before the anniversary date of the grant. A final report, including a comprehensive summary of significant accomplishments, is due within 90 days after the expiration date of the grant.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NASA collects grantee performance data and makes it available to the public in a manner that is both useful and meaningful. The agency has federally mandated external reporting requirements to the public on grants and contracts.

Evidence: Formal progress reports, which are a required output of each research and analysis activity funded under the ESS Program are submitted on an annual basis. The NASA lead scientist, together with appropriate discipline scientists, review the progress reports before recommending whether or not to continue the research activity to the procurement officers before funding is released to the grantees. The results of grants-based research are broadly disseminated to the public through the use of science forums, publications, NASA press releases and news conferences, museum displays, educational materials, and NASA's web site. NASA is currently working to develop an evolving database that will post grantees' annual reports on the Internet. The database is scheduled to become available to the public by the end of August 2005. At that point, the database will become operational for grantees to file their reports and available for public view. Consistent with the scheduled public availability, a test search engine is planned to be running by August 1, 2005. In addition, some of the highlights from the grantee annual reports are published in the "Space Science: Supporting Research and Technology (SR&T) Program Highlight" brochure.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Nearly all ESS activities (as well as competitive grants) are selected either competitively or through peer-reviewed proposals. The peer review process is used extensively in ESS and is essential for a high quality, relevant program. The use of external peer review enhances the quality of NASA's investigations and activities because it brings the best and most critical national and international experts to the evaluation process. External peer review ensures that fresh viewpoints, alternative perspectives and state-of-the-art understanding are included in the evaluation process. The program is managed as per NPR 7120.5C, NASA Program and Project Management Processes and Requirements. Included in the development process are a series of reviews which serve to demonstrate that the baseline requirements are properly established and met. Verification methods include test, analysis, independent verification and validation, demonstration and inspection.

Evidence: Most ESS activity is selected in response to fully open competition through Announcements of Opportunity, NASA Research Announcements, and Cooperative Agreement Notices (http://research.hq.nasa.gov/Formats.cfm). Mail and/or panel review is used for solicited as well as the small fraction of unsolicited research. Science investigations (science instruments and data analyses) are 100% competed and peer-reviewed prior to the start of Phase A. Typical evaluation criteria include scientific merit, technical merit and feasibility of the proposed investigation implementation (including cost risk). Once these projects are awarded and begin the definition process, they are subject to a risk mitigation phase in order to retire technical risk and ensure program quality before going into full development. During development, review boards comprising contractor and NASA personnel conduct Preliminary Design Review (PDR), the Critical Design Review (CDR), and the Design Certification Review (DCR). This certifies that the "design-to" baseline is established and meets requirements. NASA oversees all projects and holds the projects to uniform standards for management (NPR 7120.5C) including NASA-initiated independent reviews. ESS corrects deficiencies promptly (see question 3.7's evidence/data).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: A large portion of the ESS program contributes primarily to NASA Goal #1: Understand the Earth System and apply Earth System Science to improve prediction of climate, weather, and natural hazards. Last year, the Earth System Science, Earth Science Applications, and Sun-Earth Connections submissions to PART included 12 long term measures. This year, these have been consolidated into 10 measures, with some of the measures that had both long term and annual sections moving solely into annual requirements. There has not been a substantial change in the overall scope of the long-term measures. Progress has been made in all areas, in the scientific questions that need to be answered, the transition of NASA data to operational agencies, and the modelling that leads to predictive capability.

Evidence: The Performance and Accountability Report for FY04 (the most recent available) shows that the program is on track (green or blue score) for each of the the long-term goals of the ESS program. As a representive example of the long-term scientific progress, Voyager 1 crossed the Solar Wind Termination Shock in 2004, directly contrbuting to the measure of "understanding the changing flow of energy and matter thoughout the Sun, heliosphere, and planetary environments". In Earth science, continued analysis of GRACE observations led to breakthrough understanding of seasonal variability of underground water reservoirs. Continued progress in modeling at GISS efforts provided the coupled atmosphere-ocean-ice model to evaluate the "Earth's Energy Imbalance" (Science, Hansen, et. al.) providing for the first time conclusive evidence of a substantial planetary imbalance consistent with dominance of increasing green house gas forcing over other forcings. NASA GISS results are substantial contributions to the IPCC 2007 report, currently under development. In addition, NASA's short-term Prediction Research and Transition Center (SPORT) made advancements in utilization of satellite date (AIRS, MODIS) to improve several National Weather Service operational forecast products.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Earth Sun System program has achieved its annual performance goals to a large extent. The program delivered planned operating hours from its spacecraft instruments, met its target for competing research funds, and attained very high levels of customer satisfaction with its database. The program can improve its adherence to cost and schedule goals.

Evidence: A rating of green (93%) or blue (7%) for each APG evaluated in the FY04 Performance and Accountability Report shows that every part of the very diverse Earth-Sun Systems program is making significant yearly progress on the six science long term measures. Successes in the last year include the discovery that lighning can drain energy out of particles in the radiation belts, supporting "progress in understanding the response of magnetospheres and atmospheres to external and internal drivers". The successful launch of AURA on July 14, 2004 completed the EOS satellite series. Three of the four instruments aboard AURA have been operating nominally for most of the past year while validation activities were underway. MLS provided an unprecedented suite of simultaneous measurements allowing more accurate quantification of ozone destruction than previously possible, and even enabled the first observation of a solar proton flare enhancing mesospheric OH and the resulting destruction of mesospheric ozone. This discovery also highlights the "system science" aspect of much of ESS research. Areas in which goals were not achieved include an 18-month delay in the NPP Launch Readiness Date caused by a significant projected delay in delivery of a NOAA/IPO-procured science instrument, and some delays in research award selections, many of which were caused by uncertainties in the FY05 budget.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: A key area of activity with regard to efficiencies has been in the area of mission operations. NASA's re-engineering, consolidation, and automation of flight operations have led to documented cost savings. This is especially important because many Earth-Sun System missions are in extended operations. There is also a strong emphasis on reducing the validation period needed to make Earth-Sun System data available to the wider community. This has directly improved access to science and applications data at reduced costs.

Evidence: The Research and Analysis program in ESS is mostly composed of non-repetitive activities and therefore improved efficiency must come about through the evaluation of cost vs. benefit, which has always been a selection criterion for funded research. Mission development is also composed of unique projects. Here improvement is achieved with proper use of management and schedule tools that facilitate development of missions within cost and schedule such as adequate testing and adequate Phase A and B development time, maturing critical technologies off-line. Substantial effort has been put into improving the efficiency of Mission Operations, including development of robust automation procedures that have allowed missions such as the Advanced Composition Explorer (ACE) and TRMM to perform telemetry passes autonomously on weekends, reducing personnel costs.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other programs, either government of private, with similar purpose and goals, with which to compare the Earth-Sun System program.

Evidence: No other agency or organization takes the technology risks of building new experimental, one-of-a-kind instruments on unique space missions and uses data from the missions to improve understanding of the Earth-Sun System.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: For the most part, independent evaluations indicate that the ESS Program is effective and achieving results. Independent evaluations of the overall ESS program have been conducted by the Space Science Advisory Committee (SScAC) and the Earth System Science and Applications Advisory Committee (ESSAAC), FACA committees that provide regular reviews of the ESS Program -- which have generally been favorable -- and recommend any improvements. Consolidation of these two committees into one NASA Science Advisory Committee is currently in process. In addition, the National Academy of Sciences episodically reviews aspects of the program. A recent NAS review of NASA's Earth science strategy indicated that work remains in piecing together roles and responsibilities so that NASA research can be used by the broader community. Missions under development are scrutinized by Independent Review Teams (IRT), with schedule and cost efficiency being some of their primary concerns. Operating missions are reviewed every three years in a "Senior Review" and decisions on mission extensions are heavily influenced by the Senior Review's conclusions about the scientific return of each extended mission.

Evidence: In its 2003/2004 meetings, ESSAAC recommended NASA identify metrics of user satisfaction with EOSDIS for use in guiding operations and evolution planning. NASA conducted a customer satisfaction survey and presented the results to ESSAAC in its September 2004 meeting. The ESSAAC chair reported that "the Committee was encouraged to hear the results of the EOSDIS customer satisfaction survey and congratulates the team responsible for this work." He went on to describe the Committee's view of how this might be expanded in the future. Recent NAS reports relevant to the ESS program include the "Assessment of the Benefits of Extending the Tropical Rainfall Measuring Mission: A Perspective from the Research and Operations Communities, Interim Report" and the "Assessment of NASA's Draft 2003 Earth Science Enterprise Strategy: Letter Report." The detailed evaluations of the IRTs ensure that projects in development manage their resources effectively. Any significant cost or schedule overrun is referred back to the IRT for re-evaluation. The Senior Review process that has been successfully used by Space Science for many years has now been implemented by the Earth Sciences. In early 2005, the first Senior Review of the Earth Science operating fleet occurred and the results are imminent. Scientific results are the primary criteria evaluated, and therefore mission extensions depend upon scientific production.

Were program goals achieved within budgeted costs and established schedules?

Explanation: The ESS program goals were largely achieved within budget and established schedules. However, discrete missions have had challenges maintaining cost and schedule baselines.

Evidence: The IBPD has specific cost and schedule information for missions in development. The record at the division level has been one of work performed within budget and schedule, particularly in the areas of research, applied science, data management and applications. Schedule and budget erosion problems have arisen on some missions in development. For example, the NPP Launch Readiness Date has slipped 18 months due to a late instrument delivery from NOAA/Indepedent Project Office. The CloudSat/CALIPSO launch has also slipped due to a combination of necessary instrument rework and launch pad conflicts with higher priority missions (note that both spacecraft are at the launch site and currently have a launch date of September 11, 2005). ESS has been able to make trades within the programs to cover some of the overruns and has been able to minimize the impacts to other projects.