Detailed Information on the
NASA Education Program Assessment

Expand the scope of the performance data reported against the program's metrics such that the program measures and reports the majority of its portfolio's performance.

Conducting independent evaluations to assess the program's effectiveness and efficiency against the program's established metrics and performance goals and applying resources based on the results.

Offering opportunities not addressed by other agencies and that are unique in their use of NASA's resources and benefits to NASA's mission and collaborating with other agencies where appropriate.

Avoiding duplication with other NASA education programs.

Fully execute the new education investment framework, per the framework's implementation plan, to complete the strategic alignment of the Education portfolio that best supports the Agency strategic direction and the Exploration Vision. This action is a continuation of a former follow-on action to develop the investment framework and implementation plan.

Completing the consolidation of the program's three performance information databases into a single database system.

Collecting performance data consistently and annually for all program activities, reporting performance against the program's established metrics and targets, and using results to improve performance.

Filling NASA's workforce needs using a stronger effort to consider eligible program participants and facilitate their entry into positions at NASA.

Establishing baselines for all performance metrics.

Measure: Number of institutions served in designated EPSCoR states.

Explanation:Twenty-five states participate in NASA's research capacity/capability building program, EPSCoR. Data comprise a non-duplicated count of institutions served by the following programs: EPSCoR, Space Grant, MUREP, and GSRP. Data reflect the states eligible for NASA EPSCoR as determined by the National Science Foundation EPSCoR Program per the NASA EPSCoR legislation.

Measure: Number of under-represented and under-served students participating in NASA higher education programs.

Explanation:These graduate and undergraduate students are served by Space Grant and MUREP. Actual number includes students that participate in courses, scholarships, events, travel, lectures and sponsored competitions.

Measure: Percentage of elementary and secondary educators who participate in NASA training programs who use NASA resources in their classroom instruction.

Explanation:NASA training programs result in deeper content understanding and confidence in teaching STEM disciplines. Teachers who attend professional development (experiences longer than two-days) are more likely to integrate NASA STEM resources in the classroom than other teachers. The data is based on teachers reporting using NASA product and resources in their classroom instruction.

Measure: Percentage of elementary and secondary educators who haven't participated in intensive NASA training programs and use NASA content-based STEM resources in the classroom.

Explanation:Educators obtain NASA STEM resources in a number of ways such as downloading from the web, obtaining materials from NASA Educator Resource Centers and CORE, and through direct participation in training experiences where materials are demonstrated. Feedback from these educators has been the basis for setting the percentage.

Measure: Number of museums and science centers across the country that actively engage the public in major NASA events.

Explanation:The 350 number includes members from the NASA Museum Alliance (MA), Space Place Network (includes partners in every state e.g. community museums, libraries, planetariums, zoos and aquariums) the Smithsonian, NASA Visitor Centers, earmark recipients and Office of Education special projects (e.g. major events by the NASA Office of Communications Planning that engage the public in selected major NASA events). NASA's budget request reflects a reduction each year through FY 11. NASA's objective is to continue serving the same number of participants, despite these funding reductions, through achievement of efficiencies.

Measure: Percentage of NASA higher education program student participants employed by NASA, aerospace contractors, universities, & other educational institutions.

Explanation:Baseline includes national higher education and MUREP projects within the Office of Education Portfolio, and is limited to students who have received scholarships, fellowships or stipends totaling >$5K or internships of at least 160 hours. Responses are self-reported from students who have completed terminal degrees in the NASA pipeline including Space Grants, GSRP, Jenkins, and Pre-Service teachers. Future enhancements will include standardized tracking and surveys across all national higher education projects.

Measure: Percentage of undergraduate students who move on to advanced education in NASA-related disciplines.

Explanation:Data includes students as reported by Space Grant, and is limited to students who have received scholarships, fellowships or stipends totaling >$5K or internships of at least 160 hours. Responses are reported by national space grant directors. Future enhancements will include standardized tracking and surveys across all national higher education projects.

Measure: Number of elementary and secondary student participants in NASA instructional and enrichment activities.

Explanation:K-12 student involvement encompasses a range of activities including one-time, short duration enrichment activities to longer-term or sustained learning opportunities. Participants are provided with authentic first-hand opportunities in NASA mission activities through site-based and distance learning networks. Includes projects funded within the K-12 education program.

Measure: Percentage of students expressing interest in science and technology careers following their involvement in NASA elementary and secondary education programs.

Explanation:A prerequisite to student achievement in STEM subjects is a high level of interest. Therefore, NASA's projects are designed to take advantage of NASA's mission to enhance students' interest in aerospace-related, STEM subjects. Interest is measured by surveys of students conducted before and after participation in the NASA education program. This measure will report the percentage of students who report a post-participation increase in interest.

Measure: Dollar invested per number of people reached via e-education technologies.

Explanation:NASA will continue to use internet- and web-based technology to deliver content to reach ever larger numbers of participants. Percentage reductions are per year over the preceding year. The number of people reached is estimated based on the number of page views.

Measure: Cost per participant for NASA elementary and secondary education programs

Explanation:NASA will work to achieve a reduction in the cost per K-12 program participant. As articulated in the education framework, we will draw from audiences that have demonstrated interest in NASA and connect participants to the next level of engagement. A blend of projects and activities encourage continued student affiliation with NASA throughout their academic career, resulting in efficiencies in recruitment and retention. NASA's budget request reflects a reduction each year through FY 11. To continue serving the same number or participants, it will be necessary to reduce the cost per participant.

Measure: Ratio of funds leveraged by NASA funding support.

Explanation:NASA works to maximize its investment by collaborating with partner organizations. Data reported are based on matching dollars submitted in proposals to NASA for the Space Grant and EPSCoR programs. These external proposals are built on the research and education activities of the NASA programs. The non-NASA sources included state and other federal government agencies, and industry.

Measure: Number of new or revised courses targeted at the STEM skills needed by NASA that are developed with NASA support.

Explanation:These are university-level courses based on the results of NASA's science and space exploration missions. New courses are developed by Space Grant; NASA University Research Centers and MUREP.

Is the program purpose clear?

Explanation: The purpose of the NASA Education Program is clearly defined in the official NASA publication, "NASA Education Strategic Coordination Framework: A Portfolio Approach," which describes how the NASA Education Program is supporting the Agency's mission--as laid out in the 2006 NASA Strategic Plan--by actively promoting the development of a highly skilled and diverse aerospace workforce. It articulates three major education goals that together support the NASA mission: 1. Strengthen NASA and the Nation's future workforce; 2. Attract and retain students in Science, Technology, Engineering, and Mathematics (STEM) disciplines; and 3. Engage Americans in NASA's mission.

Evidence: Information on the purposes and intended outcomes of NASA's Education Program may be found in these documents: NASA Education Strategic Coordination Framework: A Portfolio Approach, NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf 2006 NASA Strategic Plan http://www.nasa.gov/pdf/142302main_2006_NASA_Strategic_Plan.pdf

Does the program address a specific and existing problem, interest, or need?

Explanation: As stated in the 2006 NASA Strategic Plan, NASA must develop and implement Agency-wide human capital management initiatives to ensure that the NASA workforce has, and will continue to have, the right mix of skills and experience, in an appropriate balance among civil service, contractor, and other workforce components. To that end, the NASA Education Program focuses on educating individuals to prepare them for employment in disciplines needed to achieve NASA's mission and strategic goals. The NASA Education Program begins by offering simple, engaging learning activities in the earliest grades; it follows these with more intensive educational experiences in high school, and culminates in higher-education internships, fellowships, and other professional training. This process creates a pipeline of people who may participate in the NASA mission through employment with NASA, industry, or academia. Evidence of the continuing need for STEM education on the part of the broader education community is provided in the National Science Foundation (NSF) publication, "Science and Engineering Indicators 2006." Furthermore, the 2007 report by the National Academies, "Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future," makes two recommendations that relate directly to the value for the Nation of NASA's Education Program: 1. Increase America's talent pool by vastly improving K-12 science and mathematics education; and 2. Sustain and strengthen the Nation's traditional commitment to long-term basic research. The NASA Education Strategic Coordination Framework and its associated projects are designed to address and implement these recommendations. "Report of the Academic Competitiveness Council" (published May 2007) provides additional evidence of the urgency of the needs addressed by the NASA Education Program. More evidence of the international need for STEM-related education is offered by the Program for International Student Assessment (PISA) report, "PISA 2006: Science Competencies for Tomorrow's World."

Evidence: The problems, interests, and needs addressed by NASA's Education Program are outlined in the following documents: 2006 NASA Strategic Plan http://www.nasa.gov/pdf/142302main_2006_NASA_Strategic_Plan.pdf Science and Engineering Indicators 2006 http://www.nsf.gov/statistics/seind06/ Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future http://www.hq.nasa.gov/office/oer/nac/documents/Gathering_Storm.pdf NASA Education Strategic Coordination Framework: A Portfolio Approach, NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf Report of the Academic Competitiveness Council http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf PISA 2006: Science Competencies for Tomorrow's World" http://www.oecd.org/document/2/0,3343,en_32252351_32236191_39718850_1_1_1_1,00.html

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Through internships, research grants, and other education opportunities, NASA's Education Program offers students and educators access to the Agency's unique state-of-the-art equipment and facilities, a unifying aeronautics and space-centered mission of scientific discovery and exploration, and the matchless expertise of a highly skilled aerospace workforce. These educational opportunities cannot be duplicated by any other U.S. organization. To avoid redundancies in its Education Program, in 2006 NASA established an Education Coordinating Committee (ECC) that is chaired by the Assistant Administrator for Education and that includes education representatives of all the Centers, each Mission Directorate, the Offices of Diversity and Equal Opportunity, Human Capital Management, Public Affairs, Legislative Affairs, External Relations, and the Astronaut Office. The Office of Education maintains a portfolio database of all NASA education investments to track and eliminate duplicative education projects, and the ECC coordinates all NASA education activities, no matter which NASA organization has direct management responsibility. NASA also cooperates with other Federal agencies through the Academic Competitiveness Council (ACC) and the National Science and Technology Council's (NSTC) Education and Workforce Development Subcommittee to coordinate education programs, share resources, and eliminate wasteful duplication. NASA uses these forums to collaborate with other scientific and education agencies (e.g., the Departments of Education and Energy, and NSF) to minimize redundancies and replicate effective practices. NASA also relies upon formal partnerships [including Memoranda of Understanding with the NSF (March 2007), the National Park Service (April 1997), and the Federal Aviation Administration (May 2007)] to minimize redundancies with other education efforts and to solve problems that transcend any particular agency. Finally, the NASA Office of Education began a comprehensive survey of all similar programs in the Federal government, universities, and the not-for-profit sector to confirm that its efforts are not redundant or duplicative; the results of that survey will be available in the next few months.

Evidence: FY08 NASA Education Portfolio sorted by Outcome https://outsidenasa.nasa.gov/portal/education/portfolio_by_outcomes_with_totals.pdf FY08 NASA Education Portfolio sorted by Funding Source https://outsidenasa.nasa.gov/portal/education/portfolio_by_funding_orgs_with_totals.pdf NASA-National Science Foundation MOU http://www.education.nasa.gov/divisions/higher/overview/F_One_Giant_Step_STEM_Education.html National Park Service MOU https://outsidenasa.nasa.gov/portal/education/NPS-NASA_MOU.pdf NASA-FAA MOU https://outsidenasa.nasa.gov/portal/education/FAA_NASA_MOU_.pdf

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program continues to be free of major flaws, but the Office of Education seeks continuous improvement opportunities. The reorganization of NASA's Office of Education in 2005 and the development of the NASA Education Strategic Coordination Framework in 2006 addressed inadequacies in effectiveness and efficiency identified by OMB in NASA's FY 2004 PART assessment. Since the FY 2004 and FY 2007 PART assessments, the Education Program has terminated, phased out, or restructured to improve effectiveness and efficiency, about half the projects that were part of the 2004 Education Portfolio. The Office of Education has cancelled 13 projects, phased out another 13, and restructured 18 more. In FY 2009, the Office of Education plans to collapse and consolidate the MUREP projects, collapse and combine Flight Projects and the Educator Astronaut Project, and aggregate three smaller efforts (COTF, DLN, ePD) into one project. Through annual internal and external portfolio reviews conducted by NASA Education Program staff and external subject matter experts, the Office of Education continues to assess projects and make termination decisions. New portfolio database tools will better enable project and program managers to review schedule, cost, and benchmarks, identify flaws in effectiveness and efficiency, and make informed decisions about whether to fix or terminate flawed projects. To ensure that NASA's Education Program is correctly focused on the needs of the education community and the Nation, the Office of Education also actively seeks and incorporates advice and counsel from other Federal agencies (e.g., NSF, DOE, NIH, NOAA), professional organizations (e.g. CCSSO, NSTA, ITEA, NEA), for-profit organizations (e.g. Northrop Grumman, Boeing, Exxon-Mobil, Lockheed Martin), and not-for-profit organizations (e.g. GSUSA, St. Louis Science Center, Adler Planetarium, CCSSE). For example, NASA used input from a broad range of stakeholders, including the National Science Teachers Association, education consultants, other Federal agencies, and NASA Education staff, to develop the program's outcomes and objectives. In FY07, the National Academies provided a program review and critique of NASA's elementary and secondary education program. The critique identified 23 recommendations that the Agency is using to shape its FY09-FY10 program plan. Additional areas for improvement in efficiency and effectiveness were identified in FY 2005 by a review team composed of internal and external stakeholders led by NASA's Office of Program Analysis and Evaluation (PA&E). One improvement recommended by the team was to revise NASA's education goals to improve alignment with the 2006 NASA Strategic Plan. The team also proposed a new governance model to improve communication and coordination, as well as to clarify roles and responsibilities. An ECC working group was established to address this issue, and in March 2008, released a Communications Strategy. The NASA Education Strategic Coordination Framework revised Education's management approach and its articulation of outcomes, objectives, and measures, addressing flaws identified in NASA's FY 2004 PART assessment. Furthermore, NASA established the position of Assistant Administrator for Education, a top-level manager who has authority over all aspects of the NASA Education Program across the Agency.

Evidence: NASA Education Strategic Coordination Framework: A Portfolio Approach, NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf PA&E Report https://outsidenasa.nasa.gov/portal/education/PA-E_Education_Review_transmittal.pdf 2006 NASA Strategic Plan http://www.nasa.gov/pdf/142302main_2006_NASA_Strategic_Plan.pdf The National Academies' review, "NASA's Elementary and Secondary Education Program: Review and Critique" (final to be printed in 2008) http://www.nae.edu/nae/naepcms.nsf/weblinks/MKEZ-79TJ8X?OpenDocument NASA Education Communication Strategy https://outsidenasa.nasa.gov/portal/education/Ed_Communic_Strategy.pdf

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: NASA's highest-priority education goal is to develop an effective pipeline of future workers to support the NASA mission. Accordingly, the Education Program budget directly targets current and future NASA human capital needs. The budget request for the Office of Education is directly mapped to the priority of each outcome, and the largest portion supports higher education. Senior management in Education and at the Agency has identified higher education participants as the highest-priority investments because they are closer to being eligible for entering the workforce, and although the investment per participant is greater, the benefit to NASA and the Nation's STEM workforce is near-term. Therefore, a representative of the Office of Human Capital Management serves on NASA's ECC to ensure development of diverse and qualified workforce through coordination of the NASA Education Program with Agency human resources requirements. In 2007, the Office of Education worked with Human Capital to ensure that NASA implemented its authority to recruit and employ students directly from its education programs into its workforce. This action reflects significant progress toward the PART improvement plan action 5, which calls for a stronger effort to consider eligible (education) program participants and facilitate their entry into positions at NASA. To provide quality STEM education experiences at all levels of education, NASA's Education Strategic Coordination Framework identifies a progression of opportunities throughout the education process to inspire, engage, educate, and ultimately employ students. These opportunities target specific populations of beneficiaries: students at all levels; elementary and secondary school teachers; higher-education faculty; underserved and underrepresented audiences; and educational institutions. The majority of NASA's Education financial resources, which are generally dispersed in the form of competitive grants, are directed toward institutions, including schools, universities, and informal education organizations. The Office of Education, in collaboration with the Office of Procurement, substantially revised NASA's Grant and Cooperative Agreement Handbook and NASA's Guidebook for Responding to a NASA Research Announcement (NRA) to include specific guidance and reporting requirements to ensure that all grants align with program purpose. The Space Grant College and Fellowships Program (Space Grant), Experimental Program to Stimulate Competitive Research (EPSCoR), and Minority University Research and Education Program (MUREP) projects make awards through competitive processes to states and institutions designated by Congress. To ensure that the greatest possible portion of NASA's Education funding reaches the intended recipients, the MUST, SEMAA, and USRP projects have capped their overhead costs. Significantly, the Office of Education has determined that only about 3% of NASA Education funding goes toward non-aligned or supporting crosscutting infrastructure costs such as conference support, database development, and evaluation.

Evidence: NASA Education Budget https://outsidenasa.nasa.gov/portal/education/NASA_Education_Budget.pdf The evaluation process articulated in the NASA Education Strategic Coordination Framework validates that Education's projects are reaching appropriate internal NASA and external audiences. NASA Education Strategic Coordination Framework: A Portfolio Approach, NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf Workforce Strategy Documents https://outsidenasa.nasa.gov/portal/education/Workforce_Strategy_2006.pdf http://www.nasa.gov/mission_pages/transition/home/index.html Examples of Employment Opportunities NA08N0002 - Professional Engineering Positions http://jobsearch.usajobs.gov/ftva.asp?opmcontrol=1101145 NA08N0003 - Physical Sciences/Biological Sciences http://jobsearch.usajobs.gov/ftva.asp?opmcontrol=1101137 NA08N0004 - Accounting and Budget/Business and Industry/Organizational Administration/Human Resources http://jobsearch.usajobs.gov/ftva.asp?opmcontrol=1101194 NA08N0005 - Computer Engineer/Computer Scientist http://jobsearch.usajobs.gov/ftva.asp?opmcontrol=1101254 Space Grant http://www.nasa.gov/audience/foreducators/Space_Grant.html EPSCoR http://education.nasa.gov/edprograms/national/epscor/home/index.html MUREP http://www.nasa.gov/audience/forstudents/postsecondary/learning/F_MUREP.html MUREP provides annual reports that document that Historically Black Colleges and Universities, Tribal Colleges, and Hispanic Serving Institutions are served as intended http://mured.nasaprs.com/report/ Portfolio spreadsheet showing crosscutting/overhead costs https://outsidenasa.nasa.gov/portal/education/portfolio_by_outcomes_with_totals.pdf Portfolio spreadsheet sorted by funding organizations https://outsidenasa.nasa.gov/portal/education/portfolio_by_funding_orgs_with_totals.pdf Pie charts showing percentages by Outcome and by Funding Sources https://outsidenasa.nasa.gov/portal/education/Education_Funding_Pie_Charts.pdf Grant and Cooperative Agreement Handbook http://ec.msfc.nasa.gov/hq/grcover.htm NRA Guidebook http://www.hq.nasa.gov/office/procurement/nraguidebook/ A new portfolio assessment process is enabling analyses of NASA's Education Program to ensure that resources are addressing the program's outcomes and objectives effectively and efficiently, and will reach intended beneficiaries. This raw data spreadsheet illustrates alignment assessments and other project data that have been collected for FY08. Using Expert Choice software, the Office of Education can analyze these data to produce reports that rank projects by priority and synthesize project costs with priorities. These reports can be used to help detect possible project duplications and give other guidance on redirecting project focus. https://outsidenasa.nasa.gov/portal/education/EC_alignment_values_20080327.pdf Space Grant Budget Call https://outsidenasa.nasa.gov/portal/education/EPSCOR_2008_ Budget_Call.pdf EPSCoR Solicitation http://nspires.nasaprs.com/external/solicitations/summary.do?method=init&solId={C9877989-D507-A9F8-898A-C5FCCDDD7879}&path=init

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: In FY 2006, NASA established two working groups to identify all projects in the NASA Education portfolio, align each with the outcomes and objectives set forth in the NASA Education Strategic Coordination Framework, and to develop short- and long-term performance measures for each of the objectives. The Portfolio Working Group developed the project inventory and alignment, while the Evaluation Working Group developed the measures. There were 15 measures selected for use in the 2008 PART review. Some of the long-term measures reflect the development of the aerospace workforce (at NASA, in academia or industry), by tracking NASA-supported students. In order to create synergy between PART and the Government Performance and Results Act (GPRA), the Office of Education's GPRA Annual Performance Goals (APGs) are derived from the PART measures.

Evidence: NASA Education PART Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf Education Annual Performance Goals in NASA FY 2009 Budget Request (Pages Edu-1 through Edu-19) http://www.nasa.gov/news/budget/index.html NASA Education Outcomes, Objectives and Measures http://education.nasa.gov/about/strategy/index.html NASA Education Project Inventory https://outsidenasa.nasa.gov/portal/education/EC_alignment_values_20080327.pdf NASA Education Project Data Collection Requirements https://outsidenasa.nasa.gov/portal/education/Guidance_FY07_performance_data.pdf Workforce Strategy Documents https://outsidenasa.nasa.gov/portal/education/Workforce_Strategy_2006.pdf http://www.nasa.gov/mission_pages/transition/home/index.html

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: NASA has ambitious targets and timeframes for its long-term measures across the Program, and strives to reach a steady-state number of participants despite budget reductions from FY07-FY11. These targets were adopted by the ECC for FY07 through FY11 and have been disseminated throughout the NASA education community to guide program development and implementation. In 2007, the Office of Education collected data on the measures and established baselines. Responding to the results of the NASA Workforce Transition Strategy Report, which studied available workforce for the human space program through 2015, NASA Education recognized the need for and is now providing leadership in the effort to address the gap between students pursuing STEM disciplines and the aerospace and NASA future workforce needs. To achieve success, NASA Education not only sets ambitious targets for drawing participants for its education projects, but is encouraging a necessary dialogue between aerospace workforce stakeholders. NASA's Education Framework to Inspire, Engage, Educate, and Employ was used as the construct for the Interagency Aerospace Revitalization Task Force Roundtable, designed to raise awareness and drive the national dialogue on aerospace and STEM education and workforce needs. This platform allowed experts to build upon previous asset mapping efforts to define promising practices, programs, technical tools and resources that may be leveraged to benefit the future work of the Task Force. Currently, the Office of Education is completing a study benchmarking PART practices employed by NASA Centers and other Federal agencies, including performance measures, methods for capturing data, and use of the resulting data. The study will also incorporate an assessment of STEM education organizations in the external education community. This process should validate the rigor of, or provide insight on how to improve, NASA's Education performance measures and targets. Furthermore, NASA derives Education's GPRA APGs from the PART performance measures, ensuring that the GPRA measures adhere to the same ambitious targets and timeframes.

Evidence: NASA Education PART Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf Education Annual Performance Goals in NASA FY 2009 Budget Request (Pages Edu-1 through Edu-19) http://www.nasa.gov/news/budget/index.html NASA Education Outcomes and Objectives http://education.nasa.gov/pdf/219824main_Outcomes%20and%20Objectives%20Hierarchy.pdf Interagency Aerospace Revitalization Taskforce Report http://www.doleta.gov/pdf/REPORT_Aerospace_2008.pdf Workforce Strategy Documents https://outsidenasa.nasa.gov/portal/education/Workforce_Strategy_2006.pdf http://www.nasa.gov/mission_pages/transition/home/index.html

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The NASA Education Program has six specific annual performance measures that demonstrate progress through baselines and measurable annual targets toward achieving long-term goals. For example, annual measures track the numbers of students involved in NASA Education programs, those employed by NASA, contracts or academic institutions, as well as efficiencies gained in program execution, such as cost to NASA per participant. The Office of Education's GPRA APGs are aligned with the PART measures.

Evidence: NASA Education Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf Education Annual Performance Goals in NASA FY 2009 Budget Request (Pages Edu-1 through Edu-19) http://www.nasa.gov/news/budget/index.html

Does the program have baselines and ambitious targets for its annual measures?

Explanation: In FY 2007 the NASA Education Program established baselines and ambitious targets for its annual measures. Furthermore, NASA derives Education's GPRA APGs from the PART performance measures, ensuring that the GPRA measures adhere to the same ambitious targets and timeframes. Program and project managers from NASA Higher Education, MUREP, Elementary/Secondary and e-Education, and Informal Education reviewed historical data related to long-term performance measurements from FY 2004 through FY 2007. They determined the rate of improvement and used this information to establish ambitious targets. To further validate the rigor of the education targets and as part of Education's ongoing effort to improve performance the Office of Education is conducting a benchmarking study of performance measures, targets, and the associated data collection and reporting techniques used by other Federal agencies engaged in scientific and education missions.

Evidence: NASA Education Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf The Office of Education's APGs can be found in the annual performance plan, included in NASA's annual Budget Estimates: http://www.nasa.gov/news/budget/index.html

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All NASA Education project proposals must address the annual and/or long-term goals of the NASA Education Program. The NRA Grant and Cooperative Agreement Handbook and other official regulations require grantees, sub-grantees, contractors, cost-sharing partners, and other government partners to be accountable for the activities specified in their proposals. NASA is currently revising the Handbook for use with 2008 congressionally directed projects that will further strengthen education reporting requirements. Proposing institutions will be required to specify how they will serve NASA's annual and/or long-term education goals. NASA Education project officers (the primary point of contact at NASA for a recipient organization) provide guidance annually to grantees regarding any changes to NASA performance measures for education. For grant programs, the project officer reviews the initial proposal, prepares a technical justification and grant package, and monitors the grant, ensuring that the project is implemented as proposed and that required performance measurement data are submitted. Education provides similar guidance to recipients of congressionally directed project funds. The terms of Space Act Agreements and Memoranda of Understanding (MOUs) include defining the roles and responsibilities of the parties involved, with a focus on commitment to annual or long-term program goals, evaluation, and reporting on measures and results. The Outcome Mangers report quarterly to the Assistant Administrator for Education on the progress of all projects in meeting the goals, objectives and data collection processes.

Evidence: NASA Office of Education projects are established by formal, signed project plans, which represent institutional commitment to supporting Agency education measures. Examples of NASA Office of Education Project Plans eEducation Small Project https://outsidenasa.nasa.gov/portal/education/Project_Plan_eEducation.pdf NSTI-MI https://outsidenasa.nasa.gov/portal/education/Project_Plan_STI.pdf USRP https://outsidenasa.nasa.gov/portal/education/Project_Plan_USRP.pdf Harriet Jenkins Pre-Doctoral Fellowships (JPFP) https://outsidenasa.nasa.gov/portal/education/Project_Plan_Jenkins.pdf LTP https://outsidenasa.nasa.gov/portal/education/Project_Plan_Learning_Technologies.pdf NETS https://outsidenasa.nasa.gov/portal/education/Project_Plan_NETS.pdf The Grant and Cooperative Agreement Handbook http://ec.msfc.nasa.gov/hq/grcover.htm Workforce Strategy Documents https://outsidenasa.nasa.gov/portal/education/Workforce_Strategy_2006.pdf http://www.nasa.gov/mission_pages/transition/home/index.html

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: NASA is implementing evaluations through credible, objective evaluators using rigorous evaluation techniques including Randomized Control Trial (RCT) methodologies, as recommended by the ACC. The rigor is determined by the guiding principles of the American Evaluation Association. NASA has developed a schedule and submitted it to OMB for ongoing independent evaluations of all projects funded by the NASA Office of Education. Each investment is scheduled for at least one independent, rigorous, and reliable evaluation within any five-year period. For example, an outcome evaluation of the NASA Explorer Schools project was completed by Paragon TEC, Inc. in FY 2007. This evaluation tested the feasibility of using RCTs and resulted in several improvements to the project. In situations in which an RCT-based method is not appropriate, the Office of Education adopts other methodologies that are rigorous, objective, and in conformance with recognized professional standards. In FY07, the National Academies provided an external independent program review and critique of NASA's elementary and secondary education program. The critique identified 23 recommendations that the Agency is using to shape its FY09-FY10 program plan. Independent evaluations of the Graduate Student Researchers Program (GSRP) documented its effectiveness and also led to an improvement in stipend levels based on recommendations. Other NASA Education projects or activities that have received independent evaluations include the Aerospace Education Services Program (2004), EarthKAM (2006), and the Faculty Fellowship Program (2006). In FY 2008 an independent evaluation is planned for the Science, Engineering, and Mathematics Aerospace Academy (SEMAA) project. All NASA Office of Education projects will have received at least one independent evaluation by the end of 2012. On March 25, 2008, NASA released a solicitation (RFP NNH08230240P ) to award a contract to plan and design information collection protocols and to conduct evaluative research. The solicitation was released via email to organizations specializing in education evaluation as found on the GSA MOBIS list.

Evidence: Project Evaluation Schedule https://outsidenasa.nasa.gov/portal/education/Evaluation_Methodology_and_Schedule.pdf Examples of evaluation reports: Explorer Schools https://outsidenasa.nasa.gov/portal/education/evaluation_report_nes.pdf USRP https://outsidenasa.nasa.gov/portal/education/evaluation_report_usrp.pdf CIP (Curriculum Improvement Project 2005) https://outsidenasa.nasa.gov/portal/education/evaluation_report_cipa.pdf AESP (Aerospace Education Services Project) https://outsidenasa.nasa.gov/portal/education/AESP_evaluation_2004.pdf EarthKam https://outsidenasa.nasa.gov/portal/education/evaluation_report_earthkam.pdf The NASA Education Program's approach to performance measurement and evaluation is outlined in "NASA Education Strategic Coordination Framework: A Portfolio Approach," NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf National Research Council report "NASA's Elementary and Secondary Education Program: Review and Critique" http://www.nae.edu/nae/naepcms.nsf/weblinks/MKEZ-79TJ8X?OpenDocument 2008 Evaluation Solicitation and Statement of Work https://outsidenasa.nasa.gov/portal/education/Evaluation_Solicitation.pdf https://outsidenasa.nasa.gov/portal/education/Evaluation_Statement_of_Work_STEM.pdf NRC Recommendations Chart https://outsidenasa.nasa.gov/portal/education/NRC_Recommendations_Chart.pdf

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The NASA Office of Education fully complies with all requirements of the NASA project management process articulated in the applicable NASA Procedural Requirements (NPRs). NASA's annual budget estimates link budget requests to specified performance standards. While operating plans for FY 2007 and FY 2008 were dramatically altered from the President's request due to budgetary constraints and Congressional redirection, decisions as to how programmatic efforts were reduced were tied directly to ensuring that remaining funds were focused on the highest-priority annual and long-term performance goals. The portfolio review process established by the NASA Education Strategic Coordination Framework is designed to ensure that performance is factored as a key element in budget decisions. Accordingly, the three education outcomes and their associated objectives have been prioritized by the ECC through the use of a group decision-making tool. These prioritizations establish an ideal budget distribution to the three outcomes, influencing budget and resource allocations. As a result of the ECC's prioritization, the majority of Education's budget is aligned with achieving Outcome 1, which focuses on building and maintaining a strong STEM workforce. In addition, through use of an online data collection tool, the Office of Education conducts assessments of the alignment of each NASA Education project to outcomes and objectives. These judgments of project alignment are provided by the project manager of each project, the outcome manager, the Center Education Director, and the relevant Mission Directorate lead.

Evidence: NASA's Annual Budget Documents http://www.nasa.gov/about/budget/ Project plans have been submitted by each NASA Center responsible for implementation of one or more NASA education projects. These plans clearly describe and budget for all necessary program costs, including personnel and financial requirements, and managers are held accountable for conformance to the plan. Similarly, grant projects must submit a budget establishing major cost categories. This budget is reviewed and approved annually and monitored for grantee compliance. Examples of project plans requiring budgets: eEducation Small Project https://outsidenasa.nasa.gov/portal/education/Project_Plan_eEducation.pdf NSTI-MI https://outsidenasa.nasa.gov/portal/education/Project_Plan_STI.pdf USRP https://outsidenasa.nasa.gov/portal/education/Project_Plan_USRP.pdf Harriet Jenkins Pre-Doctoral Fellowships https://outsidenasa.nasa.gov/portal/education/Project_Plan_Jenkins.pdf LTP https://outsidenasa.nasa.gov/portal/education/Project_Plan_Learning_Technologies.pdf NETS https://outsidenasa.nasa.gov/portal/education/Project_Plan_NETS.pdf NPR 7120.5C http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005C_&page_name=main NPR 7120.5D http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D_&page_name=main The Office of Education reports on progress made toward achieving its GPRA performance measures in the Agency's annual Performance and Accountability Report, available at: http://www.nasa.gov/news/budget/index.html FY08 NASA Education Portfolio sorted by Outcome https://outsidenasa.nasa.gov/portal/education/portfolio_by_outcomes_with_totals.pdf

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: NASA's Office of Education continues to resolve the strategic planning deficiencies identified during the FY 2007 PART assessment. Many of the ongoing program improvement plans relate to strategic planning and the Office of Education has taken significant, measurable steps to mitigate those items. The Office of Education addressed the program improvement plan calling for NASA to "Fully execute the new education investment framework ... to complete the strategic alignment of the Education portfolio..." by adopting the strategic planning process codified in "NASA Education Strategic Coordination Framework: A Portfolio Approach." NASA also established a comprehensive portfolio management process that ties all NASA Education projects to annual and long-term goals causing significant restructuring of the education portfolio. The NASA ECC, composed of individuals representing all Agency organizations with a role in education, oversees the entire strategic planning process for the NASA Education Program, ensuring that the program maintains a balanced and effective portfolio of education projects aligned with the official outcomes and objectives. During FY 2007, the Office of Education continued to use the ECC to compile investments into a single portfolio, improve alignment of the portfolio to desired outcomes, communicate between education teams across the Agency, review new project concepts, begin standardizing nomenclature for capturing and reporting investments, evaluating NRC recommendations, creating working groups to address deficiencies, and providing recommendations for improvement to the Assistant Administrator for Education. The NASA Office of Education has also established baselines for the Education Program's outcome, output, and efficiency measures, and has set appropriate targets for measuring success. Furthermore, the Office of Education has established a schedule of annual retreats for all Headquarters staff members to focus on strategic planning. To address the program improvement plans related to NASA's workforce needs and collaboration with other agencies, NASA utilizes interagency MOUs to reduce duplication of effort and leverage other Federal resources for strategic education planning. NASA also established an Education Partnerships Forum process to involve appropriate outside organizations as partners in strategic STEM education efforts. The Office of Education represents the Agency on the Interagency Aerospace Revitalization Task Force, a group of Federal agencies with a vital interest in strategic planning for STEM education to strengthen the science and technology workforce.

Evidence: NASA Education Strategic Coordination Framework: A Portfolio Approach, NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf FY08 NASA Education Portfolio sorted by Outcome https://outsidenasa.nasa.gov/portal/education/portfolio_by_outcomes_with_totals.pdf Education Office Retreat Agenda, August, 2007 https://outsidenasa.nasa.gov/portal/education/retreat_agenda_august_2007.pdf NASA Education Partnerships Forum http://education.nasa.gov/about/team/partner_forum.html NASA-National Science Foundation MOU http://education.nasa.gov/pdf/172012main_NASA-NSF_MOU_2-22rev2%5B1%5D.pdf NASA-FAA MOU https://outsidenasa.nasa.gov/portal/education/FAA_NASA_MOU_.pdf National Park Service MOU https://outsidenasa.nasa.gov/portal/education/NPS-NASA_MOU.pdf NASA Partnership Summit http://education.nasa.gov/about/team/summit.html

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Office of Education regularly uses project performance information to manage programs and improve performance and achieved two major objectives to ensure that consistent and credible performance data are obtained in 2008 and beyond. In December 2007, the program finalized and reviewed with OMB the baselines for each of its performance measures. These baselines are being used in annual planning and budgeting for FY 2008 and beyond, as well as for longitudinal measures of impact for each program area and project. As part of the project realignment to the NASA Center, the Office of Education formalized the requirement for performance monitoring, through MOUs with each Center, by requiring its Education projects (e.g., SEMAA and INSPIRE) to produce and adhere to detailed performance and project plans that include milestone charts that enable tracking of progress and success. Annual performance reviews are held with each Education project and outcome manager to determine successful practices and budgetary decisions. To improve data collection and assessment in the future, NASA has established the System requirements for a single data collection and reporting system to serve as a tool to assist more fully in the management and evaluation of NASA's Education portfolio. It will begin collecting data and metrics in FY09. For FY07 and FY08, data from NASA project managers and external partners will continue to be aggregated and reported through three separate data collection systems: NASA Education, Evaluation, and Information System (NEEIS), the Consortium Management Information System (CMIS), and the Performance Outcome and Student Tracking System (POSTrack). The new database's primary function will be to allow the NASA Education Program to collect timely and credible performance information on all NASA Education projects, including information from program partners. It is designed to be able to acquire and report the outcome, output, and efficiency measure data needed to demonstrate progress toward outcomes and objectives on the part of every NASA Education project. Performance data is also gathered from partners, subject to the legal requirements and terms of the award through the mandatory submission of annual progress reports. The Space Grant consortia annually provide progress and performance data through the CMIS system. The Space Grant project manager uses the reported results to suggest corrective actions. Additionally, the Space Grant program conducts a complete program evaluation every five years to ensure overall satisfactory performance of each participating consortium. (Such an evaluation is in progress at present.) As a result of the previous (2003) evaluation, 33 consortia were found to be satisfactory, 14 were placed on probation, and 5 were recompeted.

Evidence: NASA Education Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf Education Database Project Report https://outsidenasa.nasa.gov/portal/education/Education_Database_Project_Report.pdf INSPIRE benchmarking documents https://outsidenasa.nasa.gov/portal/education/Final_NRC_Lit_Rev_STEM_and_INSPIRE.pdf Business Case Analysis for Education Evaluation Database with Timeline https://outsidenasa.nasa.gov/portal/education/business_case_analysis_database_with_timeline.pdf 2007 Space Grant Data and Reports (19MB Zip File) https://outsidenasa.nasa.gov/portal/education/Space_Grant_Data_and_Reports.zip Selected Examples of 2007 Space Grant Data and Reports (Smaller Individual PDF Files) Arizona 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/arizona_report.pdf Colorado 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/colorado_report.pdf New York 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/new_york_report.pdf Wisconsin 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/wisconsin_report.pdf

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: In accordance with the NASA Employee Performance Communication System (EPCS), all project managers' personal performance plans document their responsibilities for project or program performance, including cost, schedule, and performance results. Clear lines of authority have been developed to ensure project managers, outcome managers, and center education directors are accountable to the Assistant Administrator for Education for cost, schedule, and performance results. All levels of management for any NASA Education project must fully comply with NASA NPR 7120.7, "NASA Information Technology and Institutional Infrastructure Program and Project Management Requirements," and NPR 7120.5C, "NASA Program and Project Management Processes and Requirements." A description of compliance is detailed in each Education project plan. A signed MOU between the Office of Education and Centers, specific to management of the national projects and the content of MOU that addresses expectations/accountability, was required during the project realignment activity. Centers competing for and receiving project management oversight responsibilities also assumed these responsibilities: integrated project planning and direction, including pursuit of innovative partnerships and opportunities for collaboration that align with goals; project budget development and cost phasing plans; project resource allocation and manage resources; integrating budget and performance management; completing and maintaining Contracting Officer's Technical Representative certification; grants management; project metrics assessment and reporting, to include evidence of achievement of NASA Education APGs and measures (output, outcome, efficiency). Education project managers are also required to submit annual performance plans that outline the process to be used to collect and provide data in support of their project's accountability measures. NASA grant officers and technical managers hold grantees accountable for detailed milestone charts, described above. In addition, the NASA Headquarters Office of Procurement and the Center procurement offices review, approve, and finalize grants. Together they provide a system of checks and balances throughout the life of grants to hold grantees accountable for cost, schedule, and performance results. For example, the newly awarded Aerospace Education Services Project (AESP) was required to begin providing monthly and semi-annual reports to show cost, schedule, and performance results. AESP monthly reporting requirements include activities performed during the period; activities planned for the next period; objective and/or strategy changes; budget expenditures during the period as compared to the total budget; and demographic makeup of program participants during the period. Grantees who fail to fulfill the requirements of their grants risk having their projects discontinued. The Space Grant program conducts a complete program evaluation every five years to ensure overall satisfactory performance of each participating consortium. Such an evaluation is in progress at present. As a result of the previous (2003) evaluation, 33 consortia were found to be satisfactory, 14 were placed on probation, and five were recompeted. Eighteen of these underperforming consortia are now achieving results to the satisfaction of the Office of Education. In one instance, NASA Education required/requested a change in project management leadership (completed in Month 2007) and is actively monitoring improvement efforts.

Evidence: Examples of Project Plans eEducation Small Project https://outsidenasa.nasa.gov/portal/education/Project_Plan_eEducation.pdf NSTI-MI https://outsidenasa.nasa.gov/portal/education/Project_Plan_STI.pdf USRP https://outsidenasa.nasa.gov/portal/education/Project_Plan_USRP.pdf Harriet Jenkins Pre-Doctoral Fellowship Program (JPFP) https://outsidenasa.nasa.gov/portal/education/Project_Plan_Jenkins.pdf LTP https://outsidenasa.nasa.gov/portal/education/Project_Plan_Learning_Technologies.pdf NETS https://outsidenasa.nasa.gov/portal/education/Project_Plan_NETS.pdf The NASA Organization: NPD 1000.3C http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPD&c=1000&s=3C Education Director Accountability -From NPD 1000.3C The NASA Organization Sec. 4.13.2.2 http://nodis3.gsfc.nasa.gov/npg_img/N_PD_1000_003C_/OE.doc Other Referenced NPRs may be found here: NASA Program and Project Management Processes and Requirements NPR 7120.5C http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005C_&page_name=main NASA Space Flight Program and Project Management Requirements NPR 7120.5D http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D_&page_name=main AESP Statement of Collaboration and Accountability https://outsidenasa.nasa.gov/portal/education/AESP_statement_of_collaboration.pdf Sample Signed NASA Center MOUs NETS https://outsidenasa.nasa.gov/portal/education/NETS_MOU.PDF USRP https://outsidenasa.nasa.gov/portal/education/USRP_MOU.pdf SEMAA https://outsidenasa.nasa.gov/portal/education/Semaa_MOU.pdf 2007 Space Grant Data and Reports (19MB Zip File) https://outsidenasa.nasa.gov/portal/education/Space_Grant_Data_and_Reports.zip Selected Examples of 2007 Space Grant Data and Reports (Smaller Individual PDF Files) Arizona 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/arizona_report.pdf Colorado 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/colorado_report.pdf New York 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/new_york_report.pdf Wisconsin 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/wisconsin_report.pdf

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Understanding that Congress provides enough budget authority to execute NASA's programs for 12 months, not more, Education's goal is to obligate funds within 12 months, and cost them within 24 months. NASA's OCFO has directed the Mission Directorates and Centers to strive for 100% obligation of FY 2008 funds identified in the February 29th, 2008 Agency Execution Plan (AEP) by year-end. To meet this goal, Education has developed detailed phasing plans, distributed funds to the Centers early, and initiated all the necessary procurement processes to aid in getting funds obligated. As directed by NPR 7120.5C, which establishes guidelines for the approval and obligation of funds, all NASA Education project managers review proposals, maintain regular communications, and evaluate progress reports to ensure that funds are spent according to plan and for intended purposes. The Office of Education works with Center Education Directors to ensure that education project managers are held accountable for obligating funds in a timely manner, spending funds for intended purposes, and reporting funds accurately. The Office of the Chief Financial Officer established effective financial controls at the Centers to allow Education outcome managers and project managers to track Education Program expenditures more closely, and to provide guidance to ensure that funds are obligated in a timely manner. Program and project managers receive monthly financial reports detailing commitments, obligations, and costs for active budget years (currently FY 2007 through FY 2009). In addition, the Office of Education is working hard to overcome structural challenges to the timely obligation of funds. For example, a large proportion of NASA Education Program funds are dedicated to summer internship projects, with the result that many funds cannot be obligated until the last quarter of the Federal fiscal year. In these situations, the Office of Education works in real-time with the receiving institutions and the NASA Office of the Chief Financial Officer to ensure that funding requirements are expedited to the greatest extent possible.

Evidence: NASA Program and Project Management Processes and Requirements NPR 7120.5C http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005C_&page_name=main OCFO Monthly Report - Office of Education Budget Commitments, Obligations, Costing, Planning, Evaluation and Policy Development Monthly Budget Report https://outsidenasa.nasa.gov/portal/education/MARCH_08_MONTH_END_STATUS_REPORT.pdf Goddard Procurement Monthly Grants Reports Monthly Grant Report - FY08 Through March https://outsidenasa.nasa.gov/portal/education/FY08-Education-Log_Grants_through_March.pdf Monthly Grant Report - FY07 Full Year https://outsidenasa.nasa.gov/portal/education/FY07_Full_Education-Log_Grants.pdf

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: NASA Education continues to implement a systemic restructuring of budgets that promote efficiency, cost savings, and appropriate reallocation despite across-the-board budget reductions. For example, NASA Education relies routinely on competitive solicitations to achieve quality services, innovative management approaches, and leveraged resources while implementing projects. For the 2007 call for a support contractor for the Aerospace Education Services Project (AESP), NASA Education established objectives and criteria for the solicitation that sought proposer creativity in delivering educator professional development services despite an anticipated budget reduction. The selected contractor, Pennsylvania State University, delivered a concept that improves effectiveness and eliminates several inefficient "legacy" type services that NASA had been providing for many years. This tightly focused approach will improve the quality and reach of the services provided, all at a lower procurement cost. Likewise, through a Request for Entrepreneurial Opportunities, NASA Education sought partnerships by which to increase distribution and access to education materials while reducing operating costs. Through a Space Act Agreement partnership with an office supply mega-store, NASA realizes cost-savings by not printing, warehousing, shipping, and providing administrative support functions for the service. In 2007, the NASA Explorer School project, faced with a reduction that would prevent selection of new schools for the program, refocused the approach to providing educator professional development services from face-to-face training to the Digital Learning Network. The project realized cost savings through travel, procurement of support facilities, and other expenses, offsetting the budget cut. In 2006-2007, the number of training events increased 62%, and the number of educators served through digital instruction increased 75%: in 2007-2008, data already show an 85% increase in the number of events, and a 95% increase in educators served. Finally, NASA Education strives to make all educator resource materials available through the NASA Education website to allow thousands of materials to be delivered without incurring the printing and other associated costs. Another example of an IT-based efficiency is the cost savings anticipated through the movement of data collection and performance monitoring from three evaluation database systems to one comprehensive database system. Although the complete life-cycle cost of the new system is still being assessed, the annual cost savings of running one system compared to three is expected to be about 50% of the current database support budget. Finally, to improve efficiencies, Education is also consolidating projects. For example, the Office of Education has merged several elements of MUREP (e.g., consolidated the Partnership Awards for Integration in Research [PAIR] and Curriculum Improvement Partnership Awards [CIPA] projects). Education's partnership initiatives achieve synergy among participating organizations and leverage the resources and external expertise of these organizations to achieve planned outcomes with greater efficiency. For example, Space Grant and EPSCoR have matching funding requirements that leverage the Federal investment in the programs.

Evidence: AESP Solicitation http://dml.larc.nasa.gov/aesp/can/ AESP Statement of Collaboration and Accountability https://outsidenasa.nasa.gov/portal/education/AESP_statement_of_collaboration.pdf NES 2007 Evaluation https://outsidenasa.nasa.gov/portal/education/evaluation_report_nes.pdf NES Annual Report and Summary Data https://outsidenasa.nasa.gov/portal/education/NES_FY_2007_Annual_Summary_Report-finaledit.pdf Office Max Space Act Agreement https://outsidenasa.nasa.gov/portal/education/Office_Max_SAA.pdf http://www.nasa.gov/home/hqnews/2003/aug/HQ_03263_consumer.html Education Web Portal Metrics: This portal metrics spreadsheet was prepared manually using data acquired from Urchin, the NASA Portal's metrics reporting system. The spreadsheet format was designed to present data from Urchin in a format that is easy to understand, interpret, compare and contrast. Urchin is a Web-based reporting tool provided by the Portal vendor, eTouch. https://outsidenasa.nasa.gov/portal/education/PortalMetrics.pdf NASA makes extensive use of the library of distance education projects to deliver information and professional development training and to conduct virtual symposiums and kickoff events for members of the formal and informal education communities. A programming schedule is available at http://www.nasa.gov/audience/foreducators/topnav/schedule/index.html CIO Business Case for evaluation database https://outsidenasa.nasa.gov/portal/education/business_case_analysis_database_with_timeline.pdf

Does the program collaborate and coordinate effectively with related programs?

Explanation: NASA employs two principal methods to collaborate and coordinate with related programs. Internally, the ECC serves as NASA Education's mechanism to ensure coordination among the three principal categories of NASA organizations involved in education: the Office of Education, the Mission Directorates, and the NASA Centers. The NASA Education Strategic Coordination Framework documents each organization's requirements for collaboration and coordination. Externally, NASA coordinates its Education Program through the ACC and the NSTC's Education and Workforce Development Subcommittee. NASA uses these forums to collaborate with other agencies (e.g., Department of Education, Department of Energy, NSF) to minimize redundancies and replicate effective practices. NASA also relies on formal partnerships to minimize redundancies with other education efforts. For example, NASA established an MOU with the NSF (March 2007). As part of the MOU, the NASA Education implemented the Joint NASA-NSF Research and Education Opportunities Conference for Principal Investigators, Faculty, and Partners on February 22-24, 2007. Since FY 2007, the Office of Education has represented NASA on the Interagency Aerospace Revitalization Task Force, an organization of Federal agencies established by law to stimulate STEM education and strengthen the STEM workforce. In October 2007, the Office of Education helped lead a Task Force roundtable on STEM education that included representatives of government agencies, industry, and education. The organizations adopted the NASA Education Framework, organizing their discussions around the Framework's areas of involvement: Inspire, Engage, Educate, and Employ. NASA Education is conducting a study, the NASA Education Partners Learning Network and Inventory, to identify successful partnerships and enable the leveraging of the best capabilities represented by those partners. By building such strategic partnerships and linkages between STEM formal and informal education providers, NASA is coordinating education programs, sharing resources, and eliminating wasteful duplication of effort. NASA Education coordinates internationally through the International Space Education Board, which includes education leaders from Japan, France, Canada, Europe, Australia, India, and the United States, to share successful practices and develop collaborative projects.

Evidence: Interagency Aerospace Revitalization Taskforce Report http://www.doleta.gov/pdf/REPORT_Aerospace_2008.pdf Description of the ECC within NASA Education Strategic Coordination Framework, NP-2007-01-456-HQ (Pages 13-14) http://www.nasa.gov/pdf/189101main_Education_Framework.pdf Press release on the first meeting of the Academic Competitiveness Council: http://www.ed.gov/news/pressreleases/2006/03/03062006.html International Education Plan https://outsidenasa.nasa.gov/portal/education/International_Education_Plan.pdf Feature on the NASA-National Science Foundation MOU: http://www.education.nasa.gov/divisions/higher/overview/F_One_Giant_Step_STEM_Education.html Information on the NASA Education Partnership Forum: http://education.nasa.gov/about/team/partner_forum.html Agenda for the joint NASA-NSF Research and Education Opportunities Conference for Principal Investigators, Faculty, and Partners on February 22-24, 2007: http://qemnetwork.qem.org/JointNASA%E2%80%93NSFConfFinalAgenda.htm

Does the program use strong financial management practices?

Explanation: Explanation: The most recent Independent Auditor report for NASA identified two (2) material weaknesses, both of which are repeat items, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: Evidence: NASA's FY 2006 Performance and Accountability Report (www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the Independent Auditor. In addition, the GAO has published numerous reports identifying shortcomings in NASA's new financial management system as well as its financial management processes (example is GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: NASA policy is that Headquarters' responsibilities focus on leadership, policy, and budget. Accordingly, in FY 2006 the Office of Education and the ECC carried out a project realignment, in which management of education projects, with the exception of Space Grant and EPSCoR, transitioned to designated Centers. Education completed the PART performance improvement plan item to establish baselines for all performance measures, and the program has taken significant, measurable steps to address the remaining performance-oriented items. For example, Education has increased the use of technology to manage its portfolio of projects, improve data review, and facilitate monitoring of projects in this newly distributed management approach. The program performs summative and formative evaluations of projects, and is using the results of independent evaluations to restructure and improve individual projects. NASA Education has implemented management decision-making software tools, which allow for tracking project performance and alignment. Education investments across the Agency have now been inventoried and assessed for alignment to outcomes. The Office of Education continues to refine the articulation of roles, responsibilities, and organizational relationships, and continues to brief the NASA Senior Management Council on progress. Office organization has been revised with new positions specified in order to define roles and responsibilities more clearly. Outcome managers continue to be responsible for ensuring that projects are implemented in a way that guarantees the desired outcomes will be achieved, and a portfolio accountability manager was hired to manage the investment capture and analysis effort. Center Education offices provide project management of national projects that leverage expertise in state education standards and requirements in their regions. Overall responsibility for portfolio management, policy development, evaluation, and budget has been retained at NASA Headquarters. Finally, numerous working groups and training sessions on evaluation, data collection, and reporting have been held during ECC and other meetings, so that all Education staff, including those at Centers, understand the importance of this information in providing performance monitoring and project oversight.

Evidence: NASA Education Strategic Coordination Framework, NP-2007-01-456-HQ http://www.nasa.gov/pdf/189101main_Education_Framework.pdf Position descriptions for positions referenced. Evaluation Manager https://outsidenasa.nasa.gov/portal/education/Announcement_evaluation_manager.pdf Outcome Manager https://outsidenasa.nasa.gov/portal/education/Announcement_outcome_manager.pdf Portfolio Manager https://outsidenasa.nasa.gov/portal/education/Announcement_portfolio_manager.pdf Deputy Assistant Administrator for Education Planning, Evaluation, and Policy Development https://outsidenasa.nasa.gov/portal/education/DAA_Education_Planning_Evaluation.pdf Deputy Assistant Administrator for Education Program Integration https://outsidenasa.nasa.gov/portal/education/DAA_Education_Program_Integration.pdf Office of Education Divisions (See Left Column) http://education.nasa.gov/about/factsheet/index.html Office of Education Organization Chart http://education.nasa.gov/about/orgchart/index.html

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Except where otherwise mandated by Congress, the NASA Education Program awards grants through full and open competition and always encourages the participation of new and less experienced participants. Two procurement offices review all solicitations for Education grant awards: the NASA Headquarters procurement office and the procurement office at the NASA Center that will manage the award. All major grants and cooperative agreements are awarded based on reviews by (a) an external panel of peers for educational merit; (b) NASA and external scientists and engineers for content merit and alignment to NASA's education goals; and (c) Mission Directorates for alignment with NASA's research and development interests. NASA Headquarters and Center staff, as well as members of the public, are invited to provide comments on any aspect of the draft solicitations, including the requirements, schedules, proposal instructions, and evaluation approaches. Additionally, comments are requested on any perceived programmatic risk issues associated with performance of the work. Indications of a clear competitive process are an integral part of these reviews. In 2007, these processes were used to award cooperative agreements for the management of the NASA AESP to Pennsylvania State University, and for the Undergraduate Student Researchers Program (USRP) to Universities Space Research Association. These processes will be used for URC, Flight Projects, and the FY08 Congressionally directed projects as they release upcoming solicitations. All NASA Education Program major grant projects, such as Space Grant, EPSCoR, University Research Centers, and the NASA Explorer Schools, select participating institutions through a competitive process. With the exception of grants mandated by law and beyond the complete control of the Agency, all NASA Education grants are awarded based on clear competitive processes that include qualified assessments of merit. While in some instances, the competition may be restricted by legislation to designated participants, such as the defined EPSCoR states, grant awards still are determined through a competitive process. Even in the case of Space Grant awards, all proposals are reviewed for merit and every award is justified, even though the enabling legislation requires that each state have a Space Grant award. Other Congressionally directed appropriations, however, represent an exception to the use of competitive processes. Congressional interest items are often directed to a specific organization, obviating the need for a competitive process. In all cases, however, the NASA Education Program requires the grantee to submit an acceptable proposal before funding is released.

Evidence: The Office of Education adheres to the requirements for grant competition which are formally specified in the NASA Grant and Cooperative Agreement Handbook http://ec.msfc.nasa.gov/hq/grcover.htm Program solicitations and proposal guidance also define the requirements for competition. Examples: INSPIRE Cooperative Agreement:\ http://nspires.nasaprs.com/external/solicitations/summary.do?method=init&solId=%7b09DBA413-21CE-51D5-28EE-14F01A09304F%7d&path=open EPSCoR Solicitation: http://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=121977/EPSCoR_2008_CAN_Dec7%20-%20Final.pdf AESP Solicitation http://dml.larc.nasa.gov/aesp/can/ Space Grant Annual Guidance https://outsidenasa.nasa.gov/portal/education/space_grant_budget_call.pdf

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Each project in the NASA portfolio is assigned a manager who is responsible for providing project oversight. The manager reviews grant proposals and approves only those meeting the stringent qualification criteria listed here: relevance, content, diversity, evaluation, continuity, and partnership sustainability. After a grant is awarded, the manager maintains frequent communications, makes site visits as necessary, reviews status reports, and evaluates performance data. Grant project directors are required by their grant award documents to submit annual progress reports, and the NASA Office of Procurement monitors and tracks grantee expenditures to ensure compliance with Federal regulations. For example, in compliance with project manager guidance, the Space Grant and EPSCoR projects collect highly reliable and comprehensive data, which they use to develop requirements for improvement in every state consortium. They also use the data to conduct reviews every five years as required by legislation. As a result of the most recent five-year review, consortia for five states were determined to be performing inadequately and were recompeted to ensure significant changes in consortium operations. Fourteen states received ratings that indicated a need for improvement and were placed on probation. All 14 of these consortia submitted improvement action plans that were approved and accepted by NASA, and all 14 are now performing adequately. MUREP also produces an annual report on its projects. Beginning in FY 2005, NASA has required each institutional recipient of a congressionally directed appropriation to submit a proposal according to guidance provided by a designated NASA manager. The proposal is then reviewed for compliance prior to award. In addition, the project realignment in FY 2006, which resulted in the migration of most NASA Headquarters Education projects to NASA Centers, has resulted in improved collaboration between the outcome managers at NASA Headquarters and the project managers at the Centers. This collaboration promotes oversight of grantee activity at all levels.

Evidence: Oversight of grantees is in compliance with the NASA Grant and Cooperative Agreement Handbook (Subpart C, 14 CFR Part 1273), available at: http://ec.msfc.nasa.gov/hq/grcover.htm 2007 Space Grant Data and Reports (19MB Zip File) https://outsidenasa.nasa.gov/portal/education/Space_Grant_Data_and_Reports.zip Selected Examples of 2007 Space Grant Data and Reports (Smaller Individual PDF Files) Arizona 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/arizona_report.pdf Colorado 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/colorado_report.pdf New York 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/new_york_report.pdf Wisconsin 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/wisconsin_report.pdf

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Requirements for all grantees to provide annual performance data are detailed in the NRA Grants handbook. NASA grants officers and technical managers hold grantees accountable for schedule (detailed milestone charts), deliverables, and cost. In addition, the NASA Headquarters Office of Procurement and the Center procurement offices review, approve, and finalize grants when awarded. Together they provide a system of checks and balances throughout the life of grants to hold grantees accountable for cost, schedule, and performance results. Grantees who fail to fulfill the requirements of their grants are required, at a minimum, to submit a performance improvement plan and risk having their projects rescoped or discontinued. Results of work accomplished through grant-based activities are disseminated through publications, NASA press releases, education materials, national conferences, and NASA's website. The redesign of the NASA portal has allowed improved navigation and accessibility by the public to NASA's education program and its performance. For example, every MUREP grantee reports performance data to POSTrack, and that information is converted into an annual report that is available from the MUREP website. The Annual White House Reports for Tribal Colleges, Hispanic Serving Institutions, and Historically Black Colleges and Universities are compiled annually to report demographics, identify funded institutions, number of recipients and funding levels. The Office of Education also encourages them to publish in peer-reviewed journals to provide research findings and project results to the education community.

Evidence: MUREP information from CY 1998 to FY 2006: http://murep.nasaprs.com/POSTRACK/reports.cfm NASA Education Performance and Accountability Report (PAR) Data for FY 2005: https://neeis.gsfc.nasa.gov/par_report.html Agency PAR Data: http://www.nasa.gov/news/budget/index.html NASA Education Press Releases http://education.nasa.gov/ednews/nasaeducation/index.html Examples of Briefings, Conferences and Other Communication Activities STS-118 http://www.nasa.gov/mission_pages/shuttle/news/sts118_education_07102007.html STS-118 launch conference https://www.tisconferences.com/sts118/node/2 Engineering Design Challenge http://www.nasa.gov/audience/foreducators/plantgrowth/home/index.html Examples of Grantee Communications Space Grant Regional Meetings http://www.astro.cornell.edu/specialprograms/spacegrant/northeast2007/agenda.html http://www.pc.spacegrant.org/2007_NM_Minutes-Attendees.pdf International Technology Education Association http://www.iteaconnect.org/ http://www.iteaconnect.org/EbD/HE/he.htm United Negro College Fund http://www.uncfsp.org/spknowledge/default.aspx?page=home.default

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: NASA has eight long-term performance goals in PART and significant progress has been made towards these goals. These goals have focused contributions to the NASA and STEM workforce; supporting educators and attracting students to STEM careers and engaging Americans in NASA's mission. During FY 2007, the NASA Office of Education established baseline performance data for all PART measures, and since then it has moved ambitiously to collect data, set targets, and benchmark NASA education programs against other Federal education programs. Baseline data from self-reporting students document that 45% of these students who received support from NASA higher education programs and are eligible for employment are working for NASA or aerospace contractors, or at universities and other educational institutions in occupations relevant to the NASA mission. These students received scholarships, fellowships, or stipends totaling at least $5,000 or participated in internships of at least 160 hours; this includes national higher education and MUREP projects within the Office of Education Portfolio. Responses come only from students who have completed terminal degrees. Six hundred and nineteen students in the NASA workforce pipeline met these criteria from Space Grant consortia, GSRP, Jenkins, and the Pre-Service Teacher Institutes. Of the 407 students reporting, 182 (45%) were employed in the following categories: 24 by NASA, 17 by aerospace contractors, 141 by universities or other educational institutions. Additionally, 177 students (43%) went into STEM-related careers, and 48 students (12%) went into non-STEM careers. The remaining 212 students did not report career progress post-NASA participation. Furthermore, as reported by the Space Grant directors, of the 2,474 students receiving support of at least $5,000 or participated in internships of at least 160 hours; 1,955 of them are still enrolled in their current degree programs. Of the Space Grant students who are not eligible for the workforce and are not still engaged in their original degree programs (519 students), 155 (30%) are seeking advanced degrees in STEM disciplines. Enhancements to our data collection include standardized tracking and surveying across all national higher-education projects. Other PART long-term goals relate to attracting and retaining students in STEM disciplines, achieved through a variety of approaches, including engaging educators as front-line implementers of NASA's content. Fall and spring surveys through NASA Explorer Schools measured 2,395 K-12 educators who participated in NASA training programs were surveyed, with 62% reporting use of NASA products and resources as part of their classroom instruction. When examined, results by NES cohort (2004, 2005, and 2006), indicate that time spent in the program is positively related to increased use of NASA resources and a statistically significant increase in the number of hours teachers used NASA resources and the number of STEM activities implemented in classrooms. Data gathered from NES eFolio indicates the presence of a direction connection between professional development activities and the subsequent implementation of these activities into the classroom/curriculum. The SEMAA project surveyed 11,547 students in grades 4 through 12, asking in what area/discipline they plan to work after completing their studies. Some 4,386 (50%) of the 8,740 respondents indicated plans to work in a STEM career (e.g., astronaut, engineer, scientist, doctor) after completing their studies. There were 408,774 elementary and secondary students participate in NASA educational activities in 2007. PART goals related to engaging the public in NASA's mission have all been met. Through the Museum Alliance, Space Place Network, NASA Visitor Centers and other informal education partners, the Agency also met its annual goal of engaging 350 museums and science centers in major NASA events.

Evidence: NASA Education PART Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf SEMAA Annual Report and Summary Data document https://outsidenasa.nasa.gov/portal/education/2006_SEMAA_AnnualReport_20070501.pdf NES 2007 Evaluation https://outsidenasa.nasa.gov/portal/education/evaluation_report_nes.pdf 2007 Space Grant Data and Reports (19MB Zip File) https://outsidenasa.nasa.gov/portal/education/Space_Grant_Data_and_Reports.zip Selected Examples of 2007 Space Grant Data and Reports (Smaller Individual PDF Files) Arizona 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/arizona_report.pdf Colorado 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/colorado_report.pdf New York 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/new_york_report.pdf Wisconsin 2007 Space Grant Report https://outsidenasa.nasa.gov/portal/education/wisconsin_report.pdf Harriet Jenkins Pre-Doctoral Fellowship Program (JPFP) http://murep.nasaprs.com/REPORT/2006_reports/Jenkinsreport_PO_06_no%20pictures.pdf GSRP Graduate Student Researchers Program https://outsidenasa.nasa.gov/portal/education/evaluation_report_gsrp.pdf USRP Undergraduate Student Research Program https://outsidenasa.nasa.gov/portal/education/evaluation_report_usrp.pdf CIPA Curriculum Improvement Partnership Award http://murep.nasaprs.com/REPORT/2006_reports/CIPAreport_PO_06_no%20pictures.pdf

Does the program (including program partners) achieve its annual performance goals?

Explanation: In FY 2007, NASA established outcomes, objectives, measures, and baselines for each PART measure. NASA has seven annual PART performance measures. In FY07 NASA exceeded the annual PART target for new or revised courses (99 courses developed). Two hundred institutions were served in designated EPSCoR states, exceeding the target of 132. NASA reached 9,746 underserved and underrepresented students through its GSRP, USRP, Space Grant and MUREP programs, exceeding the target of 8,500. The Elementary and Secondary Education Program also successfully met the FY 2007 annual performance goals in the PAR. One stated goal was to 'select 100 student experiments, involving 1,000 students, to participate in the Flight Projects program.' The Flight Projects office involved approximately 18,200 middle school and 54 undergraduate students in authentic, first-hand in NASA mission activities through the ISS EarthKAM research opportunities. Five middle school students worked on each EarthKAM science investigation, for a total of 3,040 student experiments. These results are reflected in the PART long-term goal of the percentage increase in number of elementary and secondary participants in NASA instructional and enrichment activities. Through the Museum Alliance, Space Place Network, NASA Visitor Centers and other informal education partners, the Agency also met its annual PART measures of engaging 350 museums and science centers in major NASA events. All solicitations opened in FY 2007 are now aligned to the identified annual PART performance measures and the solicitations contain detailed guidance on performance expectations for the awardees. NASA has substantially revised its NRA Proposer's Guidebook to include detailed instructions on how education grantees should document achievement of their performance goals; NASA Education project managers enforce that mandate through rigorous program oversight. Finally, Education scored a rating of "Green" on all GPRA measures, including APGs, in the Agency's 2007 PAR Annual Performance Report. The NASA education program has achieved its annual performance goals as evidenced by the coursework and student participation data collected from higher education projects. Data gathered demonstrates engagement of general public and elementary and secondary participants in NASA education opportunities. Achievement of these annual performance goals advances the development of the future NASA and STEM workforce.

Evidence: NASA Education PART Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf GSRP Annual Report https://outsidenasa.nasa.gov/portal/education/evaluation_report_gsrp.pdf NEEIS FY 2007 Final EarthKAM Report https://outsidenasa.nasa.gov/portal/education/Program_Report_EarthKAM_Final_2007.pdf FY 2007 PAR Annual Performance Report reporting on GPRA measures http://www.nasa.gov/pdf/210016main_FY07_Annual_Performance_Report.pdf Space Grant Annual Budget Call https://outsidenasa.nasa.gov/portal/education/space_grant_budget_call.pdf The EPSCoR Solicitation contains detailed guidance on performance expectations for awardees http://nspires.nasaprs.com/external/viewrepositorydocument/cmdocumentid=121977/EPSCoR_2008_CAN_Dec7 - Final.pdf The USRP Solicitation contains detailed guidance on performance expectations for awardees http://nspires.nasaprs.com/external/solicitations/summary.do?method=init&solId={B077D1EE-CB7C-C216-B792-E55FAE70F282}&path=past AESP Cooperative Agreement Notice http://dml.larc.nasa.gov/aesp/can/ NASA NRA Proposers Guidebook http://www.hq.nasa.gov/office/procurement/nraguidebook/

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Per baselines established in the Fall 2007 PART update, some 92,851 students, 14,032 teachers, and 5,877 other individuals (e.g., administrators, visitors, dignitaries, etc.) participated in a Digital Learning Network event. Dollar invested per number of people reached via e-education technologies was lowered from $0.048 (or 4.8 cents) to $0.032 (or 3.2 cents), as reflected in one of Education's efficiency measures. The number of people reached is estimated based on page views (45,305,795). The Office of Education established baseline data for the cost per participant in its elementary and secondary education programs ($13.18 per participant). NASA will work to achieve a reduction in the cost per participant in out years, meeting targets for Education's other efficiency measure. These efficiencies include revised project strategies; increased use of technology to decrease travel; decreasing technology cost; and project consolidation within Elementary & Secondary Program. Ongoing review and examination of the NASA education portfolio enables the Office of Education to improve alignment to outcomes, eliminate redundancies, and achieve improved efficiencies in management infrastructures. To identify these efficiencies, NASA Education has documented all investments in its portfolio. As a result of data gathered through this process, during the past several years the Office of Education has cancelled 13 projects (e.g., Faculty Awards for Research and Summer Faculty Fellowships), phased out another 13, and restructured 18 more (e.g. CIPA and PAIR). Also, the administrative cost of each project within the Office of Education has been reviewed by outcome managers with the goal of identifying and implementing cost savings and efficiencies (e.g. establishing an on-line application process for all participants). To achieve further efficiencies, all NASA Education project management has been transitioned away from headquarters to designated Centers, aligning with the Agency philosophy that project management is better supported in the field than in a policy office at headquarters. Education scored a rating of "Green" on both GPRA efficiency APGs in the Agency's 2007 PAR Annual Performance Report. As part of the effort to improve cost effectiveness, improve efficiency, and leverage resources within the Agency, the Exploration Systems Mission Directorate utilized the Space Grant network and infrastructure to implement two higher education projects focusing on engineering workforce and faculty development. Also, an integrated management approach between the Educator Astronaut and Education Flight Projects began, in order to maximize services, consolidate tasks, reduce costs, and increase efficiencies in operations and overall management. Additionally, starting in FY 2007, the Exploration Systems and Space Operations Mission Directorates began collaborating on developing individual projects to reduce overhead and increase project efficiencies. NASA will continue to use internet- and web-based technology to deliver content to reach larger numbers of participants. NASA is making increasing use of distance learning technologies to boost efficiency by reaching greater numbers of participants per unit of investment. Finally, a new consolidated cost-effective NASA Education evaluation database will enhance the monitoring of individual program and project information to facilitate frequent analyses of the portfolio, allowing managers to identify additional effectiveness and efficiency.

Evidence: NASA Education PART Measures and Targets with Baselines http://education.nasa.gov/pdf/219823main_NASA%20Education%20FY07-11Metrics%20(2).pdf Project Plan - Educator Astronaut and Education Flight Projects https://outsidenasa.nasa.gov/portal/education/Project_Plan_Educator_Astronaut_and_Flight_Projects.pdf NES Annual Report and Summary Data https://outsidenasa.nasa.gov/portal/education/NES_FY_2007_Annual_Summary_Report-finaledit.pdf Education Web Portal Metrics: This portal metrics spreadsheet was prepared manually using data acquired from Urchin, the NASA Portal's metrics reporting system. The spreadsheet format was designed to present data from Urchin in a format that is easy to understand, interpret, compare and contrast. Urchin is a Web-based reporting tool provided by the Portal vendor, eTouch. https://outsidenasa.nasa.gov/portal/education/PortalMetrics.pdf FY 2007 PAR Annual Performance Report reporting on GPRA measures http://www.nasa.gov/pdf/210016main_FY07_Annual_Performance_Report.pdf

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: NASA's Education portfolio was included in the 2007 report of the ACC, which summarized the STEM education programs implemented by Federal agencies and set forth ways to improve and coordinate these programs. Based on the inventory in that report and on working group discussions, it is clear that NASA's education program compares favorably with those of other agencies--for example, in NASA's efforts to measure the results of its programs and in its development of an independent evaluation approach. The recommendations of the ACC report were used by the NSTC Education and Workforce Development Subcommittee in the development of its draft report, "Federal Agency Efforts to Strengthen the Evaluation of Federal STEM Education Programs." In 2007, the National Research Council of the National Academies conducted a review and evaluation of NASA's pre-college education program. It included a review of the extent and effectiveness of coordination and collaboration between NASA and other Federal agencies that sponsor science, technology, and mathematics education activities. The NRC Committee concluded that NASA has good collaboration with cross-agency federal programs, such as the GLOBE program, a partnership between NASA and the NOAA. The NRC committee also commissioned three white papers which: 1) provided a critique of existing external evaluations of NASA's K-12 education projects; 2) provided an analysis of the NASA Explorer Schools Project in the context of what is known about successful models for comprehensive and subject specific school reform; 3) compared the longitudinal model of INSPIRE with successful models from multiyear projects focused on engaging students in science and engineering. These three papers were valuable resources for the committee in developing their conclusions and recommendations. Also, the NASA Explorer School evaluation plan received favorable comments during the NSTC agency review process. NASA is in the process of conducting a rigorous evaluation of its programs. This level of development compares favorably with the situation of agencies cited in the NSTC report that have also already established a mechanism (i.e., via external contract or internal working group) to develop rigorous evaluations or an evaluation mechanism. NASA has initiated development and implementation of tools and systems that will improve its evaluation and data collection efforts. NASA has also contracted with a firm that has begun discussions with other government agencies (e.g., Department of Energy, Department of Education, NOAA, EPA, and NSF) and not-for-profit organizations to compare NASA education performance against external benchmarks which will assist in validating performance targets and overall approaches to program evaluation. Finally, NASA SEMAA was a 2007 finalist for the Innovations in American Government Award Program. The Innovations in American Government Program annually selects exemplary models of innovation and performance in government. The performance of the NASA Education program compares favorably to other government, private, and education programs with similar goals as evidenced by information gathered through interagency committee studies, Elementary and Secondary NRC study, and work of the NSTC. NASA compares favorably to the metrics, goals and funding information provided by the ACC.

Evidence: U.S. Department of Education, "Report of the Academic Competitiveness Council, 2007" http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf DRAFT NSTC Report: "Agency Efforts to Strengthen the Evaluation of Federal STEM Education Programs" https://outsidenasa.nasa.gov/portal/education/DRAFT_NSTC_Report.pdf 2007 NASA Education Highlights Report http://www.nasa.gov/audience/foreducators/topnav/materials/listbytype/2007_NASA_Education_Highlights.html

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In FY 2007, the National Research Council conducted a thorough review and critique of the NASA Elementary and Secondary Program. The resulting report identified four broad areas for improving NASA's efforts in STEM education. The first recommendation of the National Research Council stated NASA should continue to engage in education activities at the K-12 level, designing its K-12 activities so that they capitalize on NASA's primary strengths and resources, which are found in the Mission Directorates. The Office of Education is reviewing the recommendations for implementation, and it used the report to guide resource allocations in the President's FY 2009 budget request. NASA is working with the NSTC Education and Workforce Development Subcommittee to define an approach and a standard for determining how to evaluate education programs throughout the government. Based on these discussions, it appears that NASA's approach is consistent with those of other agencies, and the degree to which evaluations have been conducted or will be conducted is also consistent. In compliance with NSTC requests, NASA has submitted an evaluation plan, approved by OMB, which defines how each project will be evaluated (on a five-year cycle) according to the NSTC standards for rigor. In FY 2007, external reviews were conducted on the NASA Explorer Schools (by Paragon TEC, Inc.) and SEMAA projects (by the Ash Institute for Democratic Governance and Innovation). In advance of a new competitive solicitation, NASA commissioned an external review of the Undergraduate Student Research Project, published in 2007. All the project's short term outcomes were met, and the sixth year of the project was determined very successful. The recommendations were incorporated into the FY07 Announcement of Opportunity. Independent evaluations are a priority for NASA Education in order to make decisions about the success and benefits of projects. In FY 07, the Office of Education reviewed or evaluated nine projects or 25% of the budget. A yearly budget for independent evaluations has been established. In April 2008, NASA will select an independent evaluation organization to conduct evaluations of all projects over a five-year period.

Evidence: National Research Council Report: NASA's Elementary and Secondary Education Program: Review and Critique http://www.nae.edu/nae/naepcms.nsf/weblinks/MKEZ-79TJ8X?OpenDocument DRAFT NSTC Report: "Agency Efforts to Strengthen the Evaluation of Federal STEM Education Programs" https://outsidenasa.nasa.gov/portal/education/DRAFT_NSTC_Report.pdf Project Evaluation Plan With Schedule https://outsidenasa.nasa.gov/portal/education/Evaluation_Methodology_and_Schedule.pdf SEMAA Annual Report and Summary Data document https://outsidenasa.nasa.gov/portal/education/2006_SEMAA_AnnualReport_20070501.pdf NES 2007 Evaluation https://outsidenasa.nasa.gov/portal/education/evaluation_report_nes.pdf NES Annual Report and Summary Data https://outsidenasa.nasa.gov/portal/education/NES_FY_2007_Annual_Summary_Report-finaledit.pdf