Detailed Information on the
NASA Innovative Partnerships Assessment

Conduct regular independent evaluations to assess the program's effectiveness and efficiency against the program's established objectives and performance goals and apply resources based on the results.

Complete, institutionalize, and document the enhanced Intellectual Property (IP) management process to enable NASA's increased use of IP to meet its mission goals.

Collect consistent performance information annually for all program activities in a system that meets performance data verification and validation requirements and report the data against the program's established metrics and targets.

Measure: Number of technologies infused into NASA programs/projects from total Innovative Partnerships Program portfolio.

Explanation:Technology infusion is the ultimate goal for much of what IPP does. Sources of technology infused could come from any element of the IPP portfolio including Small Business Innovative Research (SBIR)/Small Business Technology Transfer (STTR), Seed Fund, Centennial Challenges, Facilitating Access to Space Environment for Technology Development and Training (FAST), and other partnerships. Infusion means the technology is being used by a Mission Directorate or one of their programs/projects. This could mean it has flown on a mission, been adopted for use by a mission, been picked up for further development, or otherwise has improved the project (e.g., used in trade space driving better technology solution). This measures technologies newly-infused in the performance year, recognizing the long-term nature of infusion in that the original IPP funding for the technology may be been in prior years. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Measure: Proportion of SBIR/STTR technologies successfully infused into NASA programs/projects (ratio of the number of technology infusion successes into NASA programs/projects from the prior five years of SBIR/STTR Phase II contracts relative to the number of SBIR/STTR contracts issues over the prior five years).

Explanation:A primary objective for NASA's Small Business Innovative Research (SBIR)/Small Business Technology Transfer (STTR) program is to develop technologies needed by NASA's Mission Directorates and their programs/projects. This measure will look at the number of technology infusion successes from the prior five years of SBIR/STTR Phase II awards relative to the total number of SBIR/STTR Phase II awards over the last five years. This is taken over five years, because the infusion process can be lengthy, so a five-year window gives a better status of how well the SBIR/STTR portfolio is being infused than just the current year would. (See measure 1 for definition of infusion.) Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Measure: Number of technology readiness level (TRL) advancements achieved from the Innovative Partnerships Program portfolio of technology development.

Explanation:Advancing the maturity of technologies is essential for infusion, and measuring progress is achieved by monitoring TRLs for the technologies and tracking increases. Many elements of the IPP portfolio contribute to advancing technology maturity that can be measured by increases in TRL. Elements include Small Business Innovative Research (SBIR)/Small Business Technology Transfer (STTR) contracts, Seed Fund projects, Centennial Challenges competitions, technologies being demonstrated in the Facilitating Access to Space Environment for Technology Development and Training (FAST) project, as well as other partnerships. IPP works with all Mission Directorates to ensure all technology in our portfolio is filling a technology need for NASA. Standardized tools for making TRL assessments more objective will be used. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Measure: Number of SBIR/STTR Phase III contracts initiated or expanded.

Explanation:Phase III contracts are an important indicator of how well Small Business Innovative Research (SBIR) technologies are being infused, as evidenced by being funded from other sources by NASA directly or through NASA prime/sub-contractors. The primary goal of NASA's SBIR/Small Business Technology Transfer (STTR) program is to produce innovative technology for NASA's missions. Therefore, the number of Phase III contracts (i.e., procurement contracts) is an important measure since it reflects actual procurement of product or service or procurement of additional development of technologies of high interest to NASA missions. Total number of SBIR/STTR Phase III contracts is a key indicator of technology being successfully targeted to Agency needs. The infusion and Phase III process can be lengthy at times, so initiation or expansion of SBIR/STTR Phase III contracts building on Phase I/II SBIR/STTR awards in any prior year is measured in the year of performance. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Measure: Total number of licenses generated from the Intellectual Property (IP) portfolio of patents from the last five years relative to the number of patents in that portfolio.

Explanation:This is a measure of how efficient NASA is at generating licenses from its portfolio of patented technologies. The measure identifies the total number of items in the IP portfolio (defined as total number of patent applications filed, since licenses can be granted against patent applications) over the last five years, and also identifies the number of licenses generated (measured as instances of movement of NASA IP items) from that five-year portfolio. The ratio of licenses from the portfolio over number of patents in the portfolio reflects how well the patents have been selected (did they have good potential for licensing) and how effectively NASA has pursued licensing opportunities. This measure is taken over multiple years, because the IP management and licensing process is lengthy. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Measure: Number of notable technology transfer successes documented annually in NASA's Spinoff publication.

Explanation:Each year NASA documents 40-50 of the most notable examples of successful transfer and commercialization of NASA-derived technology. These examples cover a range of applications, including health and medicine, transportation, public safety, consumer goods, homes and recreation, environmental and agricultural resources, computer technology and industrial productivity. The annual target of published Spinoff examples is long-standing. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend.

Measure: Number of New Technology Reports (NTRs) produced each year, representing the new technologies available for potential transfer.

Explanation:NTRs, generated by NASA civil servants and contractors, provide the basis for NASA's patenting and licensing of intellectual property to which it has title. IPP proactively works with these civil servants and contractors to identify and document new technologies through in-reach activities include educating NASA technologists on the importance of reporting new technologies. IPP facilitates the reporting of new technologies through its online electronic NTR reporting (eNTRe) module which provides for relatively easy electronic submission of NTRs. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Measure: Successful application of SBIR/STTR technologies into commercial products or services (ratio of the number of technology commercialization successes from the prior five years of SBIR/STTR Phase II contracts, relative to the number of SBIR/STTR contracts issues over the prior five years)

Explanation:An important objective for the Small Business Innovative Research (SBIR)/Small Business Technology Transfer (STTR) program is commercialization of the technology. Commercialization encompasses the application of innovative technology to products and services for NASA mission programs, other government agencies and non-government markets. This measure will look at the number of commercial products/services from the prior five years of SBIR/STTR Phase II awards relative to the total number of SBIR/STTR Phase II awards over the last five years. This is taken over five years, because the commercialization process can be lengthy, so a five-year window gives a better status of how well the SBIR/STTR portfolio is being commercialized than just the current year would. Data prior to the initial 2008 PART assessment is included to show IPP's performance trend, and used to set the 2008 target as a performance baseline.

Is the program purpose clear?

Explanation: The purpose of NASA's Innovative Partnerships Program (IPP) is defined in NASA's Strategic Plan: "The Innovative Partnerships Program (IPP) will facilitate partnering with the U.S. private sector, and leverage private sector resources, to produce technologies needed for NASA missions. The IPP and NASA's Mission Directorates will identify new opportunities to adopt technologies developed through innovative partnerships." Specifically, IPP serves three major purposes. (1) IPP helps with technology infusion, obtaining needed technology and capabilities for NASA's Mission Directorates, programs, and projects through investments and partnerships with industry, academia, other government agencies, and National Laboratories. IPP increases NASA's connection to emerging technologies in external communities, particularly those not normally involved in aerospace activities. (2) IPP facilitates technology transfer through dual-use partnerships and licensing, providing important socio-economic benefits within the broader community. Technology transfer was mandated for NASA by the 1958 the National Aeronautics and Space Act. IPP is the responsible party in NASA for implementing this portion of the Act and several related legislative mandates: transfer technology, per 15 USC Sec. 3710 (Utilization of Federal technology) and Executive Order No. 12591 (Facilitating Access to Science and Technology); competitively award cash prizes to stimulate innovation in basic and applied research, technology development, and prototype demonstration that have the potential for application with NASA, per Public Law 109-155, Section 314 (Prize Authority); and provide research and development opportunities for small, high-technology companies and research institutions, per the Small Business Innovation Research Act of 1982 (PL 97-219) and the Small Business Technology Transfer Program Act of 1992 (PL 102-564). (3) IPP helps secure the government's intellectual property rights in NASA's inventions and makes them available for commercial application by U.S. industry and other public benefit. The 1958 Space Act provides NASA with the authority to patent inventions to which the Agency has title. The Act also provides for NASA to retain title to all NASA inventions and intellectual property created using government funds, unless specifically waived by the NASA Administrator.

Evidence: ?? National Aeronautics and Space Act, Public Law Number 85-568, 72 Stat. 426 438 (Jul. 29, 1958), available at http://www.nasa.gov/offices/ogc/about/space_act1.html ?? Since 1980, Congress has enacted a series of laws to promote technology transfer and allow commercial use of government-funded inventions. These laws encourage the sharing of technology and resources between federal laboratories and private industry, including personnel, facilities, methods, expertise, and technical information in general. These statutes include: - Bayh-Dole Act of 1980 - Stevenson-Wydler Technology Innovation Act of 1980 - Federal Technology Transfer Act of 1986 - NASA Authorization Act of 2005 (http://thomas.loc.gov/cgi-bin/query/D?c109:5:./temp/~c109KxEZzf::) - 15 USC Sec. 3710 (Utilization of Federal technology); and - Executive Order No. 12591 (Facilitating Access to Science and Technology). ?? In addition, the Small Business Innovation Research Development Act of 1982, the Small Business Technology Transfer Program Act of 1992, and subsequent reauthorizing Small Business Innovative Research (SBIR)/Small Business Technology Transfer (STTR) legislation provides the opportunity for small, high technology firms to provide mission technology for NASA. IPP administers the SBIR/STTR program on behalf of the Agency. The Small Business Administration SBIR/STTR site provides information on the federal SBIR/STTR program (http://www.sba.gov/SBIR/indexwhatwedo.html). ?? 2006 NASA Strategic Plan, available at www.nasa.gov/budget/

Does the program address a specific and existing problem, interest, or need?

Explanation: Given limited resources for technology development within NASA, infusion into NASA missions of technology developed jointly in partnerships with industry, academia, other federal agencies, and other external entities has become an increasingly important function for NASA. IPP addresses this need through a portfolio of program elements that facilitates and provides for leveraging of partner expertise and funds to develop technologies critical to NASA's mission research and development goals. The SBIR/STTR programs stimulate technological innovation in the private sector, strengthen the role of small businesses in meeting Federal research and development needs, increase the commercial application of these research results, and encourage participation of socially and economically disadvantaged persons and women-owned small businesses. IPP's Seed Fund, Centennial Challenges, and Facilitated Access to the Space environment for Technology Development and Training (FAST) program elements develop and mature technologies at less cost through partnering and to infuse those technologies into NASA's missions. The selected technology areas are targeted to technology gaps in NASA's Mission Directorates technology portfolios. IPP also ensures broad public benefit from NASA-developed technologies. Over the course of its history, IPP has originated and negotiated licenses and related partnerships with the private sector to facilitate the transfer of NASA-developed technologies for commercial application and other public benefits (as documented in Spinoff Magazine).

Evidence: ?? The Small Business Administration SBIR/STTR site provides information on the federal SBIR/STTR initiative (http://www.sba.gov/SBIR/indexwhatwedo.html). ?? NASA Innovative Partnerships Program Web site (http://www.ipp.nasa.gov/) ?? NASA's annual Budget Estimates (available at www.nasa.gov/budget) provides information on NASA's R&D goals and constraints and details on IPP's projects and how they are targeted to help NASA meet its technology development and acquisition needs. ?? Centennial Challenges Web site (http://centennialchallenges.nasa.gov/cc_challenges.htm) - NASA Spinoff (http://www.sti.nasa.gov/tto/) - NASA Tech Briefs (http://www.techbriefs.com/)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Although SBIR/STTR is a government-wide initiative, IPP is designed to specifically serve NASA, through the transfer of NASA-developed technology for broad public benefit and the identification and development of technology needed by NASA's program and projects through innovative partnerships and funded activities with small businesses. Although NASA's Mission Directorates acquire technologies through their own agreements, IPP is designed to tap non-traditional sources, expanding the range of technology options beyond what the Mission Directorates may acquire on their own and coordinating the availability of those technologies across NASA. IPP also helps NASA acquire technologies in cost-effective ways by expanding the pool of potential technology suppliers, increasing competition and stimulating the aerospace technology research and development community, and offering a wider menu of partnering options, including prize competitions where money is awarded only when the technology target is met. IPP has some unique activities such as Centennial Challenges, FAST, and Innovation Incubator that are not present at other agencies. The concept for the Centennial Challenges was inspired by the X-Prize challenges, a prize competition to stimulate commercial aerospace technology development run by the private, non-profit X-Prize Foundation. With the Centennial Challenges, NASA tailored the prize competition concept to meet the Agency's specific technology needs. However, IPP works with the X-Prize Foundation and other non-profit intermediaries to arrange and promote NASA's Centennial Challenges, creating synergy with these organizations while serving NASA's unique needs. Any redundancies IPP has with other Federal and private efforts are justified based on IPP's overall rewards to NASA, the program's ability to attract new, unique partners to NASA research and development, and its proven ability to transfer out NASA-developed technologies for public benefit. A NASA-commissioned Special Review Team (SRT) noted in its 2006 report that there is inherent overlap between IPP and NASA's General Counsel in regards to technology transfer, licensing, and partnerships. Using SRT recommendations, IPP has strengthened its relationship with the General Counsel to better define areas of responsibility, take advantage of synergies, and solidify IPP's role in assessing candidates for patents, and licensing.

Evidence: ?? NASA Innovative Partnerships Program Web site (http://www.ip.nasa.gov/) ?? Centennial Challenges Web site (http://centennialchallenges.nasa.gov/cc_challenges.htm) ?? Small Business Administration SBIR/STTR site provides information on the federal SBIR/STTR initiative (http://www.sba.gov/SBIR/indexwhatwedo.html). ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request)

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Over the past several years, IPP has corrected program design flaws identified by NASA and independent reviewers. A review of the program conducted in 2004 by the National Academy of Public Administration concluded that IPP would be most effective as a Mission Support Office, reporting directly to the Administrator's office, rather than an element within any particular Mission Directorate. A special review team (SRT) chartered by NASA Program Analysis and Evaluation (PA&E) in 2006 made the same recommendation. As a result, NASA reorganized IPP to report directly to the Associate Administrator and the Deputy Administrator, ensuring the attention and support of NASA's leadership. IPP established a stable organizational and operating structure at Headquarters, with branch offices at each of the Centers, meeting another recommendation made by the SRT. By accepting and acting upon the recommendations made by the SRT, IPP is building on the strengths noted in the SRT study and achieving the anticipated levels of performance. IPP (and its predecessor organizations) has a long history of demonstrated effectiveness. In transferring technology for public benefit, IPP has documented over 1,600 examples in the annual Spinoff publication. An extensive Spinoff Web site is searchable by technology, firm and state. Other federal agencies and international space agencies often look to NASA's technology transfer program as a model of effectiveness. Tech Briefs, a NASA publication and website, provides broad dissemination of NASA technology available for transfer. The SBIR/STTR program is a proven success, as demonstrated by its application at many federal agencies. Furthermore, NASA IPP has always met its Government Performance Requirements Act (GPRA) performance measures, as documented in annual Performance and Accountability Reports. Although the SBIR/STTR programs have spending constraints mandated by legislation, IPP's program design and processes ensure that all agreements made through SBIR/STTR serve specific Mission Directorate needs and are effectively managed to produce maximum results for the resources invested. The overall benefits to NASA--through tapping unique sources for technology development, encouraging interest from non-traditional communities, and stimulating the economic growth of small and disadvantaged technology providers--outweigh challenges posed by the spending constraints.

Evidence: ?? IPP website http://www.ipp.nasa.gov, and Spinoff website at http://www.sti.nasa.gov/tto/. ?? Tech Briefs is available at www.techbriefs.com ?? Performance and Accountability Reports, available at www.nasa.gov/budget . ?? SBIR reauthorization testimony http://democrats.science.house.gov/Media/File/Commdocs/hearings/2007/tech/26jun/comstock_testimony.pdf ?? Technology Transfer: Bringing Innovation to NASA and the Nation, National Academy of Public Administration, November 2004, available at http://napawash.org/Pubs/NASATechTransfer12-14-04.pdf. ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: IPP's design ensures that the needs of our customer's (the Mission Directorates) and our partners are met. IPP works closely with each Mission Directorate to ensure that IPP's technology investments and partnerships are addressing priority needs and complement their ongoing technology activities. Mission Directorates and Centers are involved in identifying technology needs and evaluating candidate IPP projects. (This organizational and process design meets a chief recommendation of the NASA PA&E-commissioned special review team report to improve the effectiveness of IPP.) In addition, IPP includes a Headquarters Chief Technologist, to assess Agency-wide technology needs in close coordination with the Mission Directorates, and a "Communications" function, which includes NASA-internal and external program outreach. IPP's budget request is based on a detailed analysis of Mission Directorate and Center needs and the budget allocation is targeted accordingly, with IPP's mission objectives in mind. With an office at each Center, IPP can ensure that each Center's specific capabilities are utilized and program/project needs are met. IPP holds quarterly program reviews as well as bi-weekly WebEXs (videoconferences enabled by Internet technology) with all NASA Center IPP offices to ensure adherence to IPP's mission with respect to each of IPP's program elements. IPP provides an annual Statement of Assurance to the NASA Administrator, based on a very detailed program analysis, to validate that the program is producing results that are highly focused the program's mission objectives.

Evidence: ?? The IPP website has examples of successes that demonstrate that the program design is effectively targeted: - http://sbir.gsfc.nasa.gov/SBIR/successvideo.html - Activities unique to each field center can be accessed via this site: http://www.ip.nasa.gov/field_centers.htm - Over 1,600 searchable technology transfer successes are documented at this site: http://www.sti.nasa.gov/spinoff/database - Recent press releases about IPP are at this site: http://www.ip.nasa.gov/news.htm - Recent papers provide further evidence: http://www.ip.nasa.gov/pdf/ieeeac_paper_138.pdf; http://www.ip.nasa.gov/pdf/isu_symp_2008_final.pdf ?? Upon request, IPP can provide samples of internal-only documents: the budget spreadsheets used to track IPP budget allocation, weekly reports that summarize accomplishments across all program elements, the latest Statement of Assurance. ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: All of IPP's long-term performance measures are outcome-oriented and focus on two of IPP's primary purposes: technology infusion into NASA and transferring technologies out of NASA for use by other government agencies and the public. One of the long-term efficiency measures also measures IPP's ability to manage NASA's intellectual property. Two of the long-term outcomes measure performance within IPP's largest program element, Small Business Innovative Research (SBIR). The targets measure performance by IPP programs line itemed in NASA's budget request (e.g., SBIR/STTR, Seed Fund, FAST, Centennial Challenges, and other partnerships) or specifically IPP's largest program, SBIR. All performance measures complement IPP's GPRA measurements, creating synergy in performance reporting and providing a holistic view of IPP performance.

Evidence: ?? The FY 2008 (Updated) and FY 2009 Performance Plans, available in the FY 2009 Budget Estimates (www.nasa.gov/pdf/210019main_NASA_FY09_Budget_Estimates.pdf); see the Management and Performance section. ?? IPP's PART measurements on ExpectMore.gov [specific address to be added when it becomes available on public site.] ?? IPP's budget request is available in NASA's FY 2009 Budget Estimates (www.nasa.gov/pdf/210019main_NASA_FY09_Budget_Estimates.pdf); see the Cross-Agency Support section.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has ambitious targets and timeframes for its measures.

Evidence: none

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: IPP has several annual outputs that provide incremental measurements of IPP's critical technology development (as measured by technology readiness levels (TRLs), technology infusion, technology transfer, and licensing activities that roll up to show progress toward the long-term measurements. These PART measurements are designed to be similar or complimentary to NASA's GPRA measurements in order to provide a consistent and holistic view of IPP performance.

Evidence: ?? Performance Plans, available in the FY 2009 Budget Estimates (www.nasa.gov/pdf/210019main_NASA_FY09_Budget_Estimates.pdf).. ?? IPP's PART measurements on ExpectMore.gov [specific address to be added when measurements become available on public site]

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has ambitious targets and baselines for its measures.

Evidence: None

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Leveraging of NASA resources to produce technology for NASA at less cost is central to IPP's mission. IPP partnerships require that partners contribute cash or in-kind resources (e.g., facilities, technical expertise, and laboratory samples) to the joint technology development effort. For example, the leveraging achieved in IPP's Seed Fund program element approximated 4:1 in terms of IPP cash resources applied. Technology developed through Seed Fund and other partnering is targeted to explicit technology gaps of the Mission Directorates and, therefore, is consistent with IPP's long-term goal of providing significant technology for infusion into NASA's missions. IPP has periodic reviews of Seed Fund projects to ensure progress towards program goals. Technology developed under NASA's SBIR/STTR, as required by the Solicitation, is targeted to Mission Directorate technology needs and, therefore, is consistent with IPP's long-term goal of providing significant technology for infusion into NASA's missions. IPP conducts rigorous coordination with Mission Directorates during development of the solicitation and evaluation and prioritization of proposals, to ensure funded activities are consistent with the goal of technology infusion. Technology that becomes the subject of Centennial Challenges is determined by NASA's Mission Directorates, competition rules are adjusted each year to ensure maximum alignment to Mission Directorate needs and technology priorities, and an annual allied organization summit affirms alignment with program goals. Licensees of NASA technology must agree in the license agreement to make a good faith effort to commercially apply the licensed NASA technology; otherwise, NASA's standard "March in Rights" clause enables NASA to revoke the license and license the subject fields of use to another entity. Under IPP's Innovation Incubator program element, partner firms will be required to agree that NASA personnel can take back and apply business principles learned at partner's facilities; dissenting entities would not be eligible to participate otherwise. Accordingly, partners, SBIR/STTR contractors, licensees, and teams participating in Centennial Challenges necessarily must work toward IPP's long-term goals. Otherwise, they are not acceptable as partners or SBIR/STTR contractors.

Evidence: ?? Progress is tracked in the SBIR/STTR electronic handbook, allowing broad access to consistent data and ensuring continued alignment with program goals and objectives. Access to the electronic handbook is available with registration and password. Public information is available at https://ehb8.gsfc.nasa.gov/contracts/public/background.do?action=BACKGROUND. ?? IPP will be preparing an annual Seed Fund report, with the initial report based on results of the 2006 effort. The report is forthcoming.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The National Research Council's (NRC's) Board on Science, Technology, and Economic Policy is in the fourth year of a six year study, mandated by Congress, of the SBIR/STTR programs of the five Federal agencies having the largest SBIR/STTR programs, including NASA. In accordance with its Congressional mandate, the scope of the effort includes conducting "a comprehensive study of how the SBIR program has stimulated technological innovation and used small business to meet Federal research and development needs," and also addresses areas such as "value to the Federal research agencies," "quality of research," "economic benefits," "non-economic benefits," "allocation of Federal research and development funds to small businesses," as well as several issues concerning the award process. In early 2006, NASA's Office of Program Analysis and Evaluation (PA&E) commissioned a Special Review Team (SRT) of experts, lead by a senior researcher at RAND Corporation, charged with conducting an independent evaluation of "the ability of IPP to meet NASA strategic goals . . . [to include] an in-depth review of the Agency's current commercialization and technology transfer activities to more effectively support the needs of NASA's Mission Directorates and to fulfill (NASA's) statutory responsibilities under the Space Act." In January 2007, the SRT reported its findings to NASA's Associate Administrator on its evaluation of the IPP program. In November 2004, the National Academy of Public Administration completed its report on NASA's technology transfer activities.

Evidence: ?? IPP provides a list of independent evaluations in NASA's annual Budget Estimates. See the NASA FY 2009 Budget Estimates (Cross-Agency Support, Agency Management and Support Theme, Innovative Partnerships Program) for the latest list: www.nasa.gov/pdf/210019main_NASA_FY09_Budget_Estimates.pdf. ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request) ?? Technology Transfer: Bringing Innovation to NASA and the Nation, National Academy of Public Administration, November 2004, available at http://napawash.org/Pubs/NASATechTransfer12-14-04.pdf ?? The Web site for the Board on Science, Technology, and Economic Policy is available at http://www7.nationalacademies.org/step/

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: IPP's PART and Government Performance and Results Act (GPRA) performance measures focus on the program's primary purposes: technology infusion, technology transfer, and the management of intellectual property. (Please note that IPP's PART and GPRA measures are designed to be similar and complimentary.) Each of IPP's budget line items serves these purposes and, therefore, IPP's performance measures. The FY 2008 and 2009 budget requests for IPP identify the three program elements of Technology Infusion, Innovation Incubator, and Partnership Development and the resources allocated across their program elements. The annual Performance Plans show how NASA's annual GPRA measurements relate to the long-term measures and Agency Strategic Goals.

Evidence: ?? NASA's FY 2008 and FY 2009 Budget Estimates available at www.nasa.gov/budget. For FY 2009, IPP can be found under Cross-Agency Support. IPP's performance measures can be found in the FY 2009 Performance Plan and FY 2008 Performance Plan Update, both located under Management and Performance.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: IPP's organizational structure, mission, definition, and implementation of program elements are strongly aligned with NASA's Strategic Plan and mandating legislation. IPP has developed ambitious long-term performance goals for PART, supported by a limited number of annual performance goals. These measures are consistent with the ambitious, well-targeted Government Performance and Results Act (GPRA) performance measures, which have been honed over the years through program experience and collaboration with NASA's Office of Program Analysis and Evaluation (PA&E) (or previous NASA policy entities) and the Office of Management and Budget. Furthermore, IPP management addresses strategic challenges at its quarterly meetings. A special review team (SRT) chartered by NASA PA&E provided an assessment of IPP, noting that doing both technology "spin-in" and "spin-out" in a single organization is reasonable, but subordinating "spin-out" to "spin-in" risks undermining political support for IPP and NASA and invites earmarks for special projects and contractors. IPP's strategic approach is to ensure that "spin-in" and "spin-out" technologies complement each other so that opportunities in one of these areas leads to opportunities in the other area. The SRT also recommended that IPP focus more on developing technologies to higher technology readiness levels (TRLs). In response, IPP has implemented an SBIR/STTR program and Seed Fund projects that track TRLs and develop higher-TRL technologies. Other recommendations for improvement focused on the organization and management processes, which IPP has addressed. The SRT stated that the major strength of IPP is that it "will centralize NASA's "partnering" programs at Headquarters (HQ) in an office external to all of the Mission Directorates, thereby allowing activities with potential synergies and a highly visible public presence to be better coordinated and to serve NASA across all Missions Directorates," and notes that IPP's programs have survived despite frequent reorganizations by Agency leadership, confirming that IPP's strategic purpose is on target.

Evidence: ?? The NASA 2006 Strategic Plan (www.nasa.gov/pdf/142302main_2006_NASA_Strategic_Plan.pdf) ?? NASA's FY 2009 Performance Plan is available in the FY 2009 Budget Estimates (www.nasa.gov/pdf/210019main_NASA_FY09_Budget_Estimates.pdf). ?? Presentations and agendas from IPP quarterlies are available upon request. ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request)

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: With the assistance of the Mission Directorates and the Office of Program Analysis and Evaluation (PA&E), IPP analyzes all potential partnerships and contracts for relevancy to NASA mission needs and potential for transfer to the public. IPP frequently reviews cost, schedule, and performance goals of program elements as part of the office's standard management practices. For the largest IPP budget line item, SBIR/STTR, an analysis of trade-offs with other programs within or outside of NASA would not be relevant since the program is legislatively required and could not be replaced with an alternative.

Evidence: ?? The IPP contracting officer's technical representatives (COTRs) are responsible for the scope, cost, and schedule of partnerships and contracts. IPP collects partnership and licensing agreement data from the IPP Center offices in a Quarterly Report Matrix. Copies of related reports are available from Headquarters IPP upon request. ?? Mission Directorate technologists at the Headquarters and Center level participate in partnership and contract discussions as needed and at scheduled review meetings. ?? IPP Chief Technologist's technology landscape documentation regarding Mission Directorate technology needs, and particular technology gaps. ?? IPP's SBIR/STTR personnel are part of all IPP telecons, WebEXs, and quarterly meetings in the same way as partnership development personnel. Transcripts/meeting minutes of these meetings are available from Headquarters IPP upon request.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Many agencies have SBIR/STTR programs, and NASA actively works with many other agencies to understand and compare benefits and share best practices. This is done during semi-annual national meetings where all agencies come together to discuss and compare their SBIR/STTR activities, and is also done during monthly meetings with other agencies and the Small Business Administration. NASA also participates with other agencies and shares/learns best practices through such activities as the Navy Opportunity Forum, which is focused on infusing technology developed from SBIR investments. In addition, independent assessments are made to compare programs across agencies, such as the National Research Council (NRC) study on the SBIR program that is currently underway. Many other agencies have inquired about NASA's innovative programs such as Centennial Challenges prize competitions, as they are interested in pursuing similar programs in their agency. With regard to technology transfer, NASA's participation in the Federal Laboratory Consortium (FLC) as a mechanism to assess and compare the benefits of NASA's technology transfer efforts with those of other agencies. The FLC is an organization of all Federal agencies with research laboratories, focused on technology transfer. They have regular regional and annual national meetings where agencies get together and compare programs and share best practices and success stories. Many agencies (both U.S. and foreign) use NASA's technology transfer activities, including our annual Spinoff publication, which documents some of our best technology transfer successes, as a benchmark.

Evidence: ?? A list of upcoming NASA and government-wide SBIR/STTR events is available at http://sbir.gsfc.nasa.gov/SBIR/SBIR.html ?? The Web site for the Board on Science, Technology, and Economic Policy, the NRC organization reviewing IPP's SBIR program, is available at http://www7.nationalacademies.org/step/ ?? List of FLC events in which IPP participated, available from Headquarters IPP.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: IPP scrutinizes budget requests from Center IPP offices and makes budget allocation decisions based on detailed, internal analysis. Budget requests are reviewed by NASA's Office of Program Analysis and Evaluation (PA&E) for consistency with Agency plans and available resources and then presented to the Agency's senior-level Operational Management Council and Strategic Management Council for approval and/or alterations as part of NASA's strategic management process. Once priorities and budget request levels are established, IPP works with PA&E to incorporate IPP's request into NASA's annual Budget Estimates. The annual Budget Estimates provides a justification for IPP's budget request, describing how budget line items serve IPP's three major purposes: technology infusion, technology transfer, and intellectual property management. Although the largest budget line item, SBIR, is a legislatively required program that NASA cannot alter greatly, this program serves IPP's technology infusion and transfer purposes. Working with the Mission Directorates, IPP identifies mission technology needs and authors the requirements for SBIR contracts based on these technology needs. For the Seed Fund, IPP incorporates NASA's technology needs in the call for solicitation sent to the Centers. IPP analyzes SBIR contract and Seed Fund proposals and chooses those that best serve NASA's technology needs.

Evidence: ?? NASA's FY 2009 Budget Estimates (www.nasa.gov/pdf/210019main_NASA_FY09_Budget_Estimates.pdf). ?? Minutes of the Operational Management Council and Strategic Management Council (internal NASA only; available to OMB on request) ?? IPP's annual SBIR solicitations are available at http://sbir.nasa.gov/SBIR/solicit.htm. ?? IPP issues Seed Fund solicitations via the Federal Business Opportunities site at www.fbo.gov.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: IPP collects partnership and licensing agreement data from the IPP Center offices in a Quarterly Report Matrix. IPP systematically collects statistics by Center on the number of patents issued, the number of patent applications filed, the number of patents and patent applications with licenses, the number of New Technology Reports (NTRs) received, and the number of NTRs assessed for which there is a patent application. These NTRs are the key measure of program success because they document NASA's meeting its statutory obligation to protect the government's rights in its inventions as well as facilitating the transfer out of NASA technology for commercial application and other public benefit . One of the methods IPP is using to measure performance of technology development activities is the comparison of planned technology maturation (e.g., increasing the technology readiness level (TRL) of a technology) to the achieved TRL. IPP currently is doing this for Seed Fund, and we are revising the SBIR/STTR electronic handbook to accommodate this new requirement, as well. Beginning in FY 2008, IPP is funding an SBIR/STTR Technology Infusion Manager (STIM) full time equivalent (FTE) employee at each of NASA's Centers. The STIM facilitates infusion of SBIR/STTR developed technology into NASA missions wherever possible and reports infusion successes. IPP recently established a Tiger Team to provide recommendations for strengthening the ties between information collected and improved program management. Also, the IPP Offices prepare Fact Sheets, Success Story write-ups, and Weekly Reports regarding selected partnerships and license agreements, as well as related activities, that provide qualitative measures of program results and focus. IPP utilizes this information to evaluate productivity of each of its Center implementing offices as well as to inform its stakeholders of program performance and contributions to the Agency and the Nation.

Evidence: ?? IPP quarterly report data is available upon request. ?? Weekly reports, which show steady performance for achievements of the program, are available upon request. ?? Tiger Team recommendations to date are available upon request. ?? Fact Sheets and Success Stories are available at IPP's website, http://www.ip.nasa.gov/. ?? NTRs, many of which are summarized as articles in Tech Briefs Magazine, available at http://www.techbriefs.com/.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: IPP Center chiefs are held accountable for cost, schedule, and performance of their programs, and they do individual stand-up presentations to Headquarters IPP management on the full range of their program results twice a year at IPP Quarterly Meetings. IPP Center offices are required to submit Quarterly Report Matrices, which require detail on partnership and licensing activity. Quarterly Report Matrices document the status of candidate partnerships and licenses through all phases, from early discussions through performance completion or an agreement being terminated. IPP Central Contractor reports are due once a month, and IPP contracting officers (COs) meet with contractors about every six weeks. The CO also exercises strong oversight. Contractor Center Task Plans are scrutinized and edited by the contracting officer's technical representative (COTR) for scope, estimated cost, and schedule. COTR and CO scrutinize Central Contractor monthly invoices. Center COTRs also monitor SBIR/STTR contracts regarding cost, performance, and schedule just as are other NASA contracts, and the COTR is required to approve SBIR/STTR contract invoices prior to payment. The Headquarters IPP office conducts a formal review of Seed Fund projects (all of which are one year projects) at mid-term and at year-end. Mission Directorate technologists at both Headquarters and Center levels participate in these reviews in person or by videoconference. Projects are reviewed for performance, cost, schedule, plans for non-IPP funding for continued technology development, and for infusion of the technologies into NASA missions. The Headquarters IPP office has developed a standardized project-reporting template that serves as the framework for the reviews. IPP communicates expectations for agreements through solicitations and guidelines, such as the SBIR/STTR handbook. IPP works with COTRs at all Centers to encourage communications and oversight to ensure awardees meet performance expectations. There are baselines and standards against which awardees and IPP managers are held accountable, including schedule milestones and technology readiness improvement plans that are part of the negotiated agreement. If a partner/awardee cannot meet the terms of the agreement, IPP terminates the partnership. These standards apply to all partnerships, including contracts made through SBIR/STTR. Through IPP's Techtracs replacement management information system, which will become available in the near term, IPP will be able to identify and track under-performing partnerships, alerting IPP to potential risks if the related entity seeks a new partnership in the future. IPP assesses proposals on their individual merits, and IPP potentially could form a partnership or sign a contract with an entity even if IPP terminated a partnership/contract with that entity in the past if that entity demonstrates that they practice sound project management.

Evidence: ?? SBIR/STTR handbook, available at http://sbir.nasa.gov. ?? IPP's annual SBIR solicitations are available at http://sbir.nasa.gov/SBIR/solicit.htm. Available from Headquarters IPP: ?? IPP Quarterly Report Matrices. ?? IPP Quarterly Meeting Agendas. ?? IPP Field Center Chiefs Quarterly Meeting Presentations. ?? IPP Bi-Weekly WebEX Agendas. ?? IPP Center budget requirements submissions. ?? IPP detailed budget allocation workup. ?? Central Contractor Monthly Reports. ?? Central Contractor Task Plans, by IPP Center office. ?? Seed Fund Mid-Term Review project presentations. ?? Seed Fund Final Review project presentations.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: IPP seeks to accelerate obligations such that all funds can be obligated in the year of appropriation, consistent with recent guidance from NASA's Office of the Chief Financial Officer (OCFO). However, SBIR/STTR contracts are a challenge to this. The largest portion of the IPP budget is funding for SBIR/STTR contracts, which are not steady state throughout the year, but occur in peaks based on the Solicitation schedule. The Solicitation schedule is not fully at NASA's discretion, but is coordinated with all other SBIR agencies through the Small Business Administration such that all solicitations for federal SBIR programs are distributed throughout the year and not all occurring at once. NASA's annual SBIR/STTR Solicitation is in July, with awards typically in the November time frame, using funds from the prior fiscal year for many of the awards, due to the uncertainty of getting timely appropriations and funds distribution under a Continuing Resolution each year. IPP is working with OCFO to develop a strategy for timelier obligation of SBIR/STTR funds. In recent years, NASA consolidated financial responsibilities at the NASA Shared Services Center as part of an effort to improve the Agency's financial management. During the initial years of operation the obligation of funding has lagged. However, timing has and will continue to improve as the Center achieves steady state operations. Headquarters IPP budget allocations to Center IPP offices are determined by detailed analysis and held fixed. Phasing Plans regarding anticipated spending profiles are required of each IPP Center office. Headquarters IPP monitors actual spending (obligations and commitments) against planned spending. It is not possible for Center offices to approach spending beyond their budgets without Headquarters being aware beforehand and without serious consequences. IPP Central Contractor (a Headquarters fixed-price contract) allocations to each of IPP's Center offices are determined and fixed by Headquarters. Central Contractor financial reports are submitted to the COTR once a month. It is not possible for Center allocation limits of Central Contract to be exceeded given the fixed price nature of contractor tasks. Also, given the level of scrutiny applied by Headquarters, it is extremely unlikely that Headquarters allocations to Centers would exceed total Central Contract funding limits. IPP is free from improper payments, as reported by NASA's Office of Inspector General.

Evidence: ?? NASA Agency-wide accounting system, using SAP, ensures funds are spent for the intended purpose and adequately reported. ?? The IPP financial manager holds regular telecons with Centers to discuss financial status and resolve any issues. ?? Central Contractor Monthly Reports, available from Headquarters IPP. ?? IPP financial performance reports ?? Please see the FY 2007 Agency Financial Report Addendum (available at www.nasa.gov/budget) for the Office of Inspector General's report on improper payments at NASA. Past year's reports are available in the annual Performance and Accountability Reports, also available at www.nasa.gov/budget.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: IPP's efficiency measures focus on the program's effectiveness in achieving its main purposes. All of IPP's contracts have been competed prior to award. Selections are made based on a proposal meeting IPP's technical requirements with high-quality products and services in the most cost-effective way practical. IPP's SBIR/STTR program element is administrated utilizing a Web-based system, the Electronic Handbook, through which proposals are received, distributed to technical and business evaluators, evaluations compiled, evaluation results sorted, and demographic information on compiled. The NRC noted in its three year mid-term report, part of an exhaustive six year review of the five largest Federal agency SBIR/STTR programs, that NASA has pioneered development of an innovative electronic interface designed to help both applicants navigate the complex process of applying for awards, and managers to track and assess program activities. Headquarters and Center IPP offices use NASA's Space Act Agreement Maker (SAAM) to construct Space Act agreements using standardized language to the maximum extent practical to achieve efficiencies in negotiating and concurrence routing of partnership and license agreements. IPP currently uses TechTracS to track contacts with potential partners and licensees, partnership and license development, partnership and intellectual property statistics, new technology reporting, and compiling success story information. However, the fundamental technology underlying TechTracS is outdated in several ways (e.g., speed, accessibility, IT security, user friendliness, architecture, and cost effectiveness). IPP is replacing TechTracS with a system that is more cost effective, standardized across NASA, user-friendly, and has greater functionality. IPP anticipates long-term savings of up to 50% annually, based on a comparison of IPP's historical $2.1 million-per-year budget to support TechTracS versus current price estimates for a replacement system of greater capability. With the budget and workforce constraints placed on IPP, such IT improvements are critical to continued program efficiency. Furthermore, IPP has procedures in place to ensure overall program efficiency and cost-effectiveness. Headquarters IPP budget allocations to Center IPP offices are determined by detailed analysis. Phasing Plans regarding anticipated spending profiles are required of each IPP Center office. Headquarters IPP monitors actual spending (obligations and commitments) against planned spending. It is not possible for Center offices to approach spending beyond their budgets without Headquarters being aware beforehand and without serious consequences. IPP Central Contractor allocations to Center offices are determined by Headquarters. Central Contactor financial reports are submitted to the contracting officer's technical representative (COTR) once a month. It is not possible for Center allocation limits of Central Contract to be exceeded given the fixed price nature of contractor tasks. Also, given the level of scrutiny applied by Headquarters, it is extremely unlikely that Headquarters allocations to the Centers would exceed total Central Contract funding limits. Recent restructuring of the SBIR/STTR program yielded efficiencies and improved communications with Mission Directorates to advance program objectives of technology infusion. Savings of about $2 million per year in the steady state (resulting from approximately 10 lesser civil servant or contractor full-time equivalents (FTEs) to implement the program at $200 thousand/FTE, including fringe benefits) enabled initiation of IPP's FAST project, which will further advance technology and support NASA's fifth strategic goal for pursuing appropriate partnerships with the emerging commercial space sector. For FY 2009, NASA has added an efficiency measure to IPP's GPRA measures.

Evidence: ?? SBIR/STTR Solicitation instructions (at http://sbir.nasa.gov) for submission of proposals utilizing the Electronic Handbook. ?? The NASA TechFinder portal (at http://technology.nasa.gov/) is part of TechTracS. Portions of the site are accessible to outside users. Available from the Headquarters IPP office: ?? Seed Fund Mid-Term Review project presentations. ?? Seed Fund Final Review project presentations. ?? Agency Annual Report on Federal Laboratory Technology Transfer in response to Technology Commercialization Act of 2000 (P.L. 106-404), which is sent to OMB and details NASA's Intellectual Property statistics. This report provides evidence of NASA's IP management efficiencies. ?? IPP Center Chiefs Quarterly Meeting Presentations. ?? Central Contractor Monthly Reports to HQ IPP. ?? Partnership and other program statistics and qualitative information extracted from TechTracS. ?? SAAM users manual (available online for SAAM users internal to NASA)

Does the program collaborate and coordinate effectively with related programs?

Explanation: IPP's purpose requires the program to collaborate and coordinate effectively with other programs throughout NASA, other Federal agencies, and the private sector. The IPP Director has quarterly meetings with Mission Directorate Associate Administrators to review IPP contributions to Mission Directorate needs and with the Assistant Administrators for relevant Mission Support Offices that IPP partners with or relies on for implementing the program. The IPP Director also meets on a regular basis with the NASA Associate Administrator, and provides weekly status reports to NASA management in the Administrator's office and other Mission Support Offices. Furthermore, IPP has offices at each Center to ensure that the program addresses the unique needs of each Center. IPP's SBIR/STTR program executive coordinates on a periodic basis with leaders of other federal SBIR/STTR programs and the SBIR/STTR leadership of the Small Business Administration (SBA) to share best practices and lessons learned, as well as to clarify SBA policy to insure accurate interpretation of Congressional intent. IPP is an active member of the Federal Laboratory Consortium. IPP staffers often chair panels and give presentations at various Federal Laboratory Consortium (FLC) functions in the interest of exchanging information on best practices and lessons-learned, and educating various business sectors on opportunities to partner with Federal agencies. IPP holds "Industry Days" at NASA Centers, which are attended by industry, technology transfer representatives from state and other Federal agencies with similar and/or complementary technology needs, and technology transfer intermediaries. IPP Headquarters and Center staffers often give presentations and chair panels at various trade association and professional association functions, which include technology transfer representatives from state and other Federal agencies. Such events provide robust opportunity for NASA and other technology transfer personnel to exchange ideas and other information regarding future business opportunities.

Evidence: ?? Upcoming IPP-related and -sponsored events are available on the IPP website at http://www.ip.nasa.gov/events.htm ?? A list of IPP Center offices (and associated websites) is available on the IPP website at http://www.ip.nasa.gov/field_centers.htm Available from the Headquarters IPP office: ?? Presentations given by IPP staffers at various business and professional association functions. ?? List of outside functions IPP attended and presented/chaired panels, by NASA Center. ?? List of NASA Industry Days, by NASA Center. ?? List of FLC events in which IPP participated, available from Headquarters IPP. ?? Weekly reports of successes and achievements from across the program. ?? Positive feedback from Mission Directorate Associate Administrators to the IPP Director that IPP is doing good work and contributing to the needs and success of the Mission Directorates.

Does the program use strong financial management practices?

Explanation: NASA's Inspector General has identified the Agency's financial systems as a material weakness. However, the Office of the Chief Financial Officer has taken significant steps to correct financial systems and process deficiencies to make them fully compliant with federal financial regulations, as documented in NASA's FY 2007 Agency Financial Report. IPP has established sound financial management processes. Headquarters IPP requires IPP Center offices to submit annual budget requirements in significant detail, along with justification for those requirements. Headquarters IPP then scrutinizes and discusses with IPP Center officers their budget requirements, and allocates budget based on informed decision-making. The Headquarters IPP financial manager conducts a bi-weekly telecon with IPP Center budget resource managers to discuss budget execution, discuss issues, clarify Headquarters requests for information on funds utilization, and interpret decisions and requirements of NASA's Office of Chief Financial Officer. The Headquarters IPP financial manager participates regularly at IPP's bi-weekly WebEXs, weekly telecons, and Quarterly Meetings to provide status and clarify actions. Headquarters IPP budget allocations to Center IPP offices are determined by detailed analysis and held fixed. Phasing Plans regarding anticipated spending profiles are required of each IPP Center office. Headquarters IPP monitors actual spending (obligations and commitments) against planned spending. It is not possible for Center offices to approach spending beyond their budgets without Headquarters being aware beforehand, and without serious consequences. IPP Central Contractor (a Headquarters fixed-price contract) allocations to each of IPP's Center offices are determined and fixed by Headquarters; Central Contractor financial reports are submitted to the contracting officer's technical representative (COTR) once a month; the COTR scrutinizes financial reports. It is not possible for Center allocation limits of the Central Contract to be exceeded given the fixed-price nature of contractor tasks. Also, given the level of scrutiny applied by Headquarters, it is extremely unlikely that Headquarters allocations to Centers would exceed total Central Contract funding limits. Headquarters IPP uses NASA's accounting system for distributing and tracking Headquarters IPP budget allocation to the Centers, obligations, and commitments. Headquarters IPP is held responsible for a section in the Agency's Operating Plan. Contractor invoices are scrutinized and validated by COTRs before being paid.

Evidence: ?? Annual IPP Center budget requirements submissions. ?? NASA's FY 2009 Budget Estimates (see Cross-Agency Support, Agency Management and Support Theme, Innovative Partnerships Program), which provides significant detail on IPP's budget requirements. This budget and previous budgets are available at www.nasa.gov/budget/. ?? IPP section of NASA Operating Plan. ?? NASA's Corrective Action Plan for Material Weaknesses is available in the Performance and Accountability Report FY 2007 Agency Financial Report; the full report of the auditors is in the FY 2007 Agency Financial Report Addendum. Both reports are available at www.nasa.gov/budget/.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: IPP's management process to identify and address quickly any management deficiencies. The IPP Director has quarterly meetings with Mission Directorate Associate Administrators to review IPP contributions to Mission Directorate needs and with the Assistant Administrators for relevant Mission Support Offices that IPP partners with or relies on for implementing the program. The IPP Director also meets on a regular basis with the NASA Associate Administrator, and provides weekly status reports to NASA management in the Administrator's office and other Mission Support Offices. IPP Center chiefs do individual stand-up presentations to Headquarters IPP management on the full range of their program results twice a year at IPP Quarterly Meetings. IPP Center offices are required to submit Quarterly Report Matrices, which detail partnership and licensing activity. IPP Central Contractor reports are due once a month, and IPP contracting officers (COs) meet with contractors about every six weeks. Actions have been taken at the Agency level and by IPP to address some of the recommendations from independent review panels. For example, the NASA PA&E-chartered Special Review Team (SRT) recommended that IPP have a highly visible public presence, with stable program leadership that receives clear and consistent direction for the Administrator. As a result, NASA made IPP an independent office that reports directly to the Administrator. IPP reorganized so that Headquarters retains overall management of the IPP portfolio, with Center IPP staff reporting to Headquarters managers. The SRT also recommended that SBIR/STTR topic/subtopic technologies be better coordinated with other Agency efforts to determine technology requirements. IPP intimately involves the four Mission Directorates in developing topics/subtopic for NASA's annual SBIR/STTR Solicitation, as well as in evaluating SBIR/STTR proposals. The SRT recommended IPP play a much stronger role in managing, or at least influencing, all programs having technology transfer significance. In response, IPP meets regularly with the heads of other NASA organizations having some involvement in partnering for the purpose of exchanging ideas for working together, best practices, and joint project implementation. To improve tracking and efficiencies, the SRT recommended that databases like TechTracS need clear objectives and better inputs. IPP is in the process of re-defining requirements for its TechTracS-related needs. IPP is also in the process of replacing TechTracS with a more effective successor system. Through these activities, IPP has taken meaningful steps to correct all major management deficiencies identified by independent evaluators and address recommendations for improvement.

Evidence: Available from Headquarters IPP: ?? Briefings on SBIR/STTR restructuring. ?? Center organizational charts show that SBIR/STTR personnel report to Field Center IPP Chief. Also, SBIR/STTR personnel are part of all IPP telecons, WebEXs, and quarterly meetings in the same way as partnership development personnel. ?? SBIR/STTR network organization chart, which shows strong integration of SBIR/STTR program element into IPP. ?? IPP top management meets quarterly with other NASA program officials having some partnership activities residing within their own offices for the purposes of coordinating efforts and identifying opportunities for new partnership initiatives. ?? IPP has taken significant steps to transition from TechTracS to a more effective and efficient system; also, program management of that function is currently being transitioned from the Langley Research Center to the Ames Research Center. ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request)

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: Deliverables are clearly defined by IPP Headquarters management, as determined in cooperation with the Mission Directorate(s) that will use the technology deliverables. IPP Center chiefs are held accountable for cost, schedule, and performance of their programs. If a partner cannot meet the terms of their agreement or contract (in the case of SBIR/STTR), which covers expectations for cost, schedule, and performance, IPP terminates the partnership or contract. Center contracting officer's technical representatives (COTRs) monitor SBIR/STTR partnerships and contracts for compliance with cost, performance, and schedule, and the COTR is required to approve SBIR/STTR contract invoices prior to payment. The Headquarters IPP office conducts a formal review of Seed Fund projects (all of which are one year projects) at mid-term and at year-end. Mission Directorate technologists at both Headquarters and Center levels participate in these reviews in person or by videoconference. Projects are reviewed for performance, cost, schedule, plans for non-IPP funding for continued technology development, and for infusion of the technologies into NASA missions. IPP communicates expectations for agreements through solicitations and guidelines, such as the SBIR/STTR and Seed Fund solicitations. The prize competitions conducted by Centennial Challenges have clearly defined targets and IPP makes awards only if competitors achieve the targets during a competition. In the area of intellectual property management, IPP conducts analyses, including market research, and prioritizes technologies for which NASA should obtain patents based on their likely benefit to NASA, other government agencies, and the public.

Evidence: ?? Central Contractor Task Plans, by IPP Center office. ?? Seed Fund Mid-Term and Final Review project presentations. ?? IPP's annual SBIR solicitations are available at http://sbir.nasa.gov/SBIR/solicit.htm. ?? SBIR/STTR handbook, available at http://sbir.nasa.gov. ?? IPP's annual SBIR solicitations are available at http://sbir.nasa.gov/SBIR/solicit.htm. ?? IPP issues Seed Fund solicitations via the Federal Business Opportunities site at www.fbo.gov. ?? IPP Center Chiefs Quarterly Meeting Presentations. ?? Centennial Challenges Web site (http://centennialchallenges.nasa.gov/cc_challenges.htm)

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: IPP's management processes include frequent contact with the Center IPP offices at formal bi-weekly WebEXs and quarterly meetings, as well as quarterly and weekly reports according to prescribed format. Through the SBIR/STTR programs NASA communicates its technology needs via announcements. If an entity meets the SBIR requirements, they may negotiate with IPP a Phase I contract that defines the cost, schedule, and deliverable to meet NASA technology needs, Proposals are assessed by IPP and the Mission Directorate customers who would use resulting technologies. The partnership is managed against this contract, and IPP will terminate the partnership if the partner does not meet the agreed-upon targets and milestones. IPP Center officers are responsible for tracking and managing contracts made through their Centers, and they report contract status to Headquarters IPP management via the process of weekly and quarterly reports and bi-weekly and quarterly meetings. Subsequent agreements, such as Phase II contracts, are made based upon performance to date, the continued needs of NASA and the partner, and the likely benefits of resulting technology to NASA and the external community. Similarly, the Seed Fund issues open announcements that communicate NASA's technology needs. Interested parties must include one or more non-NASA partners who are willing to provide cost sharing at a level equal to or greater than the IPP funding provided to the project. The project is managed against the agreed-upon terms of the partnership. The Facilitated Access to the Space Environment for Technology Development and Training (FAST) program provides partners opportunities to flight test technologies. These opportunities to partners with SBIR/STTR contracts, Seed Fund companies, and university and research institutions; the management and fund allocation processes are similar to those for SBIR/STTR and Seed Fund. For the Centennial Challenges, IPP awards prize purses only if a competitor meets the target of a competition during a specific competition.

Evidence: ?? SBIR/STTR handbook, available at http://sbir.nasa.gov. ?? IPP issues Seed Fund solicitations via the Federal Business Opportunities site at www.fbo.gov. ?? IPP FAST Web site at http://www.ipp.nasa.gov/ii_fast.htm.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Although IPP's PART performance measures were established during the program's first assessment in 2008, the PART measures were designed to be similar or complementary to IPP's existing GPRA performance measures. Furthermore, IPP's PART performance measures encompass key activities that IPP already tracked as part of the program's management. This allows IPP to demonstrate a trend of adequate progress towards meeting long-term performance goals. Throughout IPP's existence (including entities that predate IPP as a separate office) the program consistently has achieved its long-term GPRA performance goals (known as Outcomes), as evidenced in NASA's annual Performance and Accountability Reports. By meeting these performance goals IPP shows continuous success, despite budget and workforce constraints, in attracting partners who develop technologies that can be infused into NASA missions and transferred out for other government and commercial use.

Evidence: ?? NASA's Performance and Accountability Reports are available at www.nasa.gov/budget/. ?? Ambitious baselines and targets for long-term PART measures [ExpectMore.gov address to be added when measures become publicly available]. ?? For additional information on IPP successes: - Commercial applications and other public benefits of NASA licensed technologies and SBIR/STTR technologies over 40 years are documented in Spinoff Magazine, at http://www.sti.nasa.gov/tto/. - SBIR/STTR Success Stories are available at http://sbir.gsfc.nasa.gov/SBIR/success.htm. - Centennial Challenges winning technologies and near-winning technologies that have been demonstrated at the competitions. Visit http://centennialchallenges.nasa.gov/ for more information. - IPP Quarterly report matrices provide statistics on awards and licenses (available from Headquarters IPP). - FY 2006 Seed Fund project demonstrations.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Although IPP's PART performance measures were established during the program's first assessment in 2008, the PART measures were designed to be similar or complementary to IPP's existing GPRA performance measures. Furthermore, IPP's PART performance measures encompass key activities that IPP already tracked as part of the program's management. This allows IPP to demonstrate a trend of adequate progress towards meeting annual performance goals. Throughout IPP's existence (including past entities) the program consistently has achieved its annual GPRA performance goals, as evidenced in NASA's annual Performance and Accountability Reports. This continued success is due to a concerted effort between IPP and its internal and external partners.

Evidence: ?? NASA's Performance and Accountability Reports are available at www.nasa.gov/budget/. ?? Ambitious baselines and targets for annual PART measures [ExpectMore.gov address to be added when measures become publicly available]. ?? NASA's PAR documents from the years 2004 through 2007 show IPP consistently set and met their targets for establishing partnerships. Other targets set and met during this time include targets for technology transfer agreements, and award of SBIR Phase III contracts or venture capital funds.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: IPP's efficiency measures focus on the program's success in infusing technologies into NASA missions and transferring technologies outside of NASA (while also demonstrating IPP's ability to manage NASA's intellectual property). NASA obtains patents, based on recommendations from IPP, for technologies that promise the greatest benefits to NASA, other government agencies, and the public. Although IPP cannot control how technologies produced through its partnerships will be used, IPP's choices for partnerships--and the investment of funds and workforce hours associated with those partnerships--are validated when a resulting patented NASA technology becomes licensed for use either within or outside of NASA. Another efficiency measure determines the percentage of SBIR/STTR Phase II contracts that yield technologies that are infused in NASA programs and projects. This measure also validates IPP's contract selections, showing efficiency in benefits yielded as a function of contracts made. Although these efficiency measures are newly established, it is based on information that IPP collected as part of standard management practices. As a result, IPP has established a credible baseline and demonstrated efficiencies in this area prior to the existence of the measure. Furthermore, IPP uses the Web-based SAAM to construct Space Act Agreements, utilizing standardized language to the maximum practical extent and coordinating concurrence of Space Act Agreements within the Agency. SAAM has introduced efficiencies and consistency in developing partnership agreements, and through its continued development, IPP has improved the tool over time. Recent restructuring of the SBIR/STTR program yielded efficiencies and improved communications with Mission Directorates to advance program objectives of technology infusion. Savings of about $2 million per year enabled initiation of IPP's FAST project, which will further advance technology and support NASA's fifth strategic goal for pursuing appropriate partnerships with the emerging commercial space sector. IPP established the role of the SBIR/STTR Technology Infusion Manager (STIM) at each of NASA's Centers. STIMs have the sole responsibility to facilitate infusion of SBIR/STTR developed technology into NASA missions at the Center level. Through this, IPP anticipates greater efficiencies and NASA utilization of SBIR/STTR technologies. IPP's recent addition of a Chief Technologist for SBIR/STTR should contribute significantly to infusion of SBIR/STTR technology. IPP also utilizes the Electronic Handbook (EHB) to administrate its SBIR/STTR program. EHB has introduced unprecedented efficiencies into NASA's implementation of its SBIR/STTR program that is second to none among other Federal agencies. IPP has consolidated its partnership development contractor support into a Central Contract, which has a maximum contract value of $6 million per year. Previously, IPP and its predecessor organizations had relied on a collection of partnership development contracts having contract value of approximately $9 million per year, not counting the National Technology Transfer Center (NTTC), which was previously funded at $7.8 million per year. IPP now uses the NTTC cooperative agreement in a more cost-effective way. Accordingly, planned funding for the NTTC for FY09 and out-years is at most $1 million per year; IPP is providing no FY08 funding to the NTTC, given still unexhausted carry-forward by the NTTC of FY06 IPP funding. Similarly, IPP engagement with the programs and projects at the Center level has been strengthened significantly, as has been IPP's engagement of the Mission Directorates at the Headquarters level. These interactions are frequent and systematic and will lead to greater targeting of IPP's technology development partnership efforts toward Mission Directorate technology gaps that the Mission Directorates are unable to address by funding procurements.

Evidence: ?? PART efficiency measures [ExpectMore.gov address to be added when measures become publicly available]. ?? SAAM users manual. ?? SAAM users workshops. ?? Electronic Handbook guidance in NASA's SBIR/STTR Solicitation, and other guidance available at www.sbir.nasa.gov. ?? Documented current actions to replace TechTracS. ?? Headquarters IPP internal budget documents. ?? SBIR/STTR program consolidation documentation. ?? IPP Director briefings to Mission Directorate heads. ?? IPP Chief Technologist's technology landscape documentation regarding Mission Directorate technology needs, and particular technology gaps. ?? The $2 million savings in the steady state resulting from the consolidation of NASA's SBIR/STTR program results from a decrease in approximately 10 civil servant or local contractor Full Time Equivalents (FTEs) required to implement the program at approximately $200 thousand/FTE per year, including fringe benefits. ?? See the NASA 2006 Strategic Plan at www.nasa.gov/budget for IPP's GPRA strategic goals.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The National Research Council study being conducted by the Board on Science, Technology, and Economic Policy observes that only the Department of Defense and NASA assess SBIR/STTR contribution to Agency mission primarily by the extent to which the agency itself uses outputs from its SBIR/STTR program. IPP considers this type of assessment critical as a measure of IPP's success, since SBIR/STTR is IPP's largest program element and any program within IPP is successful only if it contributes useful outputs to NASA and the larger user community. IPP is unique in the way it holds partners accountable for performance, as indicated in PART results. For example, the Department of Defense's Small Business Innovation/Research Technology Transfer program received Results Not Demonstrated for their last assessment. In the assessment the program stated that it does not regularly collect performance information from partners and the program may consider a business for future partnerships even if the current partnership has not yielded outputs. IPP holds partners and IPP Center staff accountable for performance and agreements are dissolved if a partner does not meet agreed-upon schedule, cost, and deliverables. In program purpose, design, and performance management, IPP more resembles the National Science Foundation's Capability Enhancement of Researchers, Institutions, and Small Businesses program, which received an Effective for their 2006 PART assessment. IPP already achieves a sharing of best technology transfer practices with other government agencies through IPP's involvement in the Federal Laboratory Consortium (FLC). Based on IPP's FLC involvement, IPP possibly is one of the strongest SBIR/STTR programs among Federal agencies, particularly regarding the emphasis on technology infusion. However, IPP has not rigorously benchmarked outputs of the technology transfer programs of other agencies. A comprehensive and objective comparison would most likely have merit--at a cost that may be prohibitive--for learning about best practices. Informally, through attendance at conferences and seminars that include other agencies and private sector organizations, IPP regularly discusses results and issues that indicate NASA compares favorably with other organizations. Through these informal contacts it is clear that other programs consider NASA's IPP as a leader in effectiveness, and IPP often gets requests to talk about the program's activities to outside organizations in the public and private sectors, both domestic and international.

Evidence: ?? The Web site for the Board on Science, Technology, and Economic Policy is available at http://www7.nationalacademies.org/step/ ?? Unpublished and unofficial interim observations and findings of National Research Council study team, specific to NASA's SBIR/STTR program. ?? IPP regularly gets requests for briefings from other U.S. government agencies (e.g., the National Institutes of Health, the Department of Homeland Security, and the Centers for Disease Control), and foreign space agencies (e.g., the European Space Agency and the Canadian Space Agency) to talk about IPP programs and successes. Copies of papers and talks are available through IPP. ?? Department of Defense Small Business Innovation/Research Technology Transfer program PART assessment at http://www.whitehouse.gov/omb/expectmore/detail/10001027.2003.html; National Science Foundation's Capability Enhancement of Researchers, Institutions, and Small Businesses program at http://www.whitehouse.gov/omb/expectmore/detail/10004403.2006.html.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The National Research Council's (NRC's) Board on Science, Technology, and Economic Policy is conducting a study of NASA's SBIR/STTR program., Preliminary findings observe that the primary metric for IPP project success is the deployment of SBIR-funded technologies on space missions, where the Agency can point to a number of significant impacts. The study also notes that NASA assesses contribution to Agency mission primarily by the extent to which the NASA uses outputs from its SBIR/STTR program. The NRC also observes that NASA has pioneered development of an innovative electronic interface designed to help both applicants navigate the complex process of applying for awards, and managers to track and assess program activities. A special review team (SRT), commissioned in 2006 by NASA's Office of Program Analysis and Evaluation (PA&E) and lead by a senior researcher at the RAND Corporation, concluded: "Combining NASA's 'partnering' programs in a top-level HQ office is a positive step. The ultimate success of IPP, however, depends largely on the level and type of support IPP receives from NASA leadership. While the activities encompassed by IPP will likely never be pivotal to the core missions of NASA, if the proposed re-organization reviewed herein is pursued as presented and the issues raised in this report are addressed, these activities can make a significant contribution to NASA's missions and its relationships with external constituencies." As referenced in this excerpt, the SRT study made recommendations to improve IPP's strategic focus and management and, thereby, improving the overall effectiveness of IPP activities. NASA adopted these recommended changes, which include making IPP an office that reports directly to the NASA Administrator, elevating the public face of the office and enabling cross-Agency communication, and reorganizing IPP to centralize, stabilize, and strengthen management of Center activities. Before these changes were enacted, the SRT study noted that "[t]he seeds of productive relationships with non-IPPO offices and entities already exist."

Evidence: ?? The Web site for the Board on Science, Technology, and Economic Policy is available at http://www7.nationalacademies.org/step/ ?? Unpublished and unofficial interim observations and findings of NRC study team, specific to NASA's SBIR/STTR program, were provided by the NRC to IPP. ?? The PA&E-commissioned Special Review Team Report on Innovative Partnerships Program, October 20, 2006, produced by the Independent Program Assessment Office (NASA internal document, available on request)

Were program goals achieved within budgeted costs and established schedules?

Explanation: The IPP office itself operates under tight budget and workforce constraints. As a result, IPP has enacted efficiencies and made program decisions that keep IPP consistently within budget. In FY 2007, IPP achieved all of its Government Performance and Results Act (GPRA) performance measures as follows: IPP facilitated the Agency's entering into over 200 Space Act Agreements with private and other external entities for development of dual use technology targeted to Mission Directorate technology needs; and IPP provided $9.2 million in funding for 38 Seed Fund partnerships for development of a broad spectrum of technologies addressing specific Mission Directorate technology gaps.; Partner and Center contributions of cash and in-kind resources will leverage these funds by a factor of four. IPP does not have control over technology usage and, therefore, cannot establish a schedule for infusion or transfer. A NASA-developed technology may be incorporated into commercial products within years or decades of "completion." Success in intellectual property management is the result of IPP evaluating the potential of different technologies and prioritizing the technologies for patents. IPP's management functions are performed consistently within cost. During FY 2007, IPP facilitated the signing of 35 license agreements and 598 Software Use Agreements. IPP also facilitated the reporting of more than 1200 new invention disclosures. As a result of IPP's efforts, over 100 NASA patent applications were filed and 93 patents awarded. Revenues realized from licenses of NASA sponsored technologies exceeded $4 million. In addition, IPP completed six Centennial Challenge events and awarded $450 thousand combined in prize money at two of them. The Centennial Challenges only awards prizes when a challenge's targets are met, so the program has remained within cost. IPP and its non-profit partners have held challenges during anticipated time frames. The schedule for prize awards are flexible, since IPP cannot predict at what point competitors will meet the targets, however, the two prizes mentioned above were awarded during the first few years of operations for the program, within the anticipated time frame for success.

Evidence: ?? IPP Quarterly Report Matrix of Center IPP partnership and licensing activity, ?? NASA's annual Performance and Accountability Reports (available at www.nasa.gov/budget) document IPP's continued success at achieving GPRA performance measures.