Detailed Information on the
Defense Advanced Technology Development Program Assessment

Establish measures that allow future program progress to be tracked quantitatively. Measures will highlight a focus on warfighter needs and Technology Readiness Levels of emerging technologies.

Develop metrics of the technical capabilities and performance of the Department's research and development organizations that select technical efforts for funding and that shape the projects to be undertaken.

Establish an independent review process for the program.

Measure: Percent of formal demonstration programs--Advanced Technology Demonstrations, Advanced Concept Technology Demonstrations, Joint Concept Technology Demonstrations and Future Mission Capabilities [ATDs, ACTDs, JCTDs, FMCs]--completed in the prior five years that the Joint Staff rate as having supplied a key capability in a fielded system.

Explanation:The purpose of this measure is to determine how effectively the Advanced Technology Development program meets the needs of the warfighter, which aligns directly with one of the purposes of the ATD program -- i.e., deliver innovative, product-ready technology to the warfighter. The COCOMs are the field commands that are the ultimate users of this program's products; their assessment of the worth of the outputs of the program will be key in determining program results. The assessments will be documented through a survey.

Measure: Percentage of overall funding for Advanced Technology Development (ATD) programs that addresses Combatant Commanders' (COCOM's) needs as documented in Integrated Priority List(s) (specific to each COCOM) by the Joint Staff.

Explanation:Each year, new ATD projects are started by the technologists who seek to meet the needs of the warfighter. The leaders of the warfighters--the COCOMs--document their needs in Integrated Priority Lists and the Joint Chiefs of Staff, as the organization above the COCOMs, develops their own Most Pressing Military Needs lists. This measure helps the program assess whether or not the ATDs provide sufficient technology options for the warfighter, as identified in their Integrated Priority Lists and Joint Staff Most Pressing Military Needs list. The overlap with those lists should be deliberate and the percentage of projects in which there is meaningful overlap should increase with time. A baseline for this metric will be developed and the metric will be in place by 2008.

Measure: Number of patent citations from supported Advanced Technology Development (ATD) projects as a rolling total from the past 3 years.

Explanation:The ATD program, when successful, produces new or improved technologies that can be incorporated in products that the warfighter can use to do work more effectively, more safely or more efficiently. If the program is healthy, it occasionally will produce usable technologies that are truly innovative and unique--and patentable. This measure will identify how frequent that standard is met and will give technologies an incentive to meet that standard and improve upon it with time. The baseline for this measurement will be established by the end of 2008.

Measure: Average technology readiness levels (TRL) of critical technologies generated by the Advanced Technology Development program for major defense hardware and software development programs ("Acquisition Category (ACAT) 1 programs").

Explanation:TRLs are measures of the maturity of technologies slated to be incorporated into new acquisition (research and development and procurement) programs. TRLs range from 1 (basic research needed, perhaps just a concept) to 9 (fully mature, fielded and operational). If the TRLs are too low, product development will be delayed unreasonably pending breakthroughs in the technology. However, many programs are "sold" on the basis of new, revolutionary capabilities that are expected to emerge from just-about-to-materialize technology breakthroughs. If oversold, product development will slow and even stop, leaving the warfighting customer without even a modest capability improvement. By measuring the TRLs of technologies in products in development, an assessment can be made about how likely the program will be to benefit the warfighter in a reasonable time horizon.

Is the program purpose clear?

Explanation: The mission of the Defense Science and Technology (S&T) Program is to ensure that the warfighters of today and tomorrow have superior and affordable technology to defeat any adversary on any battlefield. The Defense Advanced Technology Development (ATD) program, which is the most application-oriented part of the DoD S&T program, includes development of subsystems and components and efforts to integrate subsystems and component technologies into system prototypes for field experiments and/or tests in a simulated environment. The results of ATD projects are proof of technological feasibility and assessment of subsystem and component operability and producibility (rather than the development of hardware for service use, which is the product of later stages of product development). ATD projects demonstrate the general military utility or cost reduction potential of technology when applied to military equipment or techniques. Projects proposed for funding must have direct relevance to identified military needs.

Evidence: Director, Defense Research and Engineering S&T Strategic Plan (pending signature), May, 2007, Acquisition, Technology and Logistics Goal 3, Focused Technology to Meet Warfighting Needs, provides overarching goals and direction for the Advanced Technology Development (ATD) Program. The Defense S&T Strategic Plan establishes the purpose of the S&T program. The Department of Defense Financial Management Regulation defines ATD within the larger Science and Technology program.

Does the program address a specific and existing problem, interest, or need?

Explanation: The Advanced Technology Development program addresses the need for technologies to make forces lighter, more mobile and more lethal, as well as to have the ability to respond to a variety of crises, to provide information superiority, to prevent technological surprise by adversaries and to meet new and undefined threats. The program purpose is still relevant to current force needs.

Evidence: The high level needs of the program are set out in the Quadrennial Defense Report and at a finer-grained level by the Joint Warfighting Science & Technology Plan (JWSTP), which is prepared under direction of the Joint Chiefs of Staff.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Advanced Technology Development (ATD) program is designed to ensure that DoD has superior and affordable technologies to defeat potential adversaries in the future, which is a DoD-unique mission. The Defense Science Technology Advisory Group (DSTAG), which includes the leadership of the major stakeholder organizations in the DoD Science and Technology (S&T) Program, oversees and identifies Advanced Technology Development investment, including review of potential overlap with non-DoD investment. DoD maintains an inter-service planning process to avoid redundancy and duplication among the military services and agencies in research and development. In addition, the Base Realignment and Closure (BRAC) process shifts resources needed to meet many of the ATD goals, by eliminating much of the potential overlap of responsibilities among the military services at the facilities level.

Evidence: The responsibilities of the chief of the Defense Science and Technology program are set out in DoD Directive 5143.3 ("Director of Defense Research and Engineering (DDR&E)). The unique DoD mission is set out in DDR&E Strategic Goals Letter and BRAC authorities are set out in PL 101-510, part A of Title XXXIX, as amended.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Advanced Technology Development programs are managed by the military services and agencies, but much of the actual development activity supported by the program is executed through competitively awarded contracts to the private sector and various not-for-profit organizations. The Department's cross-service and -agency strategic planning process - called "Reliance" - guides technology through all phases of of the science and technology process. The Defense Science and Technology Advisory Group (DSTAG), a group of senior military and agency R&D chiefs, identifies investment issues, provides recommendations for improvement, and reviews overall program balance, adequacy of the approach, and progress to date. Other independent review groups, such as the Air Force Science Advisory Board or the Defense Science Board, have recently reviewed specific component (service and agency) programs and found no major issues.

Evidence: Major action items identified by the DSTAG are tracked and must be resolved by the services and agencies. Components are required to define and document their review processes. There is no strong evidence that another approach to planning, programming, and executing the program would improve the program. Air Force Scientifc Advisory Board review of the AF ATD program, [need a DSTAG reference and a AFSAB reference].

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The goals of the program are developed directly by military leaders and technical experts who work for the military services and agencies, through the Joint Warfighting Science and Technology Plan, and the Defense Technology Area Plan publication. Most of the funding for the actual execution of the programs is awarded through an open competitive process. This process starts with announcements of technology areas in which the services are interested. Merit review of proposals follows the receipt of proposals. Funds for awards are made available through the Department's overall Planning, Programming, Budgeting and Execution process, which sets priorities for funding and sets the overall funding for each technical area. There are no unintended subsidies.

Evidence: The objectives are set out in the DoD Joint Warfighting Science and Technology Plan and the Defense Technology Area Plan. Service and agency Broad Agency Announcements and Quick Reaction Fund proposals are available on the world wide web; see, for example, http://www.arl.army.mil/main/main/default.cfm?Action=6&Page=8 or http://www.onr.navy.mil/02/baa/. Information on the PPBE process can be found at http://akss.dau.mil/DAG/Guidebook/Common_InterimGuidebook.asp (see section 1.2).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Defense Research & Engineering Strategic Plan and the Joint Warfighting Science and Technology Plan (JWSTP) address Advanced Technology Development program investments and list important investment areas that military leaders believe are important for their long-term increased effectiveness. Within the Strategic Plan, long-term goals are established for the purpose of demonstrating new and improved, mature technologies in an operational environment with the goal of getting new, affordable technologies into the hands of the warfighter as quickly as possible. In addition, each component (service or agency) is developing a Strategic Plan. Those documents will guide general investment strategies. In addition, however, field commanders (the higher level commanders within the Combatant Commands) have specific or general capability needs that are as yet unmet. The performance measures that are being implemented assess how much useful overlap there is between the technologists' potential capabilities and the warfighters' projected needs. The primary metric is that of the percentage of demonstration programs in the past five years that have provided key capabilities to the field commanders.

Evidence: The Defense Research and Engineering Strategic Plan documents long-term performance measures. The measures are made possible in part by the specific actions set up by the Joint Warfighting Strategic Plan, which document specific bite-sized technical capabilities that military forces want in their future arsenal, as established by the Combatant Commanders' (chief warfighters) Integrated Priorities Lists. The long-term measure metrics incorporate these lower level metrics on an aggregated statistical basis.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program does not yet have specific numerical targets and timeframes, although the Department is working on establishing the baseline measures, and by the end of summer may have the needed specifics.

Evidence: The targets, when completed, will be based on the Defense Science and Technology Strategy and the Joint Warfighting Science and Technology Strategy (JWSTP), which incorporates the needed warfighter capabilities that might be addressed through science and technology programs, including Advanced Technology Demonstration programs.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Department has identified a number of annual measures that can be used to demonstrate progress. Those measures include the portion of ATD programs that directly address warfighters' needs, numbers of patents obtained by the scientists and engineers working to find innovative technological solution to warfighters' needs, and technology readiness levels of technologies being incorporated into systems under development. Progress is demonstrated by the increase in focus on warfighters' needs while emphasizing significant advancements in the technologies available to address warfighter needs.

Evidence: See separate Performance Measures section for the list of measures. The measures include the percentage of demonstration programs that have provided key capabilities to field commanders, numbers of patents obtained and technology readiness levels for technologies to be incorporated into development programs.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines are being established for relevance of the ATD program to the warfighter's needs, patents and technology readiness levels of technologies within systems under development. Targets have not been established yet. However, many of the components of those targets are documented in individual program descriptions as documented in Congressional Budget Justification materials. Military services and defense agencies monitor program performance to ensure efforts toward achieving program targets are progressing satisfactorily.

Evidence: Program Authorization Documents and Congressional Budget Justification materials document the technical goals of individual programs. However, statistical measures of success are being developed.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Although statistical annual goals have yet to be established, the partners (military services, defense agencies, program managers and contractors) have individual program goals for which they are accountable. These goals directly relate to the overall goals for the program, such as technological feasibility and assessment of subsystem and component operability and producibility and demonstration of the general military utility or cost reduction potential of technology when applied to military equipment or techniques. If too many of the goals are unmet on the projected timeline (either due to management difficulties or to sheer technical difficulty), the project is subject to cancellation. Oversight is provided by the owning service or agency, the Office of the Secretary of Defense and occasional outside independent reviews. All partners--managers, contractors, grantees and test range support staff--work to support the overall goals of the program and they report on their performance relative to their objectives on a regular basis.

Evidence: Individual program goals are set forth in program justification materials, in the Joint Warfighting Science and Technology Plan and the Defense Technology Area Plans. Memoranda of Agreement, contracts, grants and test plans establish individual partners' responsibilities.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There currently are no regular independent review teams. Technology Focus Teams, which are made up of Department technology program managers, conduct reviews of specific programs to assess program strengths and weaknesses; provide recommendations for improvement; provide technical, budgetary, and programmatic assessment independent of the program manager but not of the Department; and review overall program balance, adequacy of the approach used, and progress to date. These teams, however, would not be considered independent.

Evidence: The Technology Focus Teams are responsible for reviewing their entire technology area. Major action items are tracked and must be resolved as directed by the Defense Science and Technology Advisory Group (DSTAG). Components also have separate review boards of the military departments. Supporting documentation includes Technology Focus Team Charters [awaiting receipt].

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Annual and long-term goals for each individual ATD project are outlined in the budget justification books (Research and Development Descriptive Summaries) provided to Congress. The military services and agencies are held accountable for meeting the technical goals and funding requirements. Although some overall statistical long-term goals and annual measures of the overall program are missing, the annual and long-term performance goals of individual projects are tied to the budget in the desired manner. All direct and indirect costs are reported in the Future Years Defense Program and in budget justification materials specific to this program.

Evidence: Budget justification materials and additional supporting materials provide information on program costs. The Future Years Defense Program is the Department's overall budget projection document and all program costs are contained therein.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Director for Defense Research and Engineering has developed a Strategic Plan (pending signature) which is intended to clarify the Science and Technology program's goals and implementation. In addition, DoD's integrated S&T planning process has been in place for about 10 years and has been refined as improvements are identified. The Department is in the process of choosing aggregate measurable performance metrics and setting long-term and annual targets for the measurements. The Department used to emphasize hundreds of individual project objectives in its annual technology plan, but found that those objectives were so numerous that overall progress was difficult to assess. The Department, therefore, is in the process of taking a more strategic view of those objectives, through what it calls the Defense Technology Area Plans (DTAPs).

Evidence: The new framework for the DoD Science and Technology Program is contained in the Defense Research and Engineering Strategic Plan (pending signature) and the annual plans are highlighted in the DTAPs.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Program managers assess and compare the potential benefits of efforts within the program. This program, however, examines the very earliest stages of development programs, before programs have become formal system development programs. Thus, this program often deliberately examines multiple technological approaches to a given military need, inexpensively and usually to determine feasibility of a particular technical approach, before much larger funds are committed in the system development stages of the chosen alternative. Demonstrations within the ATD program provide information critical to the development of the Analysis of Alternatives that will inform decision-makers prior to the approval of later stage system development programs, thus helping overseers determine what the most effective course is to meet a need. Differences between the management of the budding program and the oversight Program Analysis and Evaluation office are brought to the Milestone Decision Authority for study and decision.

Evidence: DoD Instruction 5000.2 ("Operation of the Defense Acquisition System"), Subsection E2.1.2, describes the function of the "Pre-Acquisition Category Technology Projects" which form the technology demonstrations part of the ATD program. Subsection E6.1.5 provides guidance with regard to the preparation of Analysis of Alternatives. The Analyses of Alternatives are examined used by the later-stage Milestone Decision Authority in shoosing how to pursue the later-stage development program and are overseen by the Program Analysis and Evaluation office within the Office of the Secretary of Defense. This office is independent of the technology programs and serves an analytic and oversight function, not a managerial function.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The priorities for programs come from the Integrated Priorities Lists developed by the Combatant Commanders, who are the chief military commanders and represent the interests of the warfighters. For purposes of the development of science and technology to support the IPL, the IPL is distilled into the Joint Warfighting Science and Technology Plan by the Joint Chiefs of Staff staff, which document the warfighter's technology priorities. The JWSTP is used to rank the funding requests.

Evidence: The Integrated Priorities Lists are established by the Combatant Commanders. The Joint Staff reduces the IPLs to the JWSTP. The JWSTP is used by the Technology Focus Teams (formerly Technology Area Review and Assessment process) to result in the recommended ATD programs of the military services and defense agencies.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: There are several reviews of the ATD program. The DoD Comptroller reviews programs annually, and programs are subject to losing funds depending on their abililty to perform financially. In addition, panels for each of the military services review these financial materials by technical area, assess the quality of the ATD programs and make recommendations to improve the program. Each demonstration project must go through a sequence of approved tests and must submit performance data to the Department. If the project fails too many tests, funding is terminated. Individual projects are not statutory programs, so if the projects aren't productive, funding will be cut off by organizational leadership. Most projects are of short duration (a few years). Contracted efforts are overseen by the Department and schedule information, progress reports submitted by the contractors, accounting data and briefings are used to track performance.

Evidence: Program justification materials, financial reports (e.g., appropriation status forms), and associated test schedules and backup materials are collected and reviewed annually and as additionally needed.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The program-related organizations within each military service and defense agency are expected to adhere to budgets and schedules, as underutilized funds are needed for other operational programs. Non-performing programs are likely to have funds taken away to meet more immediate needs. Use of common assets (test ranges and instrumentation) is monitored by central oversight organizations, including the Test Resource Management Center, the military service developmental and operational test organizations and the Director, Operational Test and Evaluation. Many programs have been restructured or had funding reduced when financial reports indicate a pattern of slow performance or if the parent organization has reason to believe that the program is inadequately focused on the needs of the warfighter. Good ratings on performance appraisals are contingent on satisfactory performance of manager's programs. If program performance is not up to par (as determined against program plans), the manager may be let go or reassigned. The projects in this program are usually of very limited scope and research and development programs are by nature somewhat uncertain, so program performance (including that of contractors) is usually not a concern over an extended time. However, if partners (contractors) underperform noticeably, they will be at a disadvantage in subsequent competitions.

Evidence: Budget justification material, DoD Directive 5010.38 titled "Management Control Program", and Program Budget Decisions, which adjust program funding, all provide insight into cost, schedule and performance results and allow managers to make needed funding adjustments. The Federal Acquisition Regulations (see FAR Part 42.15) requires contractor performance evaluations, which are used as input to subsequent bid evaluations for contractors.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The Department of Defense (DoD) carefully monitors obligations and expenditures for each Component and program/project. Each Component is expected to obligate and expend funds on a timely schedule (the Department-wide standard for these types of programs is obligations above 90% of appropriated funds in the first year of availability and expenditures above 50% in the first year and 80% in the second year). If not, the project is likely to have funds taken away at the Component or Department level. These reviews take place during subsequent Budget formulation and at mid-year of each execution year as funds are reallocated from lagging programs to immediate needs. Because research and development programs are incrementally funded, funds are subject to being taken away even in the absence of a major program review. Contract and grant awards are reported promptly and accurately to the financial tracking systems. In order to reach the 50% and 80% expenditure rates, partners (contractors) must file claims for expenses on a reasonably timely basis.

Evidence: Reports prepared by the military services' financial processing centers generally show obligation and expediture rates in excess of the Department-wide standards (funds are available for two years by law). Defense Finance and Accounting Services Reports record official oblgations and expenditure rates. Sometimes unofficial component financial documents are allowed as evidence of obligations and expenditures for purposes of determining projected funding needs when the financial management systems decremented in question 3.6 fail to accurately reflect actual obligation or expenditure rates. Program awards are reported within 90 days to the Federal Procurement Data System--Next Generation.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Managers must regularly use competitive award processes, mandated by statute and regulation, which result in program efficiencies. Also, the Department's Comptroller and Acquisition, Technology and Logistics organizations conduct annual program reviews. Programs with low obligation or expenditure rates or minimal technical progress are subject to reductions in funding. In addition, the Department requires that acquisition programs have acquisition and implementation plans reviewed by oversight executives. The activities of ATD programs vary so widely that overall efficiencies (in terms of single indicators) are difficult to establish and implement. Nevertheless, the Department is in the process of establishing an efficiency measure which will monitor the rate of transition from ATD program to a system development program. Baselines are not yet established.

Evidence: The Competition in Contracting Act (CICA) (Public Law 98-369) and 10 United States Code, Section 2304 and 2361 mandate that, unless specifically excepted, these awards must be executed using a competitive process. The competitive process weeds out non-performers and has proven to be one of the most efficient and cost-effective ways of developing advanced warfighting technologies. The requirements for acquisition plans are documented in the Federal Acquisition Regulations, Section 7.105.

Does the program collaborate and coordinate effectively with related programs?

Explanation: DoD's integrated science and technology planning process starts with the Director, Defense Research and Engineering Science and Technology (DDR&E S&T) Strategic Plan (pending signature) and the Combatant Commanders' Integrated Priority Lists, and results in the Basic Research Plan (BRP), the Joint Warfighting Science and Technology Plan (JWSTP), the Defense Technologies Area Plan (DTAP) and services' and agencies' strategic plans. During this process each of the military services and defense agencies work to reconcile potential technical capabilities with military needs. Each military service is allowed to attend other services' and agencies' annual program reviews and ask questions and discuss related programs and concerns, in order to coordinate before initial or additional funding is provided. As a result, programs are often run by one military service with the products being provided to the other services. Much of the output of these programs is information which can be used by later hardware and software development programs. The technical results of these programs are provided to the Defense Technical Information Center, which makes the information readily available to the other Defense organizations. Advanced Concept Technology Development and, more recently, Joint Concept Technology Development projects, are required to include active participation by at least two services and agencies. DoD coordinates in selected areas with other Federal agencies--for example with the National Aeronautics and Space Administration (NASA) and the Federal Aviation Administration on aeronautics programs and NASA on space programs. There are few related State, local and private programs.

Evidence: The Director, Defense Research and Engineering Strategic Plan (pending signature), the BRP, the JWSTP, the DTAP and Memorandums of Understanding/Memorandums of Agreement document program goals, individual project details and coordination processes. DoD has a large number of interagency agreements (Memoranda of Agreements, Memoranda of Understanding) that document interagency coordination requirements (e.g., with NASA, DoE, NIST, etc.). One example is the current Next Generation Air Transportation System, which is in the concept development stage to result in the national air transport system (aircraft avionics, fused flight information, improved air terminals, etc.). Both conept development plans and research and development plans are being produced and early research and development is being conducted.

Does the program use strong financial management practices?

Explanation: Official, certified financial reports often lag execution significantly and are unable to positively account for too much Department of Defense funding. This program doesn't have a central financial oversight process that a few centrally-managed DoD programs have, as it would be difficult and wasteful to do so here. DoD has established critical management controls that all programs must follow. In addition, through the Business Management Modernization Program, DoD's financial systems and processes are being improved. Nevertheless, it will take time before a central system is effectively implemented.

Evidence: DoD Directive 5010.38 "Management Control (MC) Program", and Business Management Modernization Report to Congress are steps toward a comprehensive system of controls.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Defense Research and Engineering Strategic Plan, pending signature, which sets out the program vision, priorities and management principles, should improve understanding of the program by all partners. Process improvements recommended by multiple ongoing and completed studies and initiatives are in the works. For example, the Business Management Modernization Program, designed to improve financial systems and processes, is organized into five areas called "domains". The Acquisition Domain is assessing Science and Technology (S&T) as one of the key capabilities and is making recommendations to improve management and financial systems. DoD also implements recommendations from the General Accounting Office (GAO), DoD's Inspector General (IG) and independent review panels to improve the ATD program. In addition, through the Defense Acquisition Workforce Program, DoD created the S&T Manager career field to improve the training and education of S&T managers.

Evidence: Supporting documentation includes GAO and IG audits, which point to prior deficiencies, reviews of ATD from internal DoD managers, and S&T Manager career field information. [Need references to specific GAO and IG reports. Also, need some documentation of the changes themselves and the linkage with the GAO and IG audits, etc.]

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Most of this program allocates funds through a competitive contract award process. Announcements for these awards are made through Broad Agency Announcements, aggregated announcements of general needs and funding availability to industry, academia and other organizations. Awards are made after ranking of proposals (technical merit, etc.) and comparison with military needs. A portion of the funds are retained by the Defense organization for contract administration, oversight, development of internal competence and internal research and development. The internal allocation process gathers and ranks proposals from internal performers and the results are submitted to review panels. Funds that are awarded to Federally Funded Research and Development Centers (FFRDCs) go through a separate process and funded projects are limited by available funds and by policies that restrict such awards to the documented unique expertise of the Center and is generally limited as well by caps imposed by Congress through appropriations law.

Evidence: The capabilities of the internal laboratories are established by the in-house laboratory assignments process, as well as the independent Base Realignment and Closure reviews that the Department of Defense infrastructure undergoes (see http://www.defenselink.mil/brac/ ). The capabilities of the FFRDCs are limited by policy set forth in the Federal Acquisition Regulations, Section 35.017, which ensure that funds awarded to FFRDCs are only for work for which there is no private sector competition. This regulation (along with Departmental oversight) ensures that the Defense Department's FFRDCs maintain unique and high quality expertise while limiting their scope of operations.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program's managers are in the process of establishing the long-term measures, including measurement baselines. At this time, the program is unable to demonstrate progress using those measures.

Evidence: The Department is in the process of establishing long-term measures.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program's managers are in the process of establishing long-term measures, including baseline measurements.

Evidence: The Department is in the process of establishing long-term measures.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program's managers are in the process of establishing the efficiency measurements, including baseline data. Until the measures are in place, the program will be unable to demonstrate improved efficiencies.

Evidence: The Department is in the process of establishing efficiency measures, including baseline data.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: This defense focus of the program is largely unique to the Department of Defense (there are some specialized programs programs run by other agencies or organizations, but nothing on a broad scale), but there are overlaps and similarities with other government agencies and some private sector prototyping and technology demonstration. However, there are as yet no known quantitative comparisons with other similar programs and there are no known independent studies that might be able to provide that comparison on a non-quantitative basis. Although there are no broad comparisons at this time, it should be noted that the program has produced some extremely valuable products, including the technologies used in the Internet, stealth aircraft, radars that track moving vehicles, and much more. The number and range of such products would not have been possible if the program had been poorly managed or poorly planned.

Evidence: There are no known meaningful comparisons.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Because there are no recent independent evaluations of the program, this question cannot be answered in the affirmative.

Evidence: Because there are no recent independent evaluations of the program, no documentation exists to support a Yes answer.