Detailed Information on the
Defense Applied Research Program Assessment

Change the expert evaluation process to use fully independent review panels in assessing the performance of the program.

Work with the research community and Congress to explain the need to limit claims on research grant funds to proposals that independently can meet the standards of a strict merit- review process.

Continue to ensure that adequate funding exists to carry promising basic research results into the realm of applied research.

Measure: Reduce by half, within three years, grant and contract award funding not (1) resulting from needs identified by military or technical experts within the Services or Agencies and (2) awarded through the merit-review process.

Explanation:Grants and contracts should meet needs validated by military and technical experts.

Measure: Percentage of ambitiously chosen Defense Technology Objectives (DTO) targets achieved.

Explanation:Annual technology targets should be set ambitiously, but a significant portion of them should be met each year. The next assessment will be in FY 2007 for FY 2005 and FY 2006 DTOs.

Measure: Certification that technology area review panels are independent, comprised of technically competent experts external to DoD.

Explanation:Review team will be independent, comprised of technically competent reviewers from outside of DoD. The DoD reviewers are necessary to ensure consistency with the strategic context and policy objectives of DoD.

Is the program purpose clear?

Explanation: The mission of the Defense Science and Technology (S&T) Program is to ensure that warfighters have superior and affordable technology to support their missions and provide revolutionary war-winning capabilities. Applied research is directed toward determining the specific operational parameters of potential technologies and developing and evaluating the feasibility and practicality of those technologies to military needs.

Evidence: The Defense S&T Strategy and Defense Technlogy Area Plan (DTAP) provide overarching goals and direction for the Applied Research Program. The work conducted by the Applied Research program is outlined in the Financial Management Regulation. Together, these plans provide the user community, S&T performer communities as well as S&T stakeholders with descriptions of military needs and specific objectives. SUPPORTING DOCUMENTATION: DoD S&T Strategy, DTAP, Financial Management Regulation.

Does the program address a specific and existing problem, interest or need?

Explanation: Overarching interests or needs are technologies to make forces lighter, more mobile and more lethal, with the ability to respond appropriately to a variety of threats and crises, as well as to prevent technological surprise by adversaries.

Evidence: The Defense Technology Area Plan (DTAP) and the Joint Warfighting S&T Plan document technology outline military requirements at a medium level of detail. The DTAP outlines an investment strategy for the Applied Research and Advanced Technology Development Programs in 12 relevant technology areas, including Materials and Processes, Sensors and Electronics, Space Technology, etc. Each area has sets of objectives for 5, 10, and 15 years. In addition to the DTAP, for some specific high priority initiatives, the DoD has developed more detailed roadmaps and conducted gap analyses. Three of these are the National Aerospace Initiative, Energy and Power Initiatives and Surveillance and Knowledge Systems. SUPPORTING DOCUMENTATION: Defense S&T Strategy, DTAP, Joint Warfighting S&T Plan

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Applied Research program is designed to ensure that DoD maintains its competitive edge over potential adversaries in the future, which is a DoD-unique mission. DoD outlines Defense Technology Objectives (DTOs) which make the research community aware of funding in particular technology areas that will help maintain that competitive warfighting advantage. Broad Agency Announcements (BAAs) also detail for researchers particular areas of interest by funding Service or Agency but allow researchers to propose specific topics and approaches. A goal of DoD's S&T integrated planning process - called Reliance - is to avoid redundanacy and duplication. Specifics can be found in the DTAP by technology area, DTOs and in Broad Agency Announcements (BAAs), published by the individual Military Services and Agencies for response by research organizations interested in working as grantees.

Evidence: SUPPORTING DOCUMENTATION: DDR&E Directive (charter), Defense S&T Advisory Group charter, Reliance charter, Reliance Planning Documentation Preparation Manual, DTAP, DTOs

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Reliance process attempts to match Science and Technology with military needs, and tries to to limit redundancy and focus resources efficiently. A biennial review (Technology Area Readiness Assessment) assesses program strengths and weaknesses; provides recommendations for improvement; provides a technical, budgetary, and programmatic assessment; and reviews overall program balance, adequacy of the approach used, and progress to date. A weakness of the process is that Reliance is overwhelmingly a consensus process steered by committees with members, from the Services and DoD agencies, with missions that sometimes compete with one another. A better S&T program might result from a melded DoD program, with less ownership of efforts by individual military Services. However, there is no strong and clear evidence that another approach would improve the program in a major way.

Evidence: The military Services and DoD agencies are required to define and document their review processes. SUPPORTING DOCUMENTATION: Reliance Planning Document Preparation Manual.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The DoD process is built around announcements of research areas (Broad Agency Announcements--BAAs), and competition (submission of proposals) by potential researchers. Merit review of proposals follows identification of problems and specific research opportunities by proposers, including in-house researchers. Funds for these purposes are made available through the DoD military Services and agencies after review through the DTO-TARA-Program linkage process, which tries to link funds to military needs.

Evidence: SUPPORTING DOCUMENTATION: All BAAs can be accessed on www.cbd-net.com

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: DoD's Defense Technology Area Plan and the Joint Warfighting Science and Technology Plan (JWSTP), which address applied research, have goals for the various technologies supported by DoD. Applied research by its very nature is a somewhat long-term and speculative activity, so most long term goals are fairly general and the goals of the program as a whole represent a portfolio of many smaller and more specific sub-goals and targets that often are defined in the process of discovery.

Evidence: The Defense Technology Area Plan lays out goals by technology area. The DTOs often contain specific goals and objectives for individual technology projects. Also, Service or Agency BAAs articulate thrusts that are meaningful to potential researchers.SUPPORTING DOCUMENTATION: DTAP, JWSTP, DTOs, DoD Service and Agency BAAs (see,e.g., www.cbd-net.com).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The DTAP establishes near-, mid- and long-term goals (5, 10 and 15 years, respectively) for the technology areas. The Defense Technology Objectives (DTOs) often present ambitious targets by specific timeframes, but the process is shaped by the process of scientific and technical discovery itself.

Evidence: SUPPORTING DOCUMENTATION: DTAP, DTOs

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Many DTOs provide annual milestones that address technical progress. DoD established a metric to evaluate the progress of DTOs that is reported in the Annual Report to the President and Congress. This metric - Monitor the Status of DTOs - measures the percentage of DTOs that are progressing satisfactorily toward their established goals. This ensures DoD maintains a technological edge across a broad spectrum of warfighting capabilities.

Evidence: SUPPORTING DOCUMENTATION: Reliance Planning Documentation Preparation Manual, DTOs, Annual Report metrics.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Milestones for DTOs are specifically outlined in many individual program descriptions. The goal for Monitoring the Status of DTOs is that 70% of DTOs are progressing satisfactorily.

Evidence: DOCUMENTS: Reliance Planning Documentation Preparation Manual, DTOs, Annual Report metrics.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Within the DTOs, all partners are held accountable for meeting the technical goals and funding requirements. Contracted efforts are documented in clearly defined statements of work with deliverables. Interagency agreements are documented by Memorandums of Understanding and Memorandums of Agreement. On the other hand, increasing numbers of recipents of Applied Research funds have sought Congressional earmarking assistance, thus circumventing in whole or in part, the merit-based award process, and DoD has not beeen notably successful in limiting them. In the long-term, the health of the research enterprise is greatly diminished by awards based on factors other than their potential contribution to the DoD mission, protecting the national security.

Evidence: SUPPORTING DOCUMENTATION: Memorandums of Understanding, DTOs

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent review panels (Technology Area Reviews and Assessments--TARAs) assess each DTAP technology area biennially. The reviews assess program strengths and weaknesses; provide recommendations for improvement; provide another technical, budgetary, and programmatic assessment; and review overall program balance, adequacy of the approach used, and progress to date. The review panels are responsible for reviewing their entire technology area. Major action items are tracked and must be resolved as directed by departmental memorandum. At least 2/3 of the TARA Panel members are from outside DoD and outside of the Applied Research management chain. Most team members are recognized experts from organizations such as the National Academy of Sciences, the National Academy of Engineering, the Institute of Medicine, the Defense Science Board and the scientific advisory boards of the Military Departments. However, the Chair and the Executive Secretary are in the departmental management chain, and thus the panels cannot be considered to be independent.

Evidence: SUPPORTING DOCUMENTATION: TARA instructions, TARA Panel member lists, TARA major action items from 2002 and 2003.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Annual and long-term goals for each Applied Research project are outlined in the budget justification books (Research and Development Descriptive Summaries) provided to Congress. Goals are also outlined in the DTOs, and the Components are held accountable for meeting the technical goals and funding requirements.

Evidence: SUPPORTING DOCUMENTATION: Annual Report metrics, DTOs, Research and Development Descriptive Summaries - www.dtic.mil/descriptivesum

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Director for Defense Research and Engineering publishes a strategic plan every four years. (The 2004 Research and Engineering Strategy should be available in August, 2004.) In addition, DoD's S&T planning process has been in place for about 10 years and is refined as improvements are identified. Many of these changes are documented in the annually published Planning Documentation Preparation Manual. Biennially, the DTAP Panels identify the strategy for the individual technology areas, flowing from the overarching S&T strategy. On the other hand, a major strategic planning deficiency is the annual addition of hungreds of projects by Congress that do not contribute fully to the national security mission (military planners and scientific and technical experts have specific technical approaches thrust upon them, rather than determine those efforts through a politically-independent process).

Evidence: SUPPORTING DOCUMENTATION: Defense S&T Strategy, DTAP, Planning Documentation Preparation Manual, conference reports accompanying annual appropriations bills/acts.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: This program is largely unique to the Department of Defense.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The program has to compete anew each year for funds within the Department. When the funding total is determined, actual number of awards is adjusted to fit within the amount available.

Evidence: SUPPORTING DOCUMENTATION: Quadrennial Defense Review, DoD annual funding guidance, Program Decision Memoranda leading to the President's Budget.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: There are several reviews of the Applied Research program. The OSD Comptroller reviews programs annually, and programs are subject to losing funds depending on their abililty to perform financially. In addition, researchers are required to document their results in technical publications or summaries. Review panels for each of the Services, made up of academic researchers, industrial researchers, scientists from other Federal agencies and managers of the research program, review these materials by technical area, assess the quality of the research programs and make recommendations to improve the program.

Evidence: SUPPORTING DOCUMENTATION: Program decision documents from annual agency-wide reviews, Reliance Planning Documentation Preparation Manual, technical publications,.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The research organizations within each Service and Agency are expected by the parent organizations to adhere to budgets and schedules, as underutilized funds are needed for other operational programs. Non-performing programs are likely to have funds taken away to meet more immediate needs. Many programs have been restructured or had funding reduced when financial reports indicate a pattern of slow performance or if the parent organization has reason to believe that the program is inadequately focused on the needs of the warfighter.

Evidence: SUPPORTING DOCUMENTATION: Management Controls DoD Directive 5010.38, budget preparation documents (Program Budget Decisions).

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: DoD monitors obligations and expenditures by each Component. Each Component (DoD military Service or Agency) obligates and spends funds in a timely way.

Evidence: Obligation reports prepared by the financial processing centers of the Department show obligation rates at 90% in the first year of availability (funds available by law for two years). SUPPORTING DOCUMENTS: Financial reports from DoD accounting centers.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Managers use competitive award processes, mandated by statute and regulation, which result in program efficiencies. In addition, the Comptroller conducts annual program reviews and programs with low obligation rates or minimal progress are subject to reductions in funding. On the other hand, the level of Congressionally-directed solutions to real or asserted military problems has reached about 20% of the overall program, making it very unlikely that the program could efficiently achieve its goals, as outlined in Section 1. The Competition in Contracting Act (CICA) mandates that, unless otherwise directed by statute, all Applied Research efforts must be done using a competitive process. The competitive process tends to weed out bidders who clearly are not qualified, but due to technology-specific solutions directed by the Congress in annual appropriation conference reports, many awards are made to a very small field of competitors, who have questionable ability or willingness to serve the defense mission.

Evidence: SUPPORTING DOCUMENTATION: Competition in Contracting Act, Program Budget Decisions, annual Conference Reports accompanying appropriations acts (statutory laws).

Does the program collaborate and coordinate effectively with related programs?

Explanation: DoD's S&T planning process ensures coordination of related efforts across the Services and Agencies. In addition, the biennial review panels (TARA Panels) also have observers from outside DoD; including the National Space and Aeronautics Agency, Department of Homeland Security, Federal Aviation Agency, Department of Justice, Department of Energy, Environmental Protection Agency; to ensure coordination across the federal government. DoD also has a large number of interagency agreements that document interagency coordination requirements.

Evidence: SUPPORTING DOCUMENTATION: TARA Major Action Items, Planning Documentation Preparation Manual, Memorandums of Understanding/Memorandums of Agreement

Does the program use strong financial management practices?

Explanation: Financial reports often lag execution and significant funding cannot be positively accounted for in the overall DoD budget.

Evidence: SUPPORTING DOCUMENTATION: Management Controls DoD Directive 5010.38, Business Management Modernization Report to Congress, monthly and yearly financial reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has implemented process improvements recommended by multiple ongoing and completed studies and initiatives. The Business Management Modernization Program, designed to improve financial systems and processes, is organized into 5 areas called "domains". The Acquisition Domain is assessing S&T as one of the key capabilities and is making recommendations to improve the management and financial systems. DoD also implements recommendations from the GAO, IG and review panels to improve the Applied Research program. In addition, through the Defense Acquisition Workforce Program, DoD created the S&T Manager career field to improve the training and education of S&T managers.

Evidence: SUPPORTING DOCUMENTATION: S&T Manager career field information, Acquisition Domain website - www.acq.osd.mil/dpap/ebiz/index.htm

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Grants and contracts both are generally executed through competitive processes.

Evidence: Title 10, U.S. Code, Section 2361 requires competition in some of these awards. Other Statutory provisions and regulations impose additional competition requirements. SUPPORTING DOCUMENTATION: 10 U.S.C. 2361

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has provided many options for improved DoD technologies over the years.

Evidence: SUPPORTING DOCUMENTATION: DTO Success Stories.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The metric - Monitor the Status of DTOs - shows DTOs are progressing satisfactorily. In addition, TARA reviews have positively assessed the progress of technology in the DTAP technology areas. DoD's DTAP and JWSTP reviews also hold the managers of technology areas accountable for meeting the goals outlined in the DTAP report and DTOs.

Evidence: SUPPORTING DOCUMENTATION: Annual Report metrics, DTAP, DTOs.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation:  

Evidence: Program is executed efficiently from an administrative point of view.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: This program is largely unique to the Department of Defense, but there are overlaps with other government agencies and private sector applied research. Results of this program have included many technical breakthroughs, and the execution of the program compares well with these other efforts.

Evidence: SUPPORTING DOCUMENTATION: DTO Success Stories, Annual Report metrics.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: As of the time of the review, there are no independent evaluations of the program. (See question 2.6.)

Evidence: