Detailed Information on the
Transportation Research and Development Program Assessment

Establishing long-term measures and annual performance baselines and targets that address the degree of success in coordination and collaboration of DOT RD&T activities and assets to address shared transportation goals and objectives.

Creating a database system to inventory and track all of DOT's RD&T activities.

Continuing development and use of R&D Investment criteria in a more robust and consistent way that evaluates how well DOT RD&T programs or projects are performing.

Measure: Percentage and value of RD&T programs that incorporate shared-research and cost-leveraging techniques.

Explanation:This measure will demonstrate how well DOT RD&T programs are actively engaged in collaboration to enhance effective use of resources and results. The measure captures those programs that are conducted in collaboration with other DOT agencies (crossmodal), other Federal agencies (interdepartmental) or private for-profit or non-profit organizations. The measure will have two components - number and dollar value of collaborative efforts. The RD&T Coordination program is developing baselines and targets for this measure.

Measure: Degree to which DOT research, development, and technology programs demonstrate effective application of the R&D investment criteria.

Explanation:All DOT RD&T programs must demonstrate that they meet the investment criteria and use evaluation methods to determine the effectiveness of their programs. The RD&T Coordination program conducts annual assessments of the DOT Operating Administrations with RD&T programs in their use of the R&D investment criteria and evaluation best practices. In FY 2005 and 2006 the reviews consisted of information sharing, identification of crossmodal areas, and verification of the existence or absence of unnecessary duplication. No research duplication was discovered. In FY 2007, the program is developing a more quantitative rating system for the evaluation best practices using a uniform definition of the elements and indicators of implementation. Based on the FY 2007, the program will develop a more rigorous evaluation tool that will establish a numerical rating scale. Since RITA is a new program, this measure is necessary as a first step requiring 4-5 years to complete in ensuring DOT RD&T programs meet the investment criteria and use adequate evaluation methods. RITA is developing baselines and targets for this measure.

Measure: Annual milestones to improve the management and ensure the effectiveness of DOT research, development, and technology activities.

Explanation:2007 Milestones: 1. Develop RITA performance measures; 2. Develop an implementation strategy for RD&T coordination 2008 Milestones: 1. Complete database of DOT RD&T programs; 2. Complete baselines and targets for RITA performance measures; 3. Develop common performance measures for DOT RD&T programs 2009 Milestones: 1. Develop methodology for evaluating RITA efforts to improve coordination and effectiveness of RD&T programs.

Measure: Cost of executing basic research, development, and technology (RD&T) coordination functions

Explanation:This measures the cost of conducting RD&T coordination activities including the continued development of the RD&T Planning & Investment Control process and the first phase of implementation of the Research Planning and Investment Control (RPIC) process per program dollar in research and development. This measure would only last through 2010 when the RPIC process is planned to be fully implemented across DOT programs.

Is the program purpose clear?

Explanation: The purpose of the Research, Development, and Technology (RD&T) Coordination program is to "coordinate, facilitate, and review the Department of Transportation's research and development programs and activities." The program aims to improve the coordination of, reduce duplication of, and improve the effectiveness of DOT's research and development programs to better achieve the DOT R&D strategic goal of fostering innovations that lead to effective, integrated, and intermodal transportation solutions. The RD&T Coordination program is part of DOT's Research and Innovative Technology Administration (RITA).

Evidence: Public Law 108-426, the Norman Y. Mineta Research and Special Programs Improvement Act of 2004 that authorized the establishment of the Research and Innovative Technology Administration .

Does the program address a specific and existing problem, interest, or need?

Explanation: The RD&T Coordination program addresses the need to improve the strategic planning and coordination, and reduce duplication, of DOT research programs. DOT research and development programs are spread across the operating administrations (e.g. in highways, railroads, and aviation). While each administration controls its own specific research to obtain its own individual mission, this can lead to duplication of effort, inefficiencies, and poor strategic planning. The RD&T Coordination program is designed to systematically improve the coordination of research and development programs across the Department, to avoid duplication of research efforts and provide strategic clarity to the Department's multi-modal and intermodal research efforts, while coordinating the multi-faceted research agenda of the Department.

Evidence: 108th Congress, House of Representatives Report 108-749, Part 1, Oct. 6, 2004, pg. 2. Letter Report, Committee on the Review of the USDOT Strategic Plan for R&D, National Research Council, August 2, 2006, Appendix A, Transportation Research, Development and Technology Strategic Plan: 2006-2010 http://www.rita.dot.gov/publications/transportation_rd_t_strategic_plan/ GAO-03-500 Transportation research: Actions Needed to Improve Coordination and Evaluation. From Conclusions on pg 32. http://www.gao.gov/new.items/d03500.pdf

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The DOT Operating Administrations (e.g. Federal Highway Administration, Federal Railroad Administration, Federal Aviation Administration, Federal Transit Administration) conduct modal research on various transportation issues, while RITA's RD&T coordination program is used to examine and coordinate the research portfolio in aggregate, across modal lines and against Department-wide priorities. No other entity has the responsibility to coordinate DOT's research portfolio or set Department-wide research goals.

Evidence: Research Activities of the Department of Transportation: A Report to Congress (March 2005) pg. 2. RITA FY 2006 Budget Estimates Submitted to the Committee on Appropriations, pg. 32.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The design of the RD&T Coordination program is based on the integrated RD&T planning process established by DOT Order 1120.39A. This order established two senior-level bodies to ensure cross-modal collaboration and coordination of RD&T efforts where common activities exist. All OAs with research funding are required to actively participate in these bodies. RD&T coordination is accomplished through the following: 1) annual budget guidance - The RD&T Coordination program works with the OAs and DOT's senior leadership to identify RD&T budget priorities and prepares annual budget guidance which the OAs use in developing budget requests and which RITA and OST use to review budget requests. By coordinating the RD&T portion of the budget planning process, the program ensures that all DOT RD&T activities are aligned with DOT goals and objectives. 2) Assessments of DOT RD&T programs on their adherence to the R&D Investment Criteria, alignment to DOT strategic goals, and identification of areas of crossmodal collaboration. These assessments examine the merit review process, peer/expert review, performance measures, and stakeholder involvement of the DOT RD&T programs. 3) promotion of best practices for managing and evaluating research programs 4) developing a common performance measurement protocol for DOT RD&T programs The RD&T program began in 2005, and there is currently no evidence that another approach or mechanism would be either more efficient or effective. The program is only beginning to fully implement the components of its design. Future performance should be more indicative of the effectiveness of the program's design.

Evidence: DOT Order 1120.39A. Research Activities of the Department of Transportation: A Report to Congress (March 2005) pgs. 21-23. Transportation RD&T RD&T Strategic Plan 2006 -2010, Documentation of RD&T Program Reviews FY 2008 Budget Estimates Submitted to the Committee on Appropriations, pgs. 92-93. GAO-06-917, Transportation Research: Opportunities for Improving the Oversight of DOT's Research Programs and User Satisfaction with Transportation Statistics, May 2006. http://www.omb.gov/part Transportation RD&T Strategic Plan: 2006-2010, November 2006, pgs. 131-140. FY 2008 RD&T Funding: A Report to Congress, Draft, February 16, 2008, pgs. 37-41.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program's beneficiaries are the DOT operating administrations (OAs) with research and development programs. The RD&T Coordination program works closely with these OAs in examining and agreeing upon crossmodal collaborative programs and R&D policies and priorities to meet the DOT strategic goal of fostering innovations leading to effective, integrated, and intermodal transportation solutions. The program has also conducted assessments on how well the R&D Investment Criteria have been applied to DOT RD&T programs as well as on the alignment of RD&T programs with the Department's strategic goals. Program resources are targeted to RD&T coordination and assessments.

Evidence: Federal Register/Volume 70, No. 33 (February 18, 2005) Section 1.46 Department of Transportation Strategic Plan: 2006-2011

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The RD&T Coordination program has not yet developed long-term performance measures that measure how the program's coordination and evaluation activities improve coordination, reduce duplication, and better enable the DOT R&D programs to foster innovations leading to effective, integrated, and intermodal transportation solutions.

Evidence: The program has not yet developed long-term performance measures. RD&T goals and milestones, upon which measurements will be based, are consistently applied in both the Transportation RD&T Strategic Plan 2006 -2010 and FY 2008 Budget Justifications.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The RD&T Coordination program is working to develop long-term measures with baselines and targets.

Evidence: The program has not yet developed long-term performance measures with baselines and targets. The Transportation RD&T RD&T Strategic Plan 2006 -2010 and FY 2008 Budget Justifications contain the program's strategic goals and mission.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The RD&T Coordination Program has developed four annual measures for progress improving the effectiveness of DOT RD&T programs. The first measure is the percentage and value of RD&T programs that incorporate shared-research and cost-leveraging techniques. The second and third measures are the degree to which DOT RD&T programs meet the R&D investment criteria and use evaluation best practices. The fourth is a milestone measure, containing different annual milestones as a way of measuring the program's progress implementing its coordination and review responsibilities. The RD&T Coordination Program is a new program that has long-term goals of improving the coordination and effectiveness of DOT RD&T programs. Although the program is developing specific long-term measures, the annual measures contribute to the long-term outcomes of efficient use of R&D resources (measured by the percentage and value of DOT RD&T programs that incorporate shared-research and cost leveraging techniques) and effective RD&T programs (measured by the effective implementation of R&D investment criteria and incorporation of evaluation best practices).

Evidence: Annual performance measures will be contained in budget justifications and annual performance plans.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The RD&T Coordination program is working to develop baselines and ambitious targets for all of its annual measures.

Evidence: The Transportation RD&T RD&T Strategic Plan 2006 -2010 and FY 2008 Budget Justifications contain the program's strategic goal and mission. Annual performance baselines and targets will be in annual budget justifications/annual performance plans.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The DOT operating administrations (e.g. highways, railroads, and aviation) are the primary partners in R&D coordination through their participation in the annual assessment of RD&T programs, annual program planning, and budget and performance planning. Each operating administration attributes its R&D activities to DOT goals in accordance with the Transportation RD&T Strategic Plan. As measures for the RD&T program are adopted, each operating administration will also incorporate those measures into their planning, implementation and reporting. Through annual budget submissions, each partner uses the RD&T budget guidance to prepare their budget submissions containing their proposed RD&T requests. These three processes - planning, budgeting and review - makes the operating administrations full partners in the collective planning cycle.

Evidence: Transportation RD&T Strategic Plan: 2006-2010 Research Activities of the Department of Transportation: A Report to Congress (March 2005)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The RD&T Coordination Program was established in FY 2005. It does not currently have evaluations that meet all of the criteria of this question - evaluations that are unbiased/independent, of a sufficient scope, are high quality, and are conducted on a regular basis to support program improvements. However, there have been a few recent evaulations conducted of the RD&T Coordination program. GAO conducted a study in 2006 and made several recommendations that the program has started implementing. The GAO study examined (1) how the RD&T Coordination Program's responsibilities for overseeing DOT's RD&T activities differ from its predecessor, (2) the program's practices for coordinating, facilitating, and reviewing RD&T activities, and (3) the progress DOT has made in implementing GAO's 2003 recommendations on how to improve the coordination and evaluation of RD&T activities. Though not a regular, scheduled review, GAO's second study was thorough in comparing the 2003 conclusion with the 2006 state of coordination providing a useful point of comparison and tracking of conclusions. In June 2006, the National Research Council (NRC), an external, independent organization, reviewed the Transportation RD&T Strategic Plan developed by the RD&T Coordination program in collaboration with the operating administrations. The reviewers identified the plan's strengths and limitations, what a RD&T strategic plan should do, and further considerations for the program's role in the effort. As a result of the NRC recommendations, the Transportation RD&T Strategic Plan was modified to include a section on research evaluation and assessment. The NRC also will review the 2007 Supplement to the Plan.

Evidence: GAO-06-917 Transportation Research: Opportunities for Improving the Oversight of DOT's Research Programs and User Satisfaction with Transportation Statistics (May 2006) National Research Council (NRC) Review under the auspices of the Transportation Research Board (TRB) (August 2006)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: No performance information for the RD&T program has been included in budget submissions to date (therefore the program cannot define the relationship between performance targets and resources). Measures and targets are currently under development, some of which will be included in the FY 2009 budget submission and all of which will be included in a supplement to the Transportation RD&T Strategic Plan. RD&T coordination includes a full cost budget request. This includes the costs of supporting overhead operations both at the RD&T Office level, for such services as rent and IT support, and at the RITA agency level, for such services as legal counsel and public affairs.

Evidence: FY 2007 RITA Budget Submissions - Section 4 FY 2008 RITA Budget Submissions - Section 4

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The RD&T Coordination program has made significant strides in establishing the framework for improving the coordination and effectiveness of DOT RD&T programs. This has been largely accomplished through the annual budget guidance and DOT RD&T strategic plan. Using the deficiencies identified by GAO and NRC, the RD&T Coordination program has developed a work agenda that will continue to improve Departmental opportunities for leveraged RD&T investments and quantitative assessment of investment efforts. The work plan includes better articulating the RD&T Coordination program's role and developing annual milestone measures and baselines and targets for long term and annual measures; promoting best practices and more clearly identifying gaps in cross-modal policy and systems research; undertaking more focused stakeholder feedback and incorporating it; and implementing more quantitative RD&T program assessments.

Evidence: National Research Council (NRC) Review in Appendix A of the Transportation RD&T Strategic Plan GAO-06-917 Transportation Research: Opportunities for Improving the Oversight of DOT's Research Programs and User Satisfaction with Transportation Statistics (May 2006)

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The RD&T Coordination program is developing baselines and targets for annual and long-term measures. This process will identify the appropriate performance information necessary to manage the program and improve performance. As part of this process, the program is developing a database for managing research program information (from its partners - the DOT Operating Administrations) and identifying areas for increased crossmodal collaboration.

Evidence: The program is developing baselines and targets for annual and longterm measures and is only beginning to collect performance information.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The RD&T Coordination program has identified the managers who are responsible for achieving program results and managers have annual performance targets that reflect the programs goals. For example, in FY 2007, these deliverables consist of the FY 2008 Funding Report to Congress, FY 2009 RD&T Budget Guidance and Priorities, Summary Report of the FY 2007 RD&T Program Reviews, Framework for RD&T Database, Supplement to the Transportation RD&T Strategic Plan, and a Stakeholder Outreach Forum. As the program completes the development of long-term and annual measures and targets, these will also be incorporated into performance plans. Program partners are the DOT operation administrations (OAs). Each operating administration attributes its R&D activities to DOT goals in accordance with the Transportation RD&T Strategic Plan. As measures for the RD&T program are adopted, each operating administration will also incorporate those measures into their planning, implementation and reporting. Through annual budget submissions, each partner uses the RD&T budget guidance to prepare their budget submissions containing their proposed RD&T requests.

Evidence: RD&T Coordination program managerial performance plans. Annual RD&T budget guidance and annual budget requests of DOT operating administrations.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The RD&T Coordination program has an established schedule for obligations and funds are obligated consistent with this RD&T budget execution plan. However, during fiscal year 2006, the program had a carryover of $735K (roughly two thirds of its multi-year funding). This was caused by a lengthy and cumbersome internal review process for procurements that often delayed the approval of procurements. The program has taken steps to streamline its procurement review process to improve the timely obligation of funds.

Evidence: 2006 year-end spending reports from FACTS II

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The RD&T coordination program does not have an efficiency measure with baselines and targets. Further, the program is fairly new and small, and does not have many examples of the use of competitive sourcing or IT improvements to improve efficiency.

Evidence: The RD&T Coordination program does not have an efficiency measure with baselines and targets and does not have, at this point, examples of activities to improve efficiencies in program execution.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The RD&T Coordination program coordinates routinely with all DOT operating administrations to produce plans, strategic priorities, and reports including the Transportation RD&T Strategic Plan issued in November 2006. This program was established in 2005, and some of the early results of this coordination are reflected in development of RD&T strategies contained in the Department's 2006 - 2011 Strategic Plan and annual budget guidance. This guidance provides DOT OAs programs with RD&T components as to how to meet R&D investment criteria, align with DOT strategic goals, and improve the cost-leveraging and shared research across DOT.

Evidence: Transportation Research, Development and Technology Strategic Plan 2006-2010 (November 2006). DOT Order 1120.39A

Does the program use strong financial management practices?

Explanation: The Research and Innovative Technology Adminsitration (RITA) had an antideficiency act violation in 2006. Total obligations were incurred in the first quarter of 2006 that exceeded the availability of the approved apportionment for RITA's research and development account. Since the violation occurred, RITA has revised its internal procedures to prevent a reoccurrence of this type of violation and to ensure that all financial management procedures are in accordance with DOT's administrative controls. RITA has recently completed an assessment of its internal financial controls and DOT completed an assessment of the financial controls of its Enterprise Service Center that processes all of RITA's financial transactions. These assessments did not identify any material weaknesses in internal controls.

Evidence: ADA violation letter. DOT Assessmnt of A-123 compliance.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has new leadership, which is focused on correcting management deficiences, including improving financial controls and approval of procurements to ensure timely and accurate obligation of funds. During 2007, the program is also developing baselines and targets for performance measures and beginning to collect the appropriate performance data.

Evidence: RITA and DOT assessement of internal financial controls. RITA documentation to address 2006 ADA violation. Development of annual performance measures and initial collection of performance data for baselines and targets.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The RD&T Coordination program does not have long-term measures with baselines and ambitious targets. The program is new and is developing measures and targets. The program has made progress in the following areas: developing DOT RD&T strategic objectives and priorities and working to align DOT RD&T activities that are spread across the many operating administrations with these objectives; developing annual RD&T budget guidance; and beginning to assess DOT RD&T programs use of R&D investment criteria and evaluation best practices.

Evidence: The RD&T Coordination program is developing long-term measures and targets.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has four annual measures, however the program is still developing baselines and ambitious targets for all of these measures.

Evidence: The RD&T Coordination program is developing baselines and targets for performance measures.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program does not have an efficiency measure with baselines and targets and does not have many examples of the use of competitive sourcing or IT improvements to improve efficiences. Since the program is relatively new, efforts to date have focused on creating the institutional infrastructure needed to establish processes for RD&T coordination and evaluation.

Evidence: The RD&T Coordination program is relatively new and does not have examples of improved efficiencies or cost effectiveness at this point.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The RD&T Coordination is comparable to other programs that coodinate research across several entities, for example the Joint Program and Development Office that is coordinating research and development of the Next Generation Air Transportation System across several agencies. The program is fairly new, so the results of its coordination and leveraging are not fully available, but the mechanisms by which is coordinates RD&T (annual budget guidance, assessments of programs on R&D investment criteria and alignment with DOT Strategic goals, and performance measurement protocols) compares favorably to those of other similar programs.

Evidence: Structure and processes of the RD&T Coordination Program and the JPDO

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: GAO and the National Research Council have noted that RD&T coordination has helped DOT more systemmatically handle its research programs. However, the RD&T Coordination program does not currently have evaluations that meet all of the criteria of this question - evalutions that are unbiased/independent, of a sufficient scope, are high quality, and are conducted on a regular basis to support program improvements. In reviewing the DOT's Transportation Research, Development and Technology Strategic Plan 2006 - 2010, NRC said the plan: (1) Describes RD&T activities across operating agencies--helpful for comprehending program coverage, funding levels, and subject matter. (2) Attempts to relate all major RD&T activities to shared goals and priorities. (3) Budget planning process has fostered cross-agency dialogue and information exchange. (4) Identifies Emerging Research Priorities (ERPs). (5) Aligned with US DOT's overarching Strategic Plan (the committee is encouraged by the latter plan's emphasis on RD&T). In its 2006 report, GAO found that RITA, through its two coordinating bodies, has taken some action to review DOT RD&T activities for duplication and opportunities for joint efforts, however, that RITA had not established performance goals, the scope of RD&T activities to be reviewed, the methodology of the review, or how the results will be used to make decisions about future RD&T activities.

Evidence: National Research Council (NRC) Review under the auspices of the Transportation Research Board (TRB) (August 2006) Transportation RD&T Strategic Plan: 2006-2010 GAO-06-917 Transportation Research: Opportunities for Improving the Oversight of DOT's Research Programs and User Satisfaction with Transportation Statistics