Detailed Information on the
Science and Technology: Threat and Vulnerability, Testing and Assessment Assessment
- View this program’s assessment summary.
- Visit ExpectMore.gov to learn more about how Federal Government programs are assessed and their plans for improvement.
- Learn more about detailed assessments.
| Program Code | 10002396 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Program Title | Science and Technology: Threat and Vulnerability, Testing and Assessment | ||||||||||
| Department Name | Dept of Homeland Security | ||||||||||
| Agency/Bureau Name | Science and Technology | ||||||||||
| Program Type(s) |
Research and Development Program |
||||||||||
| Assessment Year | 2004 | ||||||||||
| Assessment Rating | Results Not Demonstrated | ||||||||||
| Assessment Section Scores |
|
||||||||||
| Program Funding Level (in millions) |
|
- Ongoing Program Improvement Plans
- Completed Program Improvement Plans
- Program Performance Measures
- Questions/Answers (Detailed Assessment)
Ongoing Program Improvement Plans
| Year Began | Improvement Plan | Status | Comments |
|---|---|---|---|
| 2008 |
Conduct assessment of program and project milestones. |
Action taken, but not completed | The activities from the TVTA program have been split and the majority have gone to the Command, Control and Interoperability Division (CID). The program re-evaluated that status of its FY 2008 milestones earlier this year based on the impacts resulting from the continuing resolution for FY 2008. The program is gearing up for another review of program and project milestones for the upcoming fiscal year. |
| 2008 |
Collect requirements from customers and made adjustments in current and out year plans. |
Action taken, but not completed | The activities from the TVTA program have been split and the majority have gone to the Command, Control and Interoperability Division (CID). CID is working with its customers to identify additional requirements and make adjustments to current and future year plans. The discussions will impact FY 2010-2014. |
Completed Program Improvement Plans
| Year Began | Improvement Plan | Status | Comments |
|---|---|---|---|
| 2005 |
As the program was scored results not demonstrated, the FY 2006 Budget decreased program funding by $16 million. |
Completed | In response to these findings, the FY 2006 budget included a decrease of $18.8M for the Threat Awareness Portfolio (TAP) previously known as Threat and Vulnerability Testing and Assessment (TVTA). |
| 2005 |
The Administration will await the results of the program evaluation and analysis process that the Directorate is developing. That process will evaluate the progress that each Portfolio makes toward achieving their respective goals and remedying any deficiencies. Once that process is complete, it is expected that this Portfolio will achieve an increased PART score once it is reassessed. |
Completed | To enhance fiscal control and accountability, the S&T Directorate initiated comprehensive program evaluations and analysis processes in FY 2005, including regular reviews by DHS components and their customers, S&T management, and outside peer reviewers. Updated information on the recent evaluations, performance measures, portfolio progress, and specific objectives met was provided to OMB. The process now being followed by S&T has remedied any deficiencies noted before. |
| 2007 |
Realign budget and programs based on new organization. |
Completed | Realigned program efforts to Command, Control and Interoperability and Human Factors to improve efficiency and effectiveness of the Directorate. |
| 2008 |
Develop 5 Year Research and Development Plan. |
Completed | The activities from the TVTA program have been split and the majority have gone to the Command, Control and Interoperability Division (CID). CID provided input to the S&T Directorate??s 5 year R&D plan for FY 2008-2013. The plan will identify activities and planned milestones for each project within the Division and will be provided to the Hill later this summer. |
Program Performance Measures
| Term | Type | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Long-term/Annual | Output |
Measure: Improvement in the National capability to assess threats and vulnerabilities to terrorist attacksExplanation:10 capabilities to be assessed
|
Questions/Answers (Detailed Assessment)
| Section 1 - Program Purpose & Design | |||
|---|---|---|---|
| Number | Question | Answer | Score |
| 1.1 |
Is the program purpose clear? Explanation: The purpose of the program is to provide and develop capabilities that enable the creation, application and dissemination of threat and vulnerability information to prepare for, anticipate, detect, and prevent terrorist activities and help restore the Nation's operational capabilities Evidence: This program has been developed to support the Homeland Security Act of 2002, Section 302(2); National Strategy for Combating Terrorism; Official requests from other agencies for assistance; PDDs 39 and 41. |
YES | 20% |
| 1.2 |
Does the program address a specific and existing problem, interest or need? Explanation: TVTA provides technology and information to DHS, IAIP, the Intelligence Community, Federal, State and Local government by addressing their needs for: Forecasting; Information Sharing; System Optimization; Simulation; Threat Assessment; Weapons of Mass Destruction (WMD) Assessment; Nuclear Assessment Program; Advanced Scientific Computing; Threat and Vulnerability Information Systems R&D and Prototyping; Biometric (Border Safe). Evidence: This program has been developed to support the Homeland Security Act of 2002, Section 302(2); National Strategy for Combating Terrorism; Official requests from other agencies for assistance; PDDs 39 and 41. |
YES | 20% |
| 1.3 |
Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort? Explanation: TVTA provides unique technologies and knowledge services to DHS, Federal, State, and Local agencies creating enhanced capabilities to share information and expand and create new technology Evidence: Intelligence Technology Innovation Center participation; Steering Group; National Strategy; (Detailees)-informal vetting of incoming R&D/DHS wide efforts |
YES | 20% |
| 1.4 |
Is the program design free of major flaws that would limit the program's effectiveness or efficiency? Explanation: There exist legal limitations as to information sharing and Privacy Issues Evidence: Statutory and Constitutional limitations |
NO | 0% |
| 1.5 |
Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly? Explanation: Resources have been strategically balanced to meet Statutory and Policy requirements, specific end user needs, urgency and rapid application of technologies taking into account other related, complementory efforts being performed in other agencies, and Cost/Benefit impacts on National Security. Evidence: Strategic Planning process documents end user requirements |
YES | 20% |
| Section 1 - Program Purpose & Design | Score | 80% | |
| Section 2 - Strategic Planning | |||
|---|---|---|---|
| Number | Question | Answer | Score |
| 2.1 |
Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program? Explanation: TVTA's long-term performance measure is: Provide measurable advances in information assurance, threat detection and discovery, linkages of threats and vulnerabilities and capability assessments and information analysis required by Departmental missions to anticipate, detect, deter, avoid, mitigate and respond to threats to US homeland security. Evidence: Strategic Planning Documents; FYHSP; Documentation provided to OMB |
YES | 10% |
| 2.2 |
Does the program have ambitious targets and timeframes for its long-term measures? Explanation: Ambitious technology capabilities and information sharing efforts with multiple federal, state, and local agencies have been identified in TVTA's FY 05 - 10 Strategic Planning efforts; TVTA's performance measure is: Improvement in the National capability to assess threats and vulnerabilities to terrorist attacks Evidence: Strategic Planning Documents; FYHSP; Documentation provided to OMB |
YES | 10% |
| 2.3 |
Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals? Explanation: Performance measures that can demonstrate TVTA's progress in meeting its strategic objectives have been developed as part of TVTA's Strategic Planning efforts. TVTA's performance measure is: Improvement in the National capability to assess threats and vulnerabilities to terrorist attacks Evidence: Strategic Planning Documents; FYHSP; Documentation provided to OMB |
YES | 10% |
| 2.4 |
Does the program have baselines and ambitious targets for its annual measures? Explanation: End-item deliverables for each identified need have been prioritized, timelined and tied to performance measures and strategic goals as stipulated in the strategic planning documents Evidence: DHS FYHSP database |
YES | 10% |
| 2.5 |
Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program? Explanation: The TVTA portfolio functions through an Integrated Product Team (IPT). All the IPT members contribute to developing plans and executing actions to accomplish the 5 strategic program objectives. Evidence: Strategic Planning Documents; IPT execution documents, statement of work; contracts; ORD Interagency Procurement Request |
YES | 10% |
| 2.6 |
Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need? Explanation: Independent evaluation are currently being conducted on selected programs Evidence: Independent evaluation of Nuclear Assessment Program; S&T Directorate STAR Program Review Process; Broad Agency Announcements |
YES | 10% |
| 2.7 |
Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget? Explanation: The TVTA IPT mission statement is supported by specific strategic objectives that are tied to goals and capabilities. Evidence: Strategic Planning Documents |
YES | 10% |
| 2.8 |
Has the program taken meaningful steps to correct its strategic planning deficiencies? Explanation: Strategic Planning efforts in progress Evidence: Strategic Planning Documents |
YES | 10% |
| 2.RD1 |
If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals? Explanation: TVTA provides a unique Federal government program to predict and project the threat environment. Evidence: TVTA briefing to the S&T Corporate Review Board; interactions with other agencies; HSAct statutory requirements |
YES | 10% |
| 2.RD2 |
Does the program use a prioritization process to guide budget requests and funding decisions? Explanation: Methodology for prioritization: Statutory and Policy requirements; Constitutional limitations; End-user requirements; Urgency of user needs; Rapid application of technologies; balanced delivery of capabilities; build logic/sequencing; Performance status of ongoing projects; other similar work ongoing in other agencies; alignment with DHS,ST, and portfolio missions and goals; and cost/benefit considerations on National Security are all considered in relation to one another through an exhaustive internal review process. Consultations with outside experts in subject areas is done frequently. Finally, all HSPD and presidential priorites as evinced from directives and budget documents inform the prioritization process . Evidence: Strategic Planning Documents |
YES | 10% |
| Section 2 - Strategic Planning | Score | 100% | |
| Section 3 - Program Management | |||
|---|---|---|---|
| Number | Question | Answer | Score |
| 3.1 |
Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance? Explanation: Specific performance information is required for each Statement of Work Evidence: Monthly performance reports from Laboratories |
YES | 12% |
| 3.2 |
Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results? Explanation: Performance reports and progress payments are reviewed on a monthly basis prior to acceptance. Evidence: Monthly performance reports from Laboratories |
YES | 12% |
| 3.3 |
Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose? Explanation: Statement of Work specify purpose, reporting requirements, and performance measurements Evidence: Monthly performance reports from Laboratories |
YES | 12% |
| 3.4 |
Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution? Explanation: Evidence: |
NO | 0% |
| 3.5 |
Does the program collaborate and coordinate effectively with related programs? Explanation: Our mission is to support knowledge discovery for DHS, Federal, State, and Local agencies. Evidence: National Strategy; Strategic Planning Process |
YES | 12% |
| 3.6 |
Does the program use strong financial management practices? Explanation: Evidence: |
NO | 0% |
| 3.7 |
Has the program taken meaningful steps to address its management deficiencies? Explanation: Evidence: |
NO | 0% |
| 3.RD1 |
For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality? Explanation: In process of implementing project processes based on Program Management Book of Knowledge Standards, which is based upon the U.S. and international program management standard ANSI/PMI 99-001-2000. This is the global standard for program management practicres and stresses a quality approach. Evidence: Guidelines to Develop FY 04 Office of Research and Development Execution Plans (10/10/03) |
YES | 12% |
| Section 3 - Program Management | Score | 62% | |
| Section 4 - Program Results/Accountability | |||
|---|---|---|---|
| Number | Question | Answer | Score |
| 4.1 |
Has the program demonstrated adequate progress in achieving its long-term performance goals? Explanation: Relatively new program which inherited existing programs Evidence: Enabling legislation is the Homeland Security Act of 2002; National Strategy; Strategic planning process |
SMALL EXTENT | 7% |
| 4.2 |
Does the program (including program partners) achieve its annual performance goals? Explanation: Relatively new program, 1st annual performance measurements goals will be collected in 1st Quarter of FY 2005. Evidence: Enabling legislation is the Homeland Security Act of 2002; National Strategy; Strategic planning process |
SMALL EXTENT | 7% |
| 4.3 |
Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year? Explanation: Relatively new program, 1st annual performance measurements goals will be collected in 1st Quarter of FY 2005. Evidence: Enabling legislation is the Homeland Security Act of 2002; National Strategy; Strategic planning process |
SMALL EXTENT | 7% |
| 4.4 |
Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals? Explanation: TVTA is a unique Federal government program to predict and project the threat environment. Compares favorably to State/private programs. Evidence: Intelligence Community comparisons of similar efforts; MATRIX |
SMALL EXTENT | 7% |
| 4.5 |
Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results? Explanation: Evidence: |
NO | 0% |
| Section 4 - Program Results/Accountability | Score | 26% | |
- View this program’s assessment summary.
- Visit ExpectMore.gov to learn more about program assessment and improvement by the Federal Government.
- Learn more about detailed assessments.
