Detailed Information on the
Transportation Security Administration: Transportation Vetting and Credentialing Assessment

Fully incorporate best practices into the development of Secure Flight life-cycle cost & schedule estimates by FY2009: (update life-cycle costs & schedule estimates; demonstrate that the SF schedule has the logic in place to identify the critical path, integrates lower level activities in a logical manner, & identifies the level of confidence in meeting the desired end date).

TSA will institutionalize strong program management principles and techniques to ensure accountability on behalf of contractors and staff. Furthermore, TSA will ensure staff have the proper skills to manage large capital investment programs by creating an action plan for filling skill gaps.

Develop enterprise wide services as building blocks that can be combined and tailored to serve different populations: Establish a universal guidance for TTAC services by FY 2010 that will provide program offices interested in vetting with an "a la carte" menu of standardized services and costs for such things as enrollment, name based check (terrorism, criminal, and immigration), adjudication, and granting benefits (crential or access) per individual.

TSA should take steps to promulgate a rulemaking for Alien Flight School and submit financial data supporting the new fee level. This financial data should be submitted by the end of the fiscal year.

By January 2007, TSA will develop a policy on the periodicity and procedures of reviewing regulations to determine if the program is maximizing the intended benefits at the least societal cost.

By 2007, TSA will rebaseline Secure Flight, TWIC, and RT to ensure cost, schedule and performance goals are accurate and consistent with the program's goals.

TSA will establish baselines and targets for its long-term, annual and efficiency measures in the FY 2008. TSA will begin demonstrating results against these performance measures in FY 2007.

TSA will show evidence that it is using acquisition vehicles that shift much or all of the risk away from the government, such as firm fixed price contracts. TSA will improve its contracting practices to ensure that minimum performance thresholds are documented and incentives and penalties are provides for good/poor performance. Significant improvement to shift to FFP contract will be demonstrated by January 2007. Furthermore, TSA will require the use of earned value management on most contracts no later than January 2007.

Measure: Percentage of individuals undergoing a TTAC security threat assessment (STA)

Explanation:This measure indicates the what percentage of TTACs total defined population (656M) is receiving an STA. Estimated populations: SF - 650M (annually); CVP - 625K (annually); RT - 3M (cum); TWIC - 750K (cum); Hazmat - 1.5M (cum); AFSP - 15K (annually); and SIDA - 750K (annually)

Measure: Percentage of all commercial airline domestic flight passengers vetted. (Secure Flight Only)

Explanation:This is an indication of what percentage of commercial airline domestic flight passengers are being protected from potential terrorist activities.

Measure: Ratio of false positives to total names processed.

Explanation:This measure assesses how well the program is restricting access to people who do (or do not) present a threat to the nation's transportation system.

Measure: Security Threat Assessment (STA) cost per vetted individual.

Explanation:This measure ensures that the cost of performing security threat assessments remains within acceptable levels.

Measure: Number of privacy adherence infractions

Explanation:This measure assesses the extent of production of the public's privacy by measuring the rate of compliance with privacy regulations, policies, and procedures as a function of vetting throughput.

Is the program purpose clear?

Explanation: The Transportation Threat Assessment and Credentialing (TTAC) program consolidates the management of all vetting and credentialing programs within TSA. TTACs mission is to ensure that individuals engaged in various aspects of U.S. transportation do not pose a threat to national security or transportation security. The program consists of the following screening programs: Secure Flight (SF), Crew Vetting (CVP), Transportation Worker Identification Credential (TWIC), Registered Traveler (RT), Hazardous Materials Commercial Driver's License Endorsement (HAZMAT), and Alien Flight Student (AFSP).

Evidence: 1. Aviation and Transportation Security Act (ATSA) 2. Intelligence Reform and Terrorism Prevention Act 3. HAZMAT: Interim Final Rule (IFR) published by TSA on November 24, 2004 under 49 CFR part 1572. 4. AFSP: IFR published by TSA on September 20, 2004 in accordance with Section 612 of Vision 100.

Does the program address a specific and existing problem, interest, or need?

Explanation: Intelligence identifies a very real ongoing threat to transportation security as evidenced by the World Trade Center bombing in 1993, the Oklahoma City bombing in 1995, the USS Cole bombing in 2000, the attacks in Madrid in March 2004, in London July 2005, and the attacks on pipeline infrastructure in Saudi Arabia and Iraq in 2006. Since we cannot predict specific terrorist activity, the name-based threat assessment methodologies employed by the programs within TTAC to identify potential threats to the Nation's transportation system are a critical tool to strengthen the security of the transportation network.

Evidence: INTEL threat documents (Classified: "SECRET").

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Five of the six TTAC programs have distinct and unique mission requirements and no duplication exists with other government or private efforts. The Secure Flight which is being developed to replace the existing airline passenger prescreening process that is performed by U.S. air carriers themselves and some duplication of efforts will exist in the transition. However, this duplication is required to ensure the security of the aviation system until the program full operational capability.

Evidence: ATSA, Intelligence Reform and Terrorism Prevention Act, HAZMAT: Interim Final Rule (IFR) published by TSA on November 24, 2004 under 49 CFR part 1572, AFSP: IFR published by TSA on September 20, 2004 in accordance with Section 612 of Vision 100.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The HAZMAT, CVP, and AFSP programs have been operational for more than a year and there is no evidence to support another design would have improved efficiency or effectiveness. However, SF, RT, and TWIC, which comprise a majority of program resources, are each in varying stages of development. The program design generally lacks an infrastructure that ensures strong program management and techniques to ensure accountability. The department has not followed a disciplined life cycle approach to develop these programs, thus major program shifts have occurred to address privacy and security issues as well as to establish an improved program management infrastructure. Because of these challenges these programs are currently updating key life cycle cost and program justification documentation, or in the case of SF undergoing a rebaseline of cost and schedule, to establish more robust program management moving forward.

Evidence: TTAC Weekly Intel Report, 9-11 Commission Report Executive Summary, TWIC Prototype Report, RT Pilot Program Initial Launch Metrics Evaluation, GAO reports on Secure Flight.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: For the operational programs, TSA is effectively targeting intended beneficiaries by conducting threat assessments on nearly 2 million individuals last year. However, throughout the design and development phases for TWIC, RT and SF, they have experienced significant issues with defining and fixing cost, schedule, and performance requirements for the program. TSA has also relied heavily on high risk acquisition processes that are not performance-based in nature. Therefore, because of poor program design and a lack of sufficiently skilled staff, resources have not been targeted effectively in the design phase for these programs. TSA is actively addressing these issues by revising acquisition strategies to reach initial operational capability within acceptable cost and schedule thresholds.

Evidence: 1. SF GAO Report. 2. RT Pilot Program Initial Launch Metrics Evaluation. 3. TWIC Prototype Report and DHS-OIG Reports.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: TTAC has taken steps to establish long-term, outcome measures that would be representative of the entire program and focus on the risk reduced as a result of implementing program objectives. In the interim, the program has developed a long-term output goal to help gather data necessary to develop outcome measures. This output goal is percent of TTAC population receiving Security Threat Assessments (STA), which is intended to show the dramatic impact the program will have on the transportation sector by vetting nearly 700 million persons per year.

Evidence: FY 2007 and FY 2008 Congressional Justification and SF OMB-E300.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The baseline and targets for the program's long-term measure is under development.

Evidence: FY 2007 and 2008 TSA Congressional Justification.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: TTAC has developed a robust set of short-term performance measures, which directly supports the long-term measure. Current measures include average cost per security threat assessment, ratio of false-positives, and number of privacy infractions.

Evidence: FY 2007 and 2008 TSA Congressional Justification.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Ambitious targets and baselines for the annual measures are under development.

Evidence: FY 2007 and FY 2008 Congressional Justification, draft measures working papers.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: For the performance goals that partners are responsible for achieving, through MOUs and other agreements, the FBI, ICE, CIS, and state and local law enforcement agencies commit to specific costs and timelines for conducting threat assessments of individuals. These agencies provide data on these measures to ensure fees are established at levels appropriate to cover the costs of running the program. Furthermore, TSA is moving towards performance-based contracting and recently awarded a contract ties award fees directly program results by requiring earned value reporting and a performance evaluation plan.

Evidence: 1. HAZMAT MOA with FBI 2. AFSP/HAZMAT MOA with FBI (fingerprinting) 3. TWIC Inter Agency Agreement (IAA) with CBP 4. SF IAA with CBP 5. Performance-based contract

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Although relatively new programs, TWIC, RT and SF have undergone comprehensive audits by either GAO or another independent entity. These evaluations were of sufficient scope and quality to evaluate program effectiveness. Specifically, the SF report evaluated ten high-risk areas and made recommendations that need to be addressed before the program becomes operational. In addition, the RT evaluation focused heavily on the benefits provided to the traveler as well as improved TSA screening productivity as well as highlighting the success of the kiosks and biometrics.

Evidence: 1. SF - GAO Secure Flight Development and Testing Under Way, but Risks Should be Managed as System Is Further Developed (March 2005) and GAO Significant Management Challenges May Adversely Affect Implementation of Transportation Security Administration's Secure Flight Program (February 2006) 2. RT - Pilot Program Initial Launch Metrics Evaluation (August 2005). 3. TWIC - Prototype Report (August 2005) and draft DHS-OIG report.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The TTAC programs annual and long-term performance measures are integrated in budget requests. The program's Congressional Justification also includes all direct and indirect costs needed to meet performance targets. Further, for the programs in development that are not fee-funded, tradeoffs in performance can be demonstrated when resource levels increase or decrease.

Evidence: 1. FYHSP 2. FY 2007 Congressional Justification 3. Program budget documents 4. DHS PMA BPI scorecard.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: TSA is addressing all of the strategic planning deficiencies that have been identified for TTAC. Specifically, TSA is working to adopt develop baselines and targets for its annual and long-term performance measures.

Evidence: TSA Management Review (TTAC) template, draft performance measures.

Are all regulations issued by the program/agency necessary to meet the stated goals of the program, and do all regulations clearly indicate how the rules contribute to achievement of the goals?

Explanation: The operational programs (AFSP, Hazmat, and CVP) under TTAC have each issued necessary rules to clearly outline the goals of the programs and how the data being collected will be utilized to achieve the goals of the programs. The TWIC program is preparing for rollout to the nations ports and just recently released its Notice of Proposed Rule Making (NPRM) on May 10, 2006. TSA articulates the goal of the TWIC rulemaking is to meet requirements defined by the Maritime Security Act, which requires that credentialed merchant mariners and workers with unescorted access to secured areas of vessels and facilities be subject to a security threat assessment and receive a biometric credential needed to access secured areas. RT and SF are currently in the rulemaking process and plan to issue their respective regulation/guidance in the future.

Evidence: 1. Hazmat NPRM 2. TWIC NPRM 3. AFSP Interim Final Rule (IFR)

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The timely receipt of timely performance information from critical stakeholders is very important to the success and performance of all TTAC programs. Each program has its own processing time requirements that it must comply in order to meet specific performance requirements regarding time. CVP requires data from the airlines on passengers no later than one hour before flight time. The Hazmat and AFSP each have a 30-day turnaround requirement after receiving information from the applicant to process the application. While in the planning phases, SF, TWIC, and RT will also collect data from stakeholders that will have time-based performance requirements in order to operate effectively. All of these programs do or will depend upon airlines, other government entities, or Trusted Agents to gather source data and provide it to TTAC in order to meet established processing standards. Even after the receipt of the source data all TTAC programs are still dependent upon other key government partners such as FBI, ICE, CIS, CBP, etc. to provide timely receipt of unique datasets or fingerprint processing to perform security threat assessments in a timely manner. TSA uses this performance information to set program parameters, adjust resources and/or fee levels.

Evidence: 1. Numerous agreements with other government and non-government entities. 2. AFSP Customer Satisfaction Survey

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: TTAC is beginning to award performance-based contracts whenever practicable. By incorporating language regarding cost, schedule and performance the program intends to make program partners more accountable for achieving program success. However, to date the program has only awarded one performance-based contract (awarded March 2006) out of 26 contracts currently issued. Since this one contract is so large it's important to note that, in dollar terms, 52 percent of the programs total contract value is associated with performance-based contracts. Furthermore, the program does not clearly define and quantify performance standards to ensure accountability for its managers.

Evidence: 1. O&M contract (performance evaluation plan). 2. Secure Flight O&M contract.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Since many of the TTAC program are currently undergoing baseline reviews, several of the programs do not have a program plan that includes schedules for obligations. While in FYs 2004 and 2005 resources for SF and CVP obligated funding in excess of 92% to fund pilot program, the program is expecting large unobligated carryover balances in FY 2006 because of program delays and baseline changes. The AFSP and Hazmat programs are both operational and fee-funded each program is required to monitor fee receipts and obligations very closely to ensure these programs operate within their fee driven cash flow availability.

Evidence: 1. CAS financial reports

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The organization is relying more on full and open competitions, performance-based contracts, and the use of earned value management. TSA recently awarded a firm fixed price contract for SF operations and maintenance. TSA has also developed an efficiency goal to reduce the cost of each threat assessment. The program is currently developing a baseline and targets for this measure.

Evidence: TSA contract management, SF O&M contract, TWIC acquisition plans, FY 2008 Congressional Justification.

Does the program collaborate and coordinate effectively with related programs?

Explanation: TTACs biggest success story has been the coordination between SF and CBP. Since CBP was already communicating with all major airlines, with the exception of Southwast, to collect data for their Advanced Passenger Information System (APIS) their system became a logical starting point for SF which is required to communicate with all air carriers flying domestically in the U.S. The organizations work together to address issues such as messaging, data format, and automation challenges associated with smaller carriers to create a system that presents a single interface to the airlines to provide the Federal government the data it needs. To gain access to the government datasets utilized in performing STAs the organization had to establish agreements with other government agencies (i.e. DOS, FBI, ICE, CPB, CIS, etc.) that also require continual interaction to ensure the integrity of the data being accessed. The organization also collaborates regularly with major program stakeholders such as (TSC, CBP, FAA, airlines, etc.) to determine the best approach for addressing specific programmatic issues concerning IT development, connectivity, and changes in stakeholder business practices.

Evidence: 1. SF IAA with CBP 2. HAZMAT MOA with FBI 3. AFSP/HAZMAT MOA with FBI (fingerprinting) 4. TWIC Inter Agency Agreement (IAA) with CBP

Does the program use strong financial management practices?

Explanation: The draft FY 2005 Audit of Financial Statements revealed three material weaknesses in the areas of financial reporting; undelivered orders, accounts and grants payable, and disbursements; and budgetary accounting.

Evidence: FY 2005 Audit of Financial Statements

Has the program taken meaningful steps to address its management deficiencies?

Explanation: TTAC management representatives meet weekly to discuss program status where issues are addressed and, where possible, quick resolutions found. TTAC, and/or parts of its former self, have also participated in TSA's Management Control Objective Plan (MCOP) for both FY 2005 and FY 2006. The current MCOP addresses such management areas as information and physical security, HR personnel actions, time & attendance, acquisition procedures, and contract management. The MCOP is reviewed periodically and a year-end summary report is prepared outlining our success/failure in achieving our plan. TSA is developing a corrective action plan to address how the program intends fix the weakness and ensure that it's not repeated. Furthermore, the Secure Flight program is addressing all 10 areas of interest identified by the GAO before implementation so it should adequately address management deficiencies.

Evidence: TTAC Management Control Objective Plan (MCOP) for FY 2006.

Did the program seek and take into account the views of all affected parties (e.g., consumers; large and small businesses; State, local and tribal governments; beneficiaries; and the general public) when developing significant regulations?

Explanation: All TTAC program regulations have undergone the full notice and comment period required by the NPRM process before rules become final. During this phase TSA coordinates with stakeholders to gain an understanding of how the regulations/guidance will impact their operations. Furthermore, all efforts are made to try to mitigate the impact to stakeholders and economic analysis is performed to also gain an understanding of the economic impact of the rule issuance. TSA thoroughly discusses these comments in the preamble and throughout the final regulation to explain which comments have and have not been accepted and why.

Evidence: 1. Hazmat NPRM 2. TWIC NPRM 3. AFSP IFR

Did the program prepare adequate regulatory impact analyses if required by Executive Order 12866, regulatory flexibility analyses if required by the Regulatory Flexibility Act and SBREFA, and cost-benefit analyses if required under the Unfunded Mandates Reform Act; and did those analyses comply with OMB guidelines?

Explanation: TSA prepares regulatory evaluations of all rulemakings. These regulatory evaluations cover the requirements of Executive Order 12866, regulatory flexibility act, and Unfunded Mandates act. Each evaluation was reviewed by OMB OIRA as a part of the clearance process and all recommended corrections were accomplished before publishing to the public docket.

Evidence: 1. Hazmat NPRM 2. TWIC NPRM 3. AFSP IFR

Are the regulations designed to achieve program goals, to the extent practicable, by maximizing the net benefits of its regulatory activity?

Explanation: The regulations are designed to maximize the security benefit while giving full consideration to economic consequences. TSA does not use a cost benefit ratio but both benefits and costs are full discussed in the regulatory evaluation. TSA maximizes the benefits giving full consideration to qualitative as well as quantitative issues. NEED for example

Evidence: 1. Hazmat NPRM 2. TWIC NPRM 3. AFSP IFR

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: TTAC is currently developing its baseline for the long-term goal; therefore, the program has not yet demonstrated adequate progress.

Evidence: 1. FYHSP 2. SF OMB-E300

Does the program (including program partners) achieve its annual performance goals?

Explanation: TTAC is currently developing its baseline for its annual goals; therefore, the program has not yet demonstrated adequate progress.

Evidence: 1. FYHSP 2. SF OMB-E300

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: TTAC is currently developing its baseline for its efficiency goal; therefore, the program has not yet demonstrated progress in achieving its goals.

Evidence: 1. FYHSP 2. SF OMB-E300

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The SF and CVP programs have worked with the FBIs Terrorist Screening Center (TSC) to compare the effectiveness of screening subjects by matching against watch-list records. Through this relationship TTAC has been able to compare the performance of its vetting engines against other name-matching systems to determine effectiveness. The RT program has worked closely with Customs & Border Protection and Canadian Immigration officials examining the performance of Canada's Nexus Air program, which is designed to simplify border crossings for pre-approved travelers between the U.S. and Canada using biometrics. In addition, the RT program has also examined the performance of the Iris Recognition Immigration System (IRIS) at Heathrow Airport in London and the Privium program at Schiphol Airport in Amsterdam. In both cases, these operations were designed to "fast track" immigration checks at these airports using iris scans as the biometric identifier. The RT program was particularly interested in the biometric results of the Privium program since it utilized a semi-fixed camera for taking the iris scans.

Evidence: 1. Vetting Analysis Summary Report 2. Passport Control Using Iris Recognition: Heathrow Passenger Operations Experience - Results and Lessons Learned.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The SF program has been subject to a GAO audit for the past 2-years to ensure that the program is achieving its stated goals. Both reports were supportive of program goals; however, the report was critical in noting that much work remains in the areas of redress, privacy, security, oversight, and life-cycle costs. GAO noted that these areas must be addressed before the program becomes operational. Some progress has been made to address these concerns.

Evidence: 1. Independent evaluations of TWIC and RT. 2. SF GAO Audit

Were programmatic goals (and benefits) achieved at the least incremental societal cost and did the program maximize net benefits?

Explanation: The CVP, AFSP, and Hazmat programs have only been operational for a short period of time (each less than 2 years) so no program analysis has been performed to observe the effects of these regulations. While some benefit can be demonstrated, TSA has not developed guidance on procedures used to evaluate if programmatic benefits are achieved. Since the HAZMAT rule was an IFR, TSA is reviewing public comments as part of finalizing the rule. Numerous program modifications have been made to allow for flexibility which reduces burden. TSA balanced issues of federalism raised by the states against pure economy of scale issues raised by other members of the public. The costs were minimized within these other constraints.

Evidence: The AFSP, Hazmat, and recently released TWIC rule making documents each include a reference to OMB Circular A-25 requiring a review of the fee at least every 2-years. Since Hazmat and TWIC each provide a a benefit that only covers a maximum 5-year period these regulations/procedures will be revisited automatically within that time period before indidivuals reapply for extension of the same benefit. 1. Hazmat NPRM 2. AFSP IFR 3. TWIC NPRM