Detailed Information on the
Tribally Controlled Postsecondary Vocational and Technical Institutions Assessment

Pursuing strategies to improve the quality and consistency of grantee performance data.

Exploring whether efficiencies can be gained by combining this program with other programs serving similar objectives.

Adopting the Department's common measures for minority-serving postsecondary institutions and collecting baseline data for these common measures.

Seeking legislative program reforms that include increased grantee accountability, improved performance reporting, and a clear focus on strengthening the academic and technical skills of post-secondary Indian students.

Setting short and long-term targets for the common measures for minority-serving postsecondary institutions.

Developing a new measure to gauge cost-effectiveness based on successful outcomes.

Measure: Percentage of participants who entered employment in the 1st quarter after program exit

Explanation:

Measure: Percentage of participants who were employed in the 1st quarter after program exit who remain employed in 2nd and 3rd quarters after exit

Explanation:

Measure: Percentage change in earnings: Based on (1) pre-enrollment to program exit; (2) 1st quarter after exit to 3rd quarter after exit

Explanation:

Measure: Cost per participant

Explanation:OMB job training common measure for efficiency.

Is the program purpose clear?

Explanation: The program provides Federal funding for the basic support, operation, and improvement of tribally controlled postsecondary vocational and technical institutions, so that funded institutions may provide continued and expanded vocational education and training opportunities for Indian students.

Evidence: Sec. 117 of the Carl D. Perkins Vocational and Applied Technology Education Act.

Does the program address a specific interest, problem or need?

Explanation: The program addresses the postsecondary vocational and technical education and training needs of the Indian student population.

Evidence:  

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The program provides Federal funding to tribally controlled postsecondary vocational and technical institutions that do not receive Federal support under two major Federal sources of funding for Indian colleges and universities and are administered by the Bureau of Indian Affairs -- the Tribally Controlled College or University Assistance Act and the Navajo Community College Act. Without Federal support under this program, it is unlikely that the grantee institutions would be able to continue providing the vocational and technical education and training services they currently provide to Indian students.

Evidence:  

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: Since this program supports institutions not receiving funds under the other two major authorities supporting Indian postsecondary institutions, funds awarded under this program represent a sizeable share of public funds received by these institutions for the education and training of Indian students.

Evidence:  

Is the program optimally designed to address the interest, problem or need?

Explanation: There is no conclusive evidence that a different design would improve program performance. However, the absence of conclusive evidence does not mean that program improvements are not needed.

Evidence:  

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Consistent with measures established under the job training common measures framework, the Department is working to develop several long-term indicators that are tied to short term goals and are consistent with the program's scope and activities.

Evidence:  

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: Through the common measures matrix, the program has established a limited set of performance indicators designed to measure program performance/progress, including for example, placement in employment, degree attainment, and skill attainment. However, the Department must establish numerical targets and ensure that performance data exists to report on those targets. In addition, any short-term measures (whether the common measures or additional measures) must be linked to long-term goals. To the extent performance targets are set by states, a process should be put in place to ensure that state-defined targets are appropriately rigorous and that a methodology can be developed for aggregating performance data at the national level.

Evidence: There is currently one annual performance indicator which measures degree or certificate attainment. The measure, however, is flawed in that the denominator is not derived from the cohort from the time they enter school. Rather, the denominator is those students who have made it into their final semester.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: While the program receives regular and timely annual performance information from grantees, the information cannot yet be tied to a strategic planning framework where a limited number of annual performance goals demonstrate progress to achieving long-term goals.

Evidence: Instructions for this question indicate that a "no" is required if the program received a "no" for both questions 1 and 2 of this section.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: This program serves a narrow and very specific population -- those schools not served by similar BIA programs. Department and BIA staff do not collaborate on efforts to improve program outcomes.

Evidence:  

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: No evaluation is planned for this program.

Evidence:  

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The program does not have long- or short-term strategic planning performance goals that can be aligned with budget policy.

Evidence:  

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The Department has not taken the necessary steps to develop a strategic planning framework where a limited number of annual performance goals demonstrate progress to achieving long-term goals.

Evidence: Any efforts to develop a strategic planning framework would have to be done in the context of the longstanding and unique Govt.-to-Govt. relationship between Indian tribal governments and the U.S. government.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: While grantees provide regular and timely information, such information does not adequately address program performance. However, information on the grantees' compliance with program requirements and objectives is collected annually and continuation awards are dependent upon a determination of progress being made.

Evidence:  

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: This program has not instituted an appraisal system that holds Federal managers accountable for grantee performance. However, as part of the President's Management Agenda, the Department is planning to implement an agency-wide system -- EDPAS -- that links employee performance to progress on strategic planning goals. Grantee performance is monitored on an annual basis through review and approval of annual budget plans, compliance reviews, audits, and site visits. Grantees that do not meet Federal requirements are required to submit improvement plans and can have grants reduced or discontinued for serious or persistent failures to comply.

Evidence:  

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated within the timeframes set out by Department schedules and used for the purposes intended.

Evidence:  

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has not yet instituted procedures to measure and improve cost efficiency in program execution. However, as part of the President's Management Agenda, the Department is implementing an agency-wide initiative to re-evaluate the efficiency of every significant business function, including the development of unit measures and the consideration of competitive sourcing and IT improvements.

Evidence:  

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: Education's 2004 Budget satisfies the first part of the question by presenting the anticipated S&E expenditures (including retirement costs) for this program, which constitute 23.8 percent of the program's full costs. However, Education has not satisfied the second part of the question because program performance changes are not identified with changes in funding levels. The program does not have sufficiently valid and reliable performance information to assess the impact of the Federal investment.

Evidence:  

Does the program use strong financial management practices?

Explanation: The program has a positive audit history, with no evidence of internal control weaknesses.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Department has not taken the necessary steps to develop an efficiency measure or to obtain valid performance information from grantees.

Evidence:  

Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process?

Explanation: This program has a very small pool of eligible applicants, and although there is no earmark of funds, the review process has, since enactment of the program, resulted in the funding of the two largest tribally controlled postsecondary vocation and technical institutions.

Evidence:  

Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process?

Explanation: The legislation specifies very narrow eligibility criteria and a very limited number of institutions meet them. Since the program's inception, the same two grantees have been the recipients of the grants.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program has a strong relationship with its grantees as well as a high level of understanding of what grantees do with the resources allocated to them. Program oversight includes documentation of grantees' use of funds and site visits.

Evidence:  

Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Data are collected and compiled from annual reports and used for mandated reports to Congress. However, these data and reports are not readily available to the public, in print or on the internet, and do not reflect program impacts.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: Consistent with measures established under the job training common measures framework, the Department is working to develop several long-term indicators that are tied to short term goals and are consistent with the program's scope and activities.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: Through the common measures matrix, the program has established a limited set of performance indicators designed to measure program impacts, including for example, placement in employment, degree attainment, and skill attainment. However, the Department must establish numerical targets and ensure that performance data exists to report on those targets. In addition, any short-term measures (whether the common measures or additional measures) must be linked to long-term goals.

Evidence:  

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The common measures framework includes an efficiency measure -- cost per participant. The Department estimates that the annual cost per participant is $6,951. However, the lack of valid outcome data makes it impossible to link these costs to the achievement of program goals.

Evidence:  

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: To date, the Department has been unable to provide data that informs on the common measures. The answer to this question could change depending on the Department providing the necessary data.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: No evaluations have been conducted or are planned for this program.

Evidence: