Detailed Information on the
Vocational Education State Grants Assessment

The Department will set short-term targets based on the measures in the new Perkins law and develop strategies for collecting the necessary data.

Issuing regulations on implementation of performance measures systems under the new Perkins law.

Providing technical assistance on integrating challenging academic career/technical instruction and collecting and reporting better performance data under the new Perkins law.

The Department will set long-term targets based on the measures in the new Perkins law.

Grantee funding will be contingent on a rigorous assessment that student outcomes are being achieved.

Grantees will have the flexibility to focus program funds in a manner that best serve students in a given locality.

Measure: For postsecondary students - Percentage of participants who entered employment in the 1st quarter after program exit.

Explanation:

Measure: For secondary students - Attainment of a high school diploma, certificate, or GED.

Explanation:

Measure: For secondary students - Entry into employment or enrollment in postsecondary education/advanced training.

Explanation:

Measure: Cost per secondary student

Explanation:OMB job training common measure for efficiency. Cost is calculated for separately for secondary and postsecondary students.

Measure: Cost per postsecondary student.

Explanation:OMB job training common measure for efficiency. Cost is calculated for separately for secondary and postsecondary students.

Is the program purpose clear?

Explanation: The program provides financial assistance to states in support of a variety of efforts, including improving students' academic skills and technical skills, preventing drop outs, increasing graduation rates, increasing post-secondary and advanced degree placement, and improving job outcomes. These multiple and potentially overlapping objectives have caused ambiguity among stakeholders as to the central purpose of the program.

Evidence: The Department has received feedback from stakeholders that the broad scope and varied activities of the program have caused confusion at the local level about the key objectives of the program.

Does the program address a specific interest, problem or need?

Explanation: Data indicate that a significant number of students are graduating from high school and community college without the necessary academic and technical competencies to be productive members of the workforce.

Evidence: National Assessment of Vocational Education; Consolidated Annual Performance Reports.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The program is not designed such that there is consensus among stakeholders on the program's key objectives. Moreover, because the impacts of the program are not currently known, the effect of reducing or increasing the federal investment in this program is unclear.

Evidence: The lack of information on program impact is due in large part to deficiencies in performance reporting, including problems with data quality.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: The Federal contribution provides support for services that are provided to students at the state and local level. Because Federal and state funds are commingled, the extent of the value added of the Federal investment is unclear.

Evidence: There are a variety of Federal programs that seek to improve academic skills and ensure that students get into college and succeed. The diverse and varied goals of this program overlap with the goals of many other programs.

Is the program optimally designed to address the interest, problem or need?

Explanation: There are a number of program design features that warrant improvement, including for example, focusing the scope and objectives of the program and developing a more rigorous performance accountability framework.

Evidence: The Department has received feedback from stakeholders that the broad scope and varied activities of the program has caused confusion and implementation problems at the local level because of ambiguity surrounding the key objectives of the program.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Consistent with measures established under the job training common measures framework, the Department is working to develop several long-term indicators and performance targets that are tied to short term goals and are consistent with the program's scope and activities.

Evidence:  

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: Through the common measures matrix, the program has established a limited set of performance indicators designed to measure program performance/progress, including for example, placement in employment, degree attainment, and skill attainment. However, the Department must establish numerical targets and ensure that performance data exists to report on those targets. In addition, any short-term measures (whether the common measures or additional measures) must be linked to long-term goals. To the extent performance targets are set by states, a process should be put in place to ensure that state-defined targets are appropriately rigorous and that a methodology can be developed for aggregating performance data at the national level.

Evidence: The Department has made efforts to establish performance targets for each of the common performance measures. However, data integrity problems have made it difficult to assess past performance or establish a valid baseline. The Department believes that without such information, it is premature to establish performance targets.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: While the program receives regular and timely annual performance information from grantees, the information cannot yet be tied to a strategic planning framework where a limited number of annual performance goals demonstrate progress to achieving long-term goals.

Evidence: Instructions for this question indicate that a "no" is required if the program received a "no" for both questions 1 and 2 of this section.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: Considerable collaboration and coordination occurs at both the Federal level (e.g., with DOL) and at the grantee level (e.g., with WIA title I one-stops)

Evidence:  

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: The National Assessment of Vocational Education (NAVE) is an independent analysis, conducted every 5 years, and tracks appropriate program outcomes. However, the NAVE does not measure the marginal effects that the Federal investment has on state vocational education programs.

Evidence:  

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The program does not have a strategic planning framework where a limited number of annual performance goals demonstrate progress toward achieving long-term goals. Thus, at this time, performance goals are not currently aligned with budget policy.

Evidence: There is limited reliable data informing on critical performance measures. Specifically, educational and employment outcome data are not uniform across states and cannot be aggregated (e.g., states set their own thresholds, states have different definitions for who is a Voc Ed. student).

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The Department has undertaken a process to make strategic planning improvements. This process is being coordinated with the Department's ongoing development of a reauthorization proposal as well as the development of the common measures framework.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: While grantees provide regular and timely performance information for a series of existing performance measures, there are data quality problems that affect the validity and reliability of the data. Moreover, current performance information is not yet linked to a strategic goals framework (see Sec II, q 1 & q 2), nor is it consistent with the common measures at this time.

Evidence: The Department has made progress in using existing performance information to manage the program, e.g., imposing conditions on grantees through requirements for improvement plans. Data quality issues include lack of a uniform definition of who is a Voc Ed. student as well as an inability to use state-level performance data to develop national estimates.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: This program has not instituted an appraisal system that holds Federal managers accountable for grantee performance. However, as part of the President's Management Agenda, the Department is planning to implement an agency-wide system -- EDPAS -- that links employee performance to progress on strategic planning goals. Grantee performance is monitored on an annual basis through review and approval of annual budget plans, compliance reviews, audits, and site visits. Grantees that do not meet Federal requirements are required to submit improvement plans and can have grants reduced or discontinued for serious or persistent failures to comply.

Evidence:  

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated within the timeframes set out by Department schedules and used for the purposes intended.

Evidence:  

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has not yet instituted procedures to measure and improve cost efficiency in program execution. However, as part of the President's Management Agenda, the Department is implementing an agency-wide initiative to re-evaluate the efficiency of every significant business function, including the development of unit measures and the consideration of competitive sourcing and IT improvements.

Evidence: The common measures framework includes an efficiency measure -- cost per participant. The Department estimates that the cost per participant is $102 for high school students and $122 for post-secondary students. However, the lack of valid outcome data makes it impossible to link these costs to the achievement of program goals. Moreover, these figures will need further refinement once ED can establish a uniform definition of a Voc. Ed. participant.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: Education's 2004 Budget satisfies the first part of the question by presenting the anticipated S&E expenditures (including retirement costs) for this program, which constitute less than 1 percent of the program's full costs. However, Education has not satisfied the second part of the question because program performance changes are not identified with changes in funding levels. The program does not have sufficiently valid and reliable performance information to assess the impact of the Federal investment.

Evidence:  

Does the program use strong financial management practices?

Explanation: The program has a positive audit history, with no evidence of internal control weaknesses.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has improved its monitoring process through increased review of grantee budgets and performance as well as taking steps to increase accountability through conditions on state grants.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Department maintains information on grantee activities through consolidated annual reports, site visits and compliance monitoring, and technical assistance activities.

Evidence:  

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The performance reports are annual and widely disseminated. Work needs to be done to both rectify data quality problems and make data quality problems more transparent.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: Consistent with measures established under the job training common measures framework, the Department is working to develop several long-term indicators and performance targets that are tied to short term goals and are consistent with the program's scope and activities.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: Through the common measures matrix, the program has established a limited set of performance indicators designed to measure program impacts, including for example, placement in employment, degree attainment, and skill attainment. However, the Department must establish numerical targets and ensure that performance data exists to report on those targets. In addition, any short-term measures (whether the common measures or additional measures) must be linked to long-term goals.

Evidence: The Department has made considerable efforts to accumulate the necessary data to inform on the common measures. However, ongoing data quality issues make reliable aggregation of state performance data impossible. For example, the Department estimates that 37% of postsecondary student participants earn a degree or certificate. However, States have varying definitions of who a "postsecondary student participant" is, raising concerns about the reliability of ED's estimate.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The common measures framework includes an efficiency measure -- cost per participant. The Department estimates that the cost per participant is $102 for high school students and $122 for post-secondary students. However, the lack of valid outcome data makes it impossible to link these costs to the achievement of program goals. Moreover, these figures will need further refinement once ED can establish a uniform definition of a Voc. Ed. participant.

Evidence:  

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: To date, the Department has been unable to provide data that informs on the common measures. NAVE results and individual State performance reports (non-aggregated) indicate that vocational education as currently constituted is not effective in achieving academic and employment outcomes.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The most recent NAVE was released in December, 2002. Historically, the NAVE has provided mixed results in terms of whether program goals are achieved. The 1994 NAVE concluded that vocational education provides little or no measurable advantage for high school students in terms of high school completion, postsecondary enrollment, and academic achievement. Preliminary results from the 2002 NAVE confirm the 1994 findings and find further that substituting vocational courses for academic courses adversely affects student academic achievement and college enrollment. However, the 2002 NAVE did find that taking a high school vocational course (versus taking no vocational courses) may have a positive impact on earnings.

Evidence: 1994, 2002 NAVE.