Detailed Information on the
Even Start Assessment

Support grantees in the delivery of high-quality services through monitoring, by monitoring 16 States during fiscal year 2008.

Work with Congress to eliminate funding for the program due to program ineffectiveness, and redirecting funding to other education programs.

Measure outcomes, such as early literacy skills for children and high school completion for adults, and establish ambitious annual and long-term performance targets. Adjust performance targets based on best available data.

Measure: Percentage of Even Start adults showing significant learning gains on measures of literacy and Even Start LEP adults showing significant learning gains on measures of English language acquisition, as measured by the CASAS and TABE.

Explanation:

Measure: The percentage of Even Start adults with a high school completion goal who earn a high school diploma.

Explanation:This measure replaces the measure, "The percentage of Even Start adults who receive a high school diploma" to focus only on those Even Start adults who were eligible for and aspiring toward a high school completion goal. This change is consistent with guidance the Department gave to States for calculating data for this measure from 2005 on.

Measure: The percentage of Even Start adults with a goal of General Equivalency Diploma (GED) who earn a GED.

Explanation:This measure replaces the measure, "The percentage of Even Start adults who receive a GED" to focus only on those Even Start adults who were eligible for and aspiring toward a GED. This change is consistent with guidance the Department gave to States for calculating data for this measure from 2005 on.

Measure: The percentage of Even Start children entering kindergarten who achieve age-appropriate benchmarks on the Peabody Picture Vocabulary Test-III (Receptive).

Explanation:

Measure: The number of letters Even Start children can identify as measured by the PALS Pre-K Uppercase Letter Naming Subtask.

Explanation:

Measure: Percentage of Even Start parents who show improvement on measures of parental support for children's learning in the home, school environment, or through literacy activities.

Explanation:

Measure: The percentage of Even Start children reading at grade level (Targets under development)

Explanation:No data are available for this measure because a specific grade level was not defined.

Measure: The percentage of preschool-aged children participating in Even Start programs who achieve significant gains in oral language skills as measured by the Peabody Picture Vocabulary Test-III (PPVT-III, Receptive). (New measure, added February 2008)

Explanation:This is a measure for the Early Reading First program that should be added for cross-program comparison.

Measure: Number of days it takes for the Department of Education to send a monitoring report to States after monitoring visits.

Explanation: 

Is the program purpose clear?

Explanation: Purpose is to help break the cycle of poverty and illiteracy for low-income families by integrating early childhood education, adult literacy, and parenting education into a unified family literacy program.

Evidence: Section 1231 of the No Child Left Behind Act of 2001

Does the program address a specific interest, problem or need?

Explanation: About 4% of adults cannot read at all and 21 % have only rudimentary reading and writing skills. 56% of beginning kindergarteners are at risk of school failure because of factors such as low family income and low parent education.

Evidence: 1992 U.S. Department of Education survey and ED's Early Childhood Longitudinal Study, 2000.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: There is no evidence indicating that increases or decreases in Federal funding for this program would have a clear impact on family literacy.

Evidence: Third National Evaluation of Even Start.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: The program is duplicative of several other programs including: Head Start, Adult Education, Early Reading First, Reading First, and Title I of the Elementary and Secondary Education Act (ESEA).

Evidence: Head Start, Early Reading First, and Even Start serve similar early childhood populations; Adult Education and Even Start serve similar adult populations. In Title I and Reading First, family literacy efforts are allowable activities.

Is the program optimally designed to address the interest, problem or need?

Explanation: There is no evidence indicating that the structure of the program -- formula grants to States, competitive grants to the local level -- is the wrong design for the program. This does not mean that program improvements are unnecessary.

Evidence:  

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has two outcome goals for adults and two for children that directly support the program's mission and purpose. However, the program lacks numerical targets for its long-term goals.

Evidence: Even Start indicators of program quality and Section 1240 of the No Child Left Behind Act of 2001.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: The program must set numerical targets for its annual goals and ensure that data exist to report on whether those targets have been met.

Evidence:  

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: SEAs are required to develop indicators of program quality to monitor, evaluate, and improve their programs.

Evidence: While States have begun to implement the statutory requirements to set performance goals around specified measures, they do not fit into a strategic framework since the Department has not established numerical targets for its performance goals. (see Q. 1 and 2 in this section). A process should be put in place to ensure that State goals are rigorous and that would help ensure achievement of national goals set by the Department.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: Program staff at the national, state, and local levels coordinate with Title I of ESEA, Vocational Education, and Head Start programs.

Evidence: Even Start has conducted 2 National Forums jointly with Vocational Education and Head Start programs. The first brought together local teams representing the three programs that wrote action plans for how to promote family literacy. The second culminated in the publication of research papers representing each program.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: Education conducts independent evaluations of this program every 3-5 years.

Evidence:  

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The program does not have a strategic planning framework where a limited number of annual performance goals demonstrate progress toward achieving long-term goals. Thus, at this time, performance goals are not currently aligned with budget policy.

Evidence:  

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: Even Start has developed an action plan addressing the program's long term planning deficiencies.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects annual data through an extensive data collection system and uses it to target technical assistance activities.

Evidence:  

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: This program has not instituted an appraisal system that holds Federal managers accountable for grantee performance. However, as part of the President's Management Agenda, the Department is planning to implement an agency-side system -- EDPAS -- that links employee performance to progress on strategic planning goals. Grantee performance is monitered on an annual basis through review and approval of annual budget plans, compliance reviews, audits, and site visits.

Evidence:  

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Grants for the State formula grant program are obligated on schedule. In addition, evaluation and technical assistance funds are obligated on schedule based on a spending plan. However, the funds for the competitive portion of the program are often not obligated in a way to meet Education's internal schedule, even though they are obligated before the legal deadline.

Evidence:  

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has not yet implemented measures and procedures to improve cost efficiency in program execution. However, as part of the President's Management Agenda, the Department is implementing an agency-wide initiative to re-evaluate the efficiency of every significant business function, including the development of unit measures and the consideration of competitive sourcing and IT improvements.

Evidence:  

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: ED's 04 budget submission satisfies the first part of the question by presenting the anticipated S&E expenditures (including retirement costs) for this program, which constitute 1% percent of the program's full costs. However, ED has not satisfied the second part of the question because program performance changes are not identified with changes in funding levels. The program does not have sufficiently valid and reliable performance information to assess the impact of the Federal investment.

Evidence:  

Does the program use strong financial management practices?

Explanation: Recent agency-wide audits have not identified deficiencies in the financial management of this program.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Material internal management deficiencies have not been identified for this program.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ED collects and reviews extensive summaries of local activities.

Evidence:  

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects and reports annual data through an extensive data collection system, and has published summaries of local evaluations.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: Since targets have not been set, it is not currently possible to assess progress toward meeting them.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: Since targets have not been set, it is not currently possible to assess progress toward meeting them.

Evidence:  

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: This program does not lend itself to the development of efficiency measures that link the Federal investment to program outcomes because it's funds are combined with a significant amount of other program dollars from the Federal, State, and local levels to provide achieve its goals.

Evidence:  

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: No comparable data are available for other programs.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Education has conducted three major evaluations of this program, two including a small experimental design study. None of the studies could show that the parents or children who received these services made greater gains than those who did not. Results from 3 States that have conducted their own evaluations are more positive than the national results, however these evaluations were not as rigorous as the national evaluations.

Evidence: National Evaluations of the Even Start Family Literacy Program.