Detailed Information on the
Early Reading First Assessment

Develop a measure of kindergarten readiness and collect baseline data in 2008, for reporting in February 2009.

Conduct targeted technical assistance to improve the quality of phonological awareness and oral language instruction.

Meet with Head Start on a quarterly basis to discuss ways to coordinate assessment measures, instructional practices, and professional development activities across Early Reading First and Head Start.

Collecting data for the new measures: significant gains in early literacy skills, and the cost per preschool-aged child participating in Early Reading First programs who achieves significant gains.

Implementing a measure of kindergarten readiness by requiring entities that receive a grant for a second three-year period to collect former participants' academic achievement in kindergarten.

Analyze and post 2007 grantee performance data on the program's website by May 1, 2008.

Measure: The percent of 4-year-old children participating in Early Reading First programs who achieve significant gains on the Peabody Picture Vocabulary Test-III

Explanation:This measures the development of receptive language, a skill correlated with improved academic performance in kindergarten. The goal is for ERF children to catch up with their peers before entering school. This is the same test used for Head Start and for Education's Early Childhood Longitudinal Study. The baseline was set in 2004. (ED needs to provide info on its definition of "significant ")

Measure: The number of letters ERF children can identify as measured by the PALS-Pre-K Upper Case Alphabet knowledge subtask

Explanation:Letter identification at the beginning of kindergarten has been shown to be correlated with improved academic performance in kindergarten.

Measure: The percentage of preschool-aged children participating in Early Reading First Programs who demonstrate age-appropriate oral language skills as measured by the Peabody Picture Vocabulary Test-III.

Explanation:The purpose of this measure is to assess whether Early Reading First children have caught up with their peers.

Measure: Cost per preschool-aged child participating in Early Reading First programs who achieves significant gains in early literacy.

Explanation:ED will set a baseline in 2006.

Is the program purpose clear?

Explanation: The program authorization states that the purposes of the Early Reading First (ERF) program are: (1) to support local efforts to enhance the early language, literacy, and prereading development of preschool-aged children through strategies and professional development based on scientifically based reading research; (2) to provide preschool-age children with cognitive learning opportunities in high-quality language and literature-rich environments, so that the children can attain the fundamental knowledge and skills necessary for optimal reading development in kindergarten and beyond; (3) to demonstrate language and literacy activities based on scientifically based reading research that supports age-appropriate development; (4) to use screening assessments to effectively identify preschool-age children who may be at risk for reading failure; and (5) to integrate such scientific reading research-based instructional materials and literacy activities with existing programs of preschools, child care agencies and programs, Head Start centers, and family literacy services.

Evidence: Title I, Part B, Subpart 2 of the ESEA, Section 1221.

Does the program address a specific and existing problem, interest, or need?

Explanation: Many preschool programs for low-income children fail to provide the literacy skills necessary for school readiness. Studies show that a high percentage of children from low-income families attend preschools that may successfully address other developmental domains but often fail to provide the language, cognitive, and early reading instruction and activities needed. The Early Reading First program is intended to address the gaps in quality between low-income children and their peers of higher socioeconomic status. The ERF program was designed to focus on early literacy based on recommendations from the National Reading Panel (NRP). NRP reviewed more than 100,000 studies on reading and identified five components essential to a child's ability to learn to read: phonics, phonemic awareness, fluency, vocabulary, and comprehension.

Evidence: Studies, including: "Preventing Reading Difficulties in Young Children" by the National Research Council; "The State of Preschool: 2004 State Preschool Yearbook" by the National Institute for Early Education Research; and "Put Reading First: The Research Building Blocks for Teaching Children to Read" by the National Reading Panel.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Early Reading First is unique in its focus on the implementation of research based early literacy programs. The statute requires individual projects to integrate research based literacy activities in already existing programs of preschools, childcare agencies and programs, Head Start centers, and family literacy services. Therefore, Early Reading First programs function as the early literacy component of other larger programs that address other needs of the program's target population, including federally funded programs that may target the same population - that is, it builds on, rather than duplicates, other early childhood programs.

Evidence: Title I, Part B, Subpart 2 of the ESEA, as amended by NCLB, Section 1221.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program is designed to meet its goals, focusing on efficient implementation by funding already existing preschool programs to add a research based early literacy component. In reauthorization, the statute could be amended to provide support for activities, such as grantee technical assistance, designed to help ensure the fidelity of program implementation.

Evidence:

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The ERF program statute includes application requirements that help ensure targeting of intended purposes and populations, including descriptions of: (1) how the proposed project will enhance the school readiness of preschool-age children in high-quality oral language and literature-rich environments; (2) how the proposed project will provide services and use instructional materials that are based on scientifically based reading research on early language acquisition, prereading activities, and the development of spoken vocabulary skills; and (3) how the proposed project will help staff in the programs to meet more effectively the diverse needs of preschool-age children in the community, including such children with limited English proficiency, disabilities, or other special needs.

Evidence: Title I, Part B, Subpart 2 of the ESEA, as amended by NCLB, Section 1222.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has three long-term measures that focus on outcomes and reflect the program's purposes.

Evidence: The measures are: (1) The percentage of preschool-aged children participating in Early Reading First (ERF) programs who demonstrate age-appropriate oral language skills as measured by the Peabody Picture Vocabulary Test-III; (2) The number of letters ERF children can identify as measured by the PALS Pre-K Upper Case Alphabet Knowledge subtask; and (3) The percentage of preschool-aged children participating in Early Reading First (ERF) programs who achieve significant gains [in receptive language] on the Peabody Picture Vocabulary Test-III.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Department sets targets for this program based on the three-year life of the grant, and therefore on the assumption there will continually be new cohorts of grantees. Language development in young children is slow to respond to instruction and the PPVT is not very sensitive at the lower levels. To take this into account, and to collect pre- and post-intervention data, the Department has added a new measure on literacy gains, not just age-appropriate literacy. The definition of "significant" gains being used by the Department reflects high expectations, and is the same definition used by NCES' Early Childhood Longitudinal Study and the Head Start FACES study. The target for the Alphabet Knowledge Subtask of the PALS Pre-K is ambitious. The Department's original target of 15 letters in 2004 was ambitious as evidenced by the fact the NCES Longitudinal Study reports that the average low-income child enters Kindergarten knowing only 2-3 letters. According to the publisher of the PALS Pre-K assessment, children who recognize 12 to 21 letters at the end of preschool are good readers by the end of first grade.

Evidence: FY 2007 Performance Plan for the Early Reading First program.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has in place three annual performance measures that can demonstrate progress toward the program's goals. The Department has adopted a new efficiency measure that will evaluate the cost per preschool-aged child participating in Early Reading First programs who achieves significant gains in early literacy.

Evidence: The annual performance measures are: (1) The percentage of preschool-aged children participating in Early Reading First (ERF) programs who demonstrate age-appropriate oral language skills as measured by the Peabody Picture Vocabulary Test-III; (2) The number of letters ERF children can identify as measured by the PALS Pre-K Upper Case Alphabet Knowledge subtask; and (3) The percentage of preschool-aged children participating in Early Reading First (ERF) programs who achieve significant gains [in receptive language] on the Peabody Picture Vocabulary Test-III. The efficiency measure is the cost per preschool-aged child participating in Early Reading First programs who achieves significant gains in early literacy.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The Department has set targets for annual improvement over the three-year life of a grant. Language development in young children is slow to respond to instruction and the PPVT is not very sensitive at the lower levels. The definition of "significant" gains being used by the Department reflects high expectations, and is the same definition used by NCES' Early Childhood Longitudinal Study and the Head Start FACES study. The target for the Alphabet Knowledge Subtask of the PALS Pre-K is ambitious. The Department's original target of 15 letters in 2004 was ambitious as evidenced by the fact the NCES Longitudinal Study reports that the average low-income child enters Kindergarten knowing only 2-3 letters.

Evidence: FY 2007 Performance Plan for the Early Reading First program.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Through annual performance reports, desktop monitoring, site visits, and monitoring visits, the Department verifies grantee commitment to working towards the program goal of increasing the early literacy skills of preschool-aged children. Grantees submit complete and accurate data in a timely manner. The program office has awarded a contract to a firm that aggregates and analyses grantee data to ensure its accuracy.

Evidence: The Department determines how well grantees are meeting the program's goals through the annual performance reports and through the evaluation of the Early Reading First program.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Institute of Education Sciences (IES) is conducting a rigorous national evaluation of the ERF program, which is designed to address the following research questions: (1) Does the ERF program improve language and literacy outcomes for program participants compared to children who are enrolled in non-ERF preschools? (2) What social and emotional outcomes do program participants attain relative to children enrolled in non-ERF preschools? (3) How does language and literacy instructional practice differ between ERF preschools and non-ERF preschools? (4) Are variations in language and literacy instructional practice and other preschool characteristics associated with differences in children's outcomes in both ERF and non-ERF preschools? The final report will be available in September 2006.

Evidence: "Status Report to Congress: Early Reading First National Evaluation." January 2005.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program is unable to tie funding levels to the achievement of a specific level of program outcome. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program. ED's 07 integrated budget and performance plan includes the program's annual and long-term goals.

Evidence:

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Based on conversations with ERF program staff and experts in the field, the ERF program modified its performance indicators to make them more meaningful. The program office strongly encouraged the 2002, 2003, and 2004 cohorts to use the GPRA indicators as part of their assessments tools. The Department also provided training for all projects on the administration, scoring, interpretation, and use of the data generated by the measures. The 2005, 2006, and future cohorts are and will be required to report annually on the project's performance against the GPRA indicators.

Evidence: The original indicator of receptive language analyzed the percentage of children who demonstrated age-appropriate development of receptive language. This indicator was not optimal because most of the participating children entered preschool with receptive language development far below what was "age-appropriate," and the Department was not collecting information on the progress of children at the lower levels of development. In order to have longitudinal data the Department kept that indicator, but added another indicator on participants' early literacy gains.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Each grantee is required to submit an annual performance report. This report is submitted 90 days after the end of the annual budget period. Each ERF cohort has a different annual report submission date because they all received funds at different times. The ERF staff ensure the accuracy of data submitted through onsite and desktop monitoring. Documentation of screening assessments, GPRA measures, and progress monitoring instruments are verified during monitoring. ERF grantees are provided written feedback on the completeness of the data included in the annual performance report. The Department also collects performance information from projects in the first year of implementation through site visits conducted by Department consultants using a uniform protocol. The information in the site visit reports is used to provide useful feedback to projects regarding project implementation and the need for improvement and to identify projects that should be monitored by the Department.

Evidence: Site visit reports and follow-up calls.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED managers are held accountable for tracking program performance through the EDPAS system. Managers' performance evaluations include considerations of the degree to which a manager contributes to improving program performance. Grantees are held accountable through their annual performance reports, through which they must report their progress toward targets required by ED. These annual performance reports are reviewed by Federal managers and inform reporting under the Government Performance and Results Act of 1993 (GPRA). Grantees are held accountable for results through their Annual Performance Report (APR), desk monitoring, and monitoring visits conducted by the Department. When Department staff identified serious issues related to the financial management of one project, the Inspector General's office was called and asked to conduct an audit. The audit's results caused the Department to request voluntary project termination.

Evidence: Program manager's EDPAS agreement and grantee Annual Performance Reports.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are obligated within the timeframes set out by Department schedules and used for the intended purposes.

Evidence: The program office staff reviews GAPS system drawdowns to monitor the timely obligation of funds. To ensure that funds are spent for the intended purposes, program office staff then review grantee budgets and communicate with grantees about project implementation.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The ERF program uses webex and webcast technology to communicate with application reviewers and grantees in an effort to provide feedback in an efficient manner to improve the effectiveness of the program. ED uses the e-Reader system (at no cost to the program) for application review, thus eliminating the need for experts to travel to meet in Washington DC. This has resulted in significant savings to the program. For example, the cost of a review with 60 reviewers for five days including travel, lodging, and per diem in Washington DC is $218,000. Conducting conference calls instead costs about $6,000, for a savings of over $210,000. In addition, the program office monitors ERF projects during Title I, Part A and Even Start monitoring trips to save the cost of a separate trip. A risk analysis is used to determine which ERF projects need onsite monitoring as a follow-up to their site reviews, annual performance reports, or correspondence with program staff.

Evidence: Contractor reports.

Does the program collaborate and coordinate effectively with related programs?

Explanation: As members of the Good Start Grow Smart Interagency Work Group, which meets on a monthly basis, Early Reading First staff collaborate and communicate with Head Start and the Child Care Bureau, participate in joint training, and share information about policies and priorities. Grantees are required to coordinate their ERF activities with the Reading First program in the event that the school district receives funding from both ERF and Reading First. Grantees that are Head Start providers or participants in other federally funded programs are also required to coordinate their activities across funding streams. The ERF program staff also coordinate with the Early Childhood Educator Professional Development program, Title I, Part A preschool, and the Even Start Family Literacy program. The program office establishes the same or related priorities for competitions for ERF, ECEPD, and Tribal Even Start and coordinates guidance and policy setting for all three programs. Further, as part of a comprehensive monitoring effort for programs administered in SASA, Early Reading First programs are monitored with Even Start and Title I, Part A programs. This coordinated approach to monitoring is now in its third year of a three-year cycle.

Evidence: Priorities for Early Reading First grant competitions, and grant applications received.

Does the program use strong financial management practices?

Explanation: Agency-wide audits have not identified deficiencies in the financial management of this program. The Early Reading First program provides all of its projects with training on strong financial management practices and the application of the appropriate OMB cost principles via PowerPoint presentations at the New Grantee Meeting and webex technology. The program office also carefully monitors Early Reading First projects for compliance with EDGAR and the applicable OMB cost principles. The Early Reading First program managers also review GAPS drawdown reports on a regular basis to identify possible implementation problems and appropriate financial management. The ERF staff provide technical assistance to projects who have insufficient or excessive drawdowns.

Evidence: One 2003 ERF grantee, identified though the expert consultant site visit process for having slow and inadequate implementation, was asked to voluntarily terminate its project in part because of its poor financial management practices. After the expert consultant flagged the project for slow implementation, the project was monitored by Department staff, placed on high-risk status, and then monitored a second time - in coordination with the Office of Migrant Education whose programs appeared to be having similar problems. When Department staff identified serious issues related to the financial management of the project, the Inspector General's office was called and asked to conduct an audit. The audit's results caused the Department to request voluntary project termination.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: In an effort to be more accountable for the monitoring of the ERF projects and keep the grant file folders properly organized, all oral agreements with grantees are put in writing via email messages. All grantee personnel changes are also immediately placed in the file. These messages provide ED, as well as the grantees, documentation of all oral communication. The program staff monitors excessive and deficient drawdowns of funds by grantees to prevent high-risk situations. The ERF program office communicates to grantees through the use of an online listserv. Project officers provide timely and appropriate technical assistance to meet the individual needs of grantees.

Evidence: Program emails and documentation of listserv postings.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The program statute clearly states the program purposes and priorities, which are reflected in the Federal Register notices when funds for the program are announced. All local educational agencies (LEAs) that are eligible to receive funds under the Reading First program are eligible to apply for ERF grants. Public or private organizations or agencies located in a community served by an eligible LEA may also apply. The awards are made on a competitive basis according to the priorities and criteria stated in the application.

Evidence: Title I, Part B, Subpart 2 of the ESEA, Section 1221.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program office reviews grantee data on an annual basis and conducts systematic and comprehensive monitoring. Top experts in the field visit every project within 6 months of award to assess the extent to which grantees are adhering to the requirements of the program's authorizing statute and other applicable statutes, and ensures compliance with applicable regulations, such as the Education Department General Administrative Regulations (EDGAR). Furthermore, monitoring is conducted to determine the degree to which compliance with NCLB results in educational outcomes for students.

Evidence: By fall 2006, 30 grantees will have received site visits. By fall 2006, 18 grantees will have gone through the monitoring process.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Early Reading First staff collect and compile data from annual performance reports and make those data available to the public via the Department's website. Grantee-level and aggregated performance data and program indicators are available online. The Department is planning to release the first national evaluation report to Congress in fall 2006.

Evidence: http://www.ed.gov/programs/earlyreading/performance.html

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The ERF program has established long-term measures and targets through 2008. The program collected baseline data in 2004, and met or exceeded its targets for measure #1 and measure #2 in 2005. The program has not been in existence long enough to achieve long-term goals. However, because the program met or exceeded its performance goals in 2005, it appears to be on track toward being able to meet its longer-term goals.

Evidence: FY 2006 Performance Report for the Early Reading First program.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has developed three performance measures, and the Department collected baseline data on two measures in 2004. In 2005, the first year where actual performance can be compared to a target, the program significantly exceeded its target for the first measure, and met its target for the second measure.

Evidence: The percentage of preschool-aged children participating in Early Reading First (ERF) programs who demonstrate age-appropriate oral language skills as measured by the Peabody Picture Vocabulary Test-III. 2004: Actual - 56 percent (Baseline); 2005: Actual - 67.9 percent, Target - 57 percent. Measure #2: The number of letters ERF children can identify measured by the PALS Pre-K Upper Case Alphabet Knowledge subtask. 2004: Actual - 15 (Baseline); 2005: Actual - 16, Target - 16.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The ERF program uses webex and webcast technology to communicate with application reviewers and grantees in an effort to provide feedback in an efficient manner to improve the effectiveness of the program. ED uses the e-Reader system (at no cost to the program) for application review, thus eliminating the need for experts to travel to meet in Washington DC. This has resulted in significant savings to the program. For example, the cost of a review with 60 reviewers for five days including travel, lodging, and per diem in Washington DC is $218,000. Conducting conference calls instead costs about $6,000, for a savings of over $210,000. In addition, the program office monitors ERF projects during Title I, Part A and Even Start monitoring trips to save the cost of a separate trip. A risk analysis is used to determine which ERF projects need onsite monitoring as a follow up to their site reviews, annual performance reports, or correspondence with program staff. The Department will report baseline data for the program's efficiency measure in March 2007.

Evidence: Analyses prepared by contractor, and site visit reports.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No data are available for such a comparison. No other federal programs perform a similar function.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Institute of Education Sciences (IES) is conducting a rigorous national evaluation of the ERF program. The study will use comparison group design to assess the impact of ERF by comparing child outcomes and instructional practice in the treatment and comparison groups. To measure the impact of the ERF program on child outcomes, the evaluation team collected data on children in both the treatment and comparison groups at the beginning and end of their pre-kindergarten year (Fall 2004 and Spring 2005). Results will be available in fall 2006. ED will update the response to this question once the results are available.

Evidence: