Detailed Information on the
Comprehensive School Reform Assessment

Working with Congress to redirect program funding to the Title I Grants for Local Educational Agencies program to reduce program duplication and administrative burden. Redirecting the CSR funds to Title I will allow troubled schools to carry out comprehensive reform without the extra administrative burden of applying to a separate grant program.

Measure: The percentage of CSR schools that have or have had a CSR grant and made AYP in mathematics.

Explanation:

Measure: The percentage of Comprehensive School Reform (CSR) schools that have or have had a CSR grant and made adequate yearly progress (AYP) in reading/language arts.

Explanation:

Is the program purpose clear?

Explanation: The purpose of the program is to improve student achievement by supporting the implementation of comprehensive school reform, especially in low-performing, high poverty schools

Evidence: Statutory purpose: "to provide incentives for schools to undertake comprehensive school reform based upon scientifically based research and effective practices" (Section 1601 of the No Child Left Behind Act of 2001).

Does the program address a specific interest, problem or need?

Explanation: With increasing numbers of schools being identified as in need of improvement, this program addresses a relevant and clearly defined problem

Evidence: Currently nearly 8,700 schools nationwide have been identified as in need of improvement.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: Funds are for start-up costs of implementing comprehensive reform at the school level. Therefore this program provides an organizing framework to improve the use of all other State and local dollars in the school.

Evidence: Early findings from the National Longitudinal Survey of Schools indicate that CSR may be helping to leverage Title I funds to undertake strategies associated with successful schools

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: CSR is duplicative of Title I Part A of the Elementary and Secondary Education Act. This program supports comprehensive school reform, which is also the purpose of Title I schoolwide programs, and helps improve low-performing schools, which is the purpose of the State school improvement set-aside in Title I.

Evidence: Title I schoolwide project statutory provisions (sec. 1114 of the No Child Left Behind Act of 2001) and State school improvement set-aside (sec. 1003 of the No Child Left Behind Act of 2001).

Is the program optimally designed to address the interest, problem or need?

Explanation: The program design provides for formula distribution to States who then compete the funds, giving priority to lowest-performing schools which have assurance of district support for reform. No evidence indicates there is a better design for the program. This does not mean that program improvements are not needed.

Evidence: Section 1604(c)(1) of the No Child Left Behind Act of 2001 requires SEAs to give priority to applications that plan to use program funds in schools identified for improvement or corrective action under section 1116 of the No Child Left Behind Act of 2001.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has two long-term performance goals: (1) by 2014 all students in schools that have received CSR funding will meet or exceed proficiency on State assessments in reading and mathematics; (2) by 2014 no schools that have received CSR funds will be designated as in needs of improvement.

Evidence: GPRA performance report

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: (1) The percentage of students in schools that have received CSR funds who meet or exceed proficiency on State assessments in Reading and Math will increase by 2% annually (3% in Reading at High School). (2) the number of schools that have received CSR funds designated as in need of improvement will decrease by 2.5% annually. Education collects data from States on the performance measures.

Evidence: GPRA performance report

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: In its consolidated application each State describes how it will measure the extent to which the reforms have resulted in increased student achievement; subgrant process gives priority to schools in need of improvement.

Evidence: Annual consolidated performance report from each State provides data that addresses program outcome indicators; statute requires that State conduct program evaluations and share them with the Department

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: The program collaborates with related programs and technical assistance providers that target low-performing schools in need of improvement. These programs share similar school improvement and student achievement goals. The program also works with Education's Institute of Education Sciences to evaluate whole school reform models.

Evidence: CSR co-sponsors technical assistance initiatives with student achievement and school accountability program for State staff related to both programs. Activities include co-presentations and shared publications. The program also partners with regional educational labs on technical assistance and product creation.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: Statute requires national evaluation and a report to Congress; the Department may reserve up to 1% of the amount appropriated each year for evaluation activities. The statute requires each State to evaluate annually the implementation of reforms and measure the extent to which reforms have resulted in increased student achievement. These evaluations must be submitted to the Department.The statute also requires that each LEA evaluate the implementation of comprehensive reforms and measure the results achieved.

Evidence: National Longitudinal Study of Schools (NLSS); Field-Focused Study; Longitudinal Assessment of comprehensive School Reform Implementation and Outcomes (LACIO)

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The database on CSR grantees identifies the number of awards made, the actual and average amount of awards, and the number of awards made to low-performing schools. However, the Department has not determined a direct relationship between funding levels and performance goals.

Evidence:  

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The Department has not identified any strategic planning deficiencies related to this program.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: States report information annually to the Department and provide data on grantees to a contractor for inclusion in a grantee database. Performance information provided by SEAs is used to shape technical assistance provided by program office. Program guidance is key to program management and improved performance.

Evidence: The program collects information about subgranting procedures, timelines, priorities. It also collects data from States through annual consolidated report. States are now required to submit annual subgrantee evaluations to the Department of Education.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: The Department of Education has not instituted an appraisal system that holds Federal managers accountable for grantee performance. However, as part of the President's Management Agenda, the Department is planning to implement an agency-wide system -- EDPAS -- that links employee performance to progress on strategic planning goals. In that context, the CSR program staff have created seven common performance standards aligned with the Department Strategic Plan goals, objectives and strategies. These standards, along with an individualized work plan, dictate the performance results that will be the basis for each employee's evaluation when that system is fully implemented. If sub-grantees do not make adequate implementation progress annually, continuation funds are withheld. Grantee performance is monitored annually through review and approval of application for funds, compliance reviews and site visits.

Evidence:  

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Federal funds are obligated July 1 as required by law. The program office provides guidance and encourages States to create subgrant competition timelines that allow subgrantees adequate time to obligate funds. States held accountable for performance results.

Evidence:  

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has not yet implemented measures and procedures to improve cost efficiency in program execution. However, as part of the President's Management Agenda, the Department is implementing and agency-wide initiative to re-evaluate the efficiency of every significant business function, including the development of unit measures and the consideration of competitive sourcing and IT improvements.

Evidence:  

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: Education's 2004 Budget satisfies the first part of the question by presenting the anticipated S&E expenditures (including retirement costs) for this program, which constitute 1% percent of the program's full costs. However, ED has not satisfied the second part of the question because program performance changes are not identified with changes in funding levels.

Evidence:  

Does the program use strong financial management practices?

Explanation: Recent agency-wide audits have not identified deficiencies in the financial management of this program; the program follows the Departmental guidelines for financial management

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Material internal management deficiencies have not been identified for this program

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Reporting system is in place that documents grantees' distribution of funds to subgrantees. Program guidance available. State program coordinators maintain contact with program office through regularly scheduled outreach meetings and communication.

Evidence: SEDL database tracks subgrant and funding distribution by school; annual evaluation submitted by States.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The Department collects data annually from States, including whether the number of funded schools that have been identified for improvement has decreased.

Evidence: Consolidated performance report; report to Congress (years one and three); GPRA indicators; grantee database; required submission of States' evaluation

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: Performance information shows improvements in elementary school, but mixed results in middle and high schools. However, these data are self-reported, are based on responses from 26 states, and are not nationally representative. They therefore should be considered to be only a preliminary measure of the progress of CSR grantees.

Evidence: Consolidated state reports

Does the program (including program partners) achieve its annual performance goals?

Explanation: Performance information shows improvements in elementary school, but mixed results in middle and high schools. However, these data are self-reported, are based on responses from 26 states, and are not nationally representative. They therefore should be considered to be only a preliminary measure of the progress of CSR grantees.

Evidence: Consolidated state reports.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: This program does not lend itself to the development of efficiency measures that link the Federal investment to program outcomes because it's combined with a significant amount of other program dollars from the Federal, State, and local levels to achieve its goals.

Evidence: SHOULD BE NA

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: No comparable data are available for other programs.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The Department of Education evaluations of both comprehensive reform models and of this program are incomplete. Early implementation data show that the program seems to be helping to catalyzes some changes in how States think about and support school improvement efforts. However, little rigorous evaluation of evidence is available to document that comprehensive school reforms are effective interventions for improving student achievement. One study found that only 3 of 24 comprehensive approaches met the criteria for having "strong evidence of positive effects on student achievement" while another study found that 3 of the 29 most commonly used comprehensive reform models had the "strongest evidence of effectiveness."

Evidence: Department of Education's NLSS (National Longitudinal Survey of Schools), consolidated State reports, Longitudinal Assessment of Comprehensive School Reform Implementation and Outcomes (data collection has not yet begun); An Educators' Guide to Schoolwide Reform (1999; American Institutes for Research); Comprehensive School Reform and Student Achievement: A Meta-Analysis (2002; Center for Reserach on the Education of Students Placed At Risk)