Detailed Information on the
Mathematics and Science Partnerships Assessment

Establish baseline data and ambitious targets for the program's performance measures.

Post program performance data on the program's Web page.

Revise the program's protocol for State monitoring.

Establish baseline data and ambitious targets for the program's efficiency measures.

Measure: The percentage of MSP teachers who significantly increase their content knowledge, as reflected in project-level pre- and post-assessments

Explanation:

Measure: The percentage of students in classrooms of MSP teachers who score at the basic level or above in State assessments of mathematics or science

Explanation:

Measure: The percentage of students in classrooms of MSP teachers who score at the proficient level or above in State assessments of mathematics or science

Explanation:

Measure: The percentage of MSP projects that report using an experimental or quasi-experimental design for their evaluations.

Explanation:

Measure: The percentage of MSP projects that use an experimental or quasi-experimental design for their evaluations that are conducted successfully and that yield scientifically valid results

Explanation:

Measure: The percentage of SEAs that submit complete and accurate data on MSP performance measures in a timely manner

Explanation:

Is the program purpose clear?

Explanation: The purpose of the Department of Education's (ED) Mathematics and Science Partnerships (MSP) program is to improve the academic achievement of students in mathematics and science by encouraging State educational agencies (SEAs), institutions of higher education (IHEs), local educational agencies (LEAs), elementary schools, and secondary schools to participate in programs that: (1) increase teachers' content knowledge in mathematics and science in order to improve the effectiveness of their instruction; (2) focus on the education of math and science teachers as a career-long process; (3) encourage long-term partnerships between math and science teachers in elementary schools and secondary schools and scientists, mathematicians, and engineers to increase the subject-matter knowledge of math and science teachers; and (4) improve and expand training of math and science teachers, including training such teachers in the effective integration of technology into curricula and instruction.

Evidence: Elementary and Secondary Education Act (ESEA) Section 2201(a)

Does the program address a specific and existing problem, interest, or need?

Explanation: The National Assessment of Educational Progress (NAEP) study, Trends in Academic Progress, specified a level of performance aligned with the skills required by a college-prep mathematics course sequence. In 1999, only 40% of the Nation's 17-year olds were able to perform at that level. In addition, the 2003 Program for International Student Assessment (PISA) focused on the performance of 15-year-olds in mathematics literacy and problem solving; the United States ranked 24th out of 29 countries belonging to the Organization for Economic Cooperation and Development, which represents the world's richest countries. Further, many elementary teacher preparation programs only require future teachers to take a course or two in mathematical methods. In the 2003 study, "Looking Inside the Classroom: A Study of K-12 Mathematics and Science Education in the United States" many elementary- and middle-school teachers reported that they do not feel that they have sufficient background in the mathematics they teach.

Evidence: National Assessment of Educational Progress; Program for International Student Assessment; "Looking Inside the Classroom: A Study of K-12 Mathematics and Science Education in the United State," Horizon Research, Inc., Weiss, Pasley, Smith, Banilower, and Heck, May 2003

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: This is the only Federal program that both focuses on improving teachers' knowledge in mathematics and science and that requires local educational agencies (LEAs) and institutions of higher education (IHEs) in all States to form partnerships to improve the quality of math and science teaching. The National Science Foundation (NSF) also administers a Mathematics and Science Partnerships program. However, the two programs complement rather than duplicate each other. NSF's program is primarily funding large research and development projects at approximately $1.5 million to $5 million a year for five years, and the NSF grants are focused primarily on universities, which must be the grant recipients. The Department's MSP program is a formula-grant program to the States, which are awarding much smaller implementation grants for professional development of teachers.

Evidence: Improving Teacher Quality State Grants (ESEA, Title II, Part A); NSF's Mathematics and Science Partnerships program

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence indicating that the structure of the program is a flawed design for the program. The program has a supplement/not supplant provision, which prevents States and localities from using program funds in lieu of their own funds for teacher quality activities. In addition, (1) subgrants are awarded by States on a competitive basis, which should result in higher quality local projects getting funded than would happen through a straight formula allocation; (2) subgrant projects must be based on the results of a comprehensive needs assessment; and (3) activities carried out through the local projects must focus on teacher content knowledge and must be based on a review of scientifically based research. Although the program has no major design flaws, one concern that might be addressed in ESEA reauthorization is that the statute does not carefully focus support on teachers from the most needy schools. The law requires that LEAs that participate in the partnerships be high need. However, because there is no definition of "high-need LEA," most States use a district's level of poverty or low student achievement to determine if it is high need. Because only the district must be high need, and not the participating schools in the district, ED is finding that often teachers from the more advantaged schools in the LEA participate in the program, especially because program participation is often voluntary for teachers. While ED is encouraging LEAs to focus on the most needy schools, there is no legal authority to require LEA cooperation. The program statute also lacks a requirement for subgrantees to conduct rigorous, high-quality evaluations.

Evidence: ESEA Section 2202

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: ED provides MSP grants to States by formula based on the number of children aged 5 to 17 who are from families with incomes below the poverty line, and States then award funds competitively to partnerships. Eligible partnerships must always include an engineering, math, or science department from an IHE and a "high-need LEA" (such as a school district with a high concentration of low-income students, a high percentage of teachers who are not "highly qualified," low student achievement, etc.). All funds are used for activities directly related to the program's purpose. However, as noted in the explanation to question 1.4, the program statute does not specifically target funds to high-need schools.

Evidence: ESEA Sections 2202(a)(1) and 2201(b)(1)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has not yet established a long-term performance measure for this program.

Evidence:

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has not yet established a long-term performance measure for this program, and, therefore, has not set targets.

Evidence:

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ED has established five annual performance measures and one efficiency measure for this program.

Evidence: The five annual performance measures are: (1) The percentage of MSP teachers who significantly increase their content knowledge, as reflected in project-level pre- and post-assessments; (2) The percentage of students in classrooms of MSP teachers who score at the basic level or above in State assessments of mathematics or science; (3) The percentage of students in classrooms of MSP teachers who score at the proficient level or above in State assessments of mathematics or science; (4) The percentage of MSP projects that use an experimental or quasi-experimental design for their evaluations; and (5) The percentage of MSP projects that use an experimental or quasi-experimental design for their evaluations that are conducted successfully and that yield scientifically valid results. The efficiency measure for the program is: The percentage of SEAs that submit complete and accurate data on MSP performance measures in a timely manner.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baseline data should be available in 2007. Targets will be established once baseline data are available.

Evidence:

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ED is working with a contractor to collect and analyze descriptive data annually from partnerships about the program. Because ED is collecting these data for the first time, the data vary in quality to some extent; because of that issue, the contractor is working closely with partnerships to ensure that the data is of high-quality and is available on time. Partnerships also provide a narrative description of their activities annually to ED; these narratives contain information about teacher participation, teacher content knowledge growth, and student learning. Every year, States complete a survey describing their activities to award subgrants and submit a list of their subgrantees and copies of their applications to ED.

Evidence: State plans; subgrant applications

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There is currently no Federal evaluation of the entire program. However, partnerships are required to conduct evaluations and report on these evaluations as part of their annual reports. An example of a high-quality MSP project and evaluation is the Vermont Mathematics Partnership, which builds on work that was started seven years ago to improve mathematics learning in Vermont's elementary schools. It is a partnership of the University of Vermont, the Vermont Department of Education, local school districts, the State Colleges of Vermont and several private colleges and universities in the state. The core work of the partnership offers a three-year masters' degree in mathematics to elementary teachers. The goals for the program are to: (1) build a deep knowledge and understanding of mathematics content; (2) demonstrate effective mathematics instruction; (3) conduct research that informs instructional decisions at the classroom level and beyond; and, (4) provide leadership that supports school-wide improvements in mathematics learning. To date (2006), the project has trained 185 teachers from over 50 school districts, which represents about 85 percent of the school districts in the state. The program is being replicated in Massachusetts, New Mexico, and Nebraska. A formal evaluation of the project was conducted in 2004-2005. Both quantitative and qualitative results confirm that participating teachers have attained a high degree of mathematics content knowledge, that participating teachers have made extensive and important contributions to mathematics teaching and learning in their schools and across the state, and there is significantly higher achievement by students of participating teachers in comparison with students from control groups of teachers across Vermont.

Evidence: ESEA Section 2202(e); descriptive data study

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The request is not aligned with attainment of long-term goals. However, ED has satisfied the second part of the question in its budget submissions show the full cost of the program (including S&E). ED's Fiscal Year 2007 integrated budget and performance plan includes the program's annual and long-term goals.

Evidence:

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has established ambitious annual performance measures, and the results will help ED with strategic planning for the program. ED has hired a contractor to work with MSP projects to improve data collection and reporting issues for the descriptive data study. ED addresses these issues through technical assistance to States and projects through conference calls with States and their subgrantee partnerships, annual meetings, conference calls with the State coordinators, a listserv, annual State data collections, and collecting information about State evaluations and successful project applications.

Evidence: Descriptive data study

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ED is collecting and analyzing descriptive data and narrative information annually from partnerships about the program; the first year's data should be available in 2007. ED has hired a contractor to analyze the data and provide technical assistance to partnerships to ensure that the data are high quality. In addition, States complete a survey describing their activities to award subgrants and submit a list of their subgrantees and copies of their applications to ED every year. The Brookings Institution recently conducted a preliminary analysis of winning MSP proposals from the first year of the program after it became a formula-grant program. Information about the activities planned by successful subgrantee applicants has helped shaped the technical assistance that ED provides to States. ED has also been encouraging partnerships to develop more rigorous evaluations and has developed two guidebooks to help States provide assistance to partnerships. One guidebook helps State coordinators develop a concrete, low-cost strategy to solicit rigorous evaluations of their State's MSP projects and provides advice in three areas: (1) overall evaluation strategy; (2) how to solicit rigorous evaluations; and (3) how to review applicants' evaluation plans and monitor the evaluations once underway. A second guidebook assists partnership project directors and evaluators with clear, practical advice on how to conduct rigorous evaluations of MSP projects at a low cost.

Evidence: Descriptive data and narratives from partnerships, State surveys, State information about funded applications, evaluation guidebooks

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers, in their employee performance plans (part of ED's EDPAS system), are held accountable for achieving tangible indicators of success (such as meeting GPRA performance measure targets) in reaching program goals and supporting the Department's Strategic Plan. In addition, some States have withheld funding, or have stated that they will withhold funding, from partnerships that are not performing adequately.

Evidence: EDPAS agreements, contractor reports, State and partnership reports and descriptive data

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are obligated within the timeframes set out by ED schedules and used for the purposes intended. ED reserves some funds for program evaluation, which are obligated based on an evaluation plan.

Evidence: Evidence indicates that States are drawing funds down at an acceptable rate.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ED recently established an efficiency measure for the program, but has not yet established baselines or targets for the measure.

Evidence: See measures section.

Does the program collaborate and coordinate effectively with related programs?

Explanation: MSP program staff collaborate and coordinate with ED's Title I program and with other ED teacher quality programs, including Improving Teacher Quality State Grants. For example, ED has held joint meetings of the State directors for the MSP and Improving Teacher Quality State Grants programs, and ED has encouraged States to hold joint meetings and has allowed States to conduct joint grant competitions. The National Science Foundation also has a Mathematics and Science Partnerships program, and ED and NSF coordinate on the programs in several ways: (1) ED and NSF staff meet regularly to discuss the programs; (2) the two agencies cosponsor national grantee meetings in order to share resources and discuss ways in which the projects funded under these programs might collaborate; (3) ED's MSP State coordinators have access to restricted areas of NSF's interactive MSP website that makes available activities and results from NSF MSP projects and encourages discussions on issues in mathematics and science education; and (4) some of ED's subgrantees are using models developed by NSF grantees in areas such as teacher testing and professional development models for teachers.

Evidence: Meeting agendas, NSF program website, models developed by NSF -- In addition, ED has developed common performance measures for teacher quality programs.

Does the program use strong financial management practices?

Explanation: Recent agency-wide audits have not identified deficiencies in the financial management of this program. ED works with States to resolve any financial problems States may encounter in managing the program. ED also holds 3 to 4 regional meetings per year to discuss a variety of topics with MSP grantees, including financial issues.

Evidence: State and partnership budgets, ED's Grant Administration and Payment System (GAPS)

Has the program taken meaningful steps to address its management deficiencies?

Explanation: While material internal management deficiencies have not been identified for this program, the program is in frequent contact with States and immediately contacts States if there are problems.

Evidence: Program staff monitor excessive drawdowns of funds to prevent high-risk situations.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ED maintains information on partnership activities through annual reports and technical assistance activities. The program has one national meeting with State coordinators every year and 3 to 4 regional meetings with States and subgrantees. ED has developed a Technical Working Group of 8 to 10 State MSP Directors that discusses program direction and evaluation activities with ED staff. Finally, ED also monitors and maintains ongoing communication with States via telephone calls and electronic mail.

Evidence: Descriptive data study

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: No data are available yet. ED expects to have baseline performance data for the MSP program in 2007.

Evidence:

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: ED has not yet established long-term performance measures for this program.

Evidence:

Does the program (including program partners) achieve its annual performance goals?

Explanation: Data are not yet available, and ED has not yet established targets for the annual performance measures.

Evidence:

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ED has recently developed an efficiency measure for the program, but has not yet established baseline data for it. ED and States are conducting several activities to improve the cost effectiveness of the program. For example, several States are working together on evaluation activities. In addition, NSF is creating tools for assessing teacher content knowledge, evaluation, and professional development; some of ED's MSP subgrantees have participated in field tests of those tools and all ED MSP subgrantees can use them to improve their performance.

Evidence: NSF tools.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No comparable data are available for other programs.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: ED is not conducting an evaluation of this program.

Evidence: