Detailed Information on the
Institute of Education Sciences Research Assessment

Implement the recommendations of the National Board for Education Sciences' forthcoming independent evaluation of the IES research programs.

Work with other Education offices to increase IES involvement in developing technical assistance materials and guidance on the use of effective research to improve program outcomes in critical areas.

Produce budget requests that specify how IES will use the funds to support the achievement of each of its long-term goals for the Research, Development, and Dissemination program.

Measure: Minimum number of IES-supported interventions on reading or writing that are reported by the What Works Clearinghouse to be effective at improving student outcomes by 2013-2014.

Explanation:The published priorities of IES are organized around a principal goal of developing or identifying a substantial number of programs, practices, policies, and approaches that enhance academic achievement and that can be widely deployed. It follows that an appropriate measure of long-term success would be a reliable count of the number of such education interventions that have been developed or identified by IES. This count, for both the annual and long-term measures on the efficacy of IES-supported interventions, employs the What Works Clearinghouse (WWC) standards of intervention effectiveness. The WWC synthesizes the best evidence of the effectiveness of education programs, policies, and practices, and reports these findings through its website. Because the WWC is concerned with determining the impact of interventions on achievement outcomes, it only admits results from research designs that support strong causal inferences, specifically randomized controlled trials and quasi-experiments in which selection bias and other threats to causal validity have been well-controlled. Further, it frequently solicits additional information from study authors and carries out reanalyses of data in order to obtain results that are comparable across studies and based on appropriate statistical techniques. The WWC employs transparent, rule-based evidence standards, rating schemes, and reporting systems that are published on its website (e.g., http://whatworks.ed.gov/reviewprocess/standards.html; http://whatworks.ed.gov/reviewprocess/rating_scheme.pdf). The ability of the WWC to vet research on program effectiveness using published rules makes WWC reports relatively immune from perceptions that conflicts of interest or political considerations affected the findings. This rule-based system also allows independent replication of the findings of WWC reports. This long-term measure is based on finished work that has been identified by the WWC as fitting the protocol for one of its ongoing topic reviews. It can take many years of research and development, sometimes across independent teams of researchers, to generate interventions that can be adopted by multiple schools and districts and that have been shown to be effective in field trials that employ the research designs that are favored by the WWC. For reading and writing, the long-term goal of 15 interventions that are effective and can be widely deployed is based on an analysis of the current state of the field and the portfolio of grants currently funded by IES. In its review of beginning reading programs and its review of interventions in elementary school for English language learners, the WWC has identified to date only two interventions that have strong evidence of a positive effect on reading and writing outcomes. In this context, the goal of IES researchers generating 15 such programs by 2013 is ambitious but potentially obtainable given that IES has funded 25 grants that are intended to demonstrate the effectiveness of an intervention in reading and writing and the annual measure from 2007 identified 6 interventions that had already produced statistically significant positive findings that met the evidence standards of the WWC.

Measure: Minimum number of IES-supported interventions on mathematics or science education that are reported by the What Works Clearinghouse to be effective at improving student outcomes by 2013-2014.

Explanation:The published priorities of IES are organized around a principal goal of developing or identifying a substantial number of programs, practices, policies, and approaches that enhance academic achievement and that can be widely deployed. It follows that an appropriate measure of long-term success would be a reliable count of the number of such education interventions that have been developed or identified by IES. This count, for both the annual and long-term measures on the efficacy of IES-supported interventions, employs the What Works Clearinghouse (WWC) standards of intervention effectiveness. The WWC synthesizes the best evidence of the effectiveness of education programs, policies, and practices, and reports these findings through its website. Because the WWC is concerned with determining the impact of interventions on achievement outcomes, it only admits results from research designs that support strong causal inferences, specifically randomized controlled trials and quasi-experiments in which selection bias and other threats to causal validity have been well-controlled. Further, it frequently solicits additional information from study authors and carries out re-analyses of data in order to obtain results that are comparable across studies and based on appropriate statistical techniques. The WWC employs transparent, rule-based evidence standards, rating schemes, and reporting systems that are published on its website (e.g., http://whatworks.ed.gov/reviewprocess/standards.html; http://whatworks.ed.gov/reviewprocess/rating_scheme.pdf). The ability of the WWC to vet research on program effectiveness using published rules makes WWC reports relatively immune from perceptions that conflicts of interest or political considerations affected the findings. This rule-based system also allows independent replication of the findings of WWC reports. This long-term measure is based on finished work that has been identified by the WWC as fitting the protocol for one of its ongoing topic reviews. It can take many years of research and development, sometimes across independent teams of researchers, to generate interventions that can be adopted by multiple schools and districts and that have been shown to be effective in field trials that employ the research designs that are favored by the WWC. For mathematics and science education, the long-term goal of 12 interventions that are effective and can be widely deployed is based on an analysis of the current state of the field and the portfolio of grants currently funded by IES. In its review of elementary and middle-school mathematics curricula, the WWC has identified to date only two interventions that have strong evidence of a positive effect on mathematics achievement. Also, the Academic Competitiveness Council (ACC) identified only four STEM education interventions across federal agencies that had been evaluated with randomized controlled trials or quasi-experiments and found to generate positive outcomes (note that the evidence standards of the WWC are more rigorous than those applied by the ACC). In this context, the goal of IES researchers generating 12 such programs by 2013 is ambitious but potentially obtainable given that IES has funded 36 grants that are intended to demonstrate the effectiveness of an intervention in mathematics and science education and the annual measure from 2007 identified 4 interventions that had already produced statistically significant positive findings that met the evidence standards of the WWC.

Measure: Minimum number of IES-supported interventions on teacher quality that are reported by the What Works Clearinghouse to be effective at at enhancing teacher characteristics with demonstrated positive effects on student outcomes by 2013-2014.

Explanation:The published priorities of IES are organized around a principal goal of developing or identifying a substantial number of programs, practices, policies, and approaches that enhance academic achievement and that can be widely deployed. It follows that an appropriate measure of long-term success would be a reliable count of the number of such education interventions that have been developed or identified by IES. This count, for both the annual and long-term measures on the efficacy of IES-supported interventions, employs the What Works Clearinghouse (WWC) standards of intervention effectiveness. The WWC initiates reviews of topics under a protocol that identifies outcomes of interest and the types of interventions to be reviewed. Because the WWC is concerned with determining the impact of interventions on teacher quality outcomes, it only admits results from research designs that support strong causal inferences, specifically randomized controlled trials and quasi-experiments in which selection bias and other threats to causal validity have been well-controlled. Further, it frequently solicits additional information from study authors and carries out re-analyses of data in order to obtain results that are comparable across studies and based on appropriate statistical techniques. The WWC employs transparent, rule-based evidence standards, rating schemes, and reporting systems that are published on its website (e.g., http://whatworks.ed.gov/reviewprocess/standards.html; http://whatworks.ed.gov/reviewprocess/rating_scheme.pdf). The ability of the WWC to vet research on program effectiveness using published rules makes WWC reports relatively immune from perceptions that conflicts of interest or political considerations affected the findings. This rule-based system also allows independent replication of the findings of WWC reports. This long-term measure is based on finished work that has been identified by the WWC as fitting the protocol for one of its ongoing topic reviews. It can take many years of research and development, sometimes across independent teams of researchers, to generate interventions that can be adopted by multiple schools and districts and that have been shown to be effective in field trials that employ the research designs that are favored by the WWC. For teacher quality, the long-term goal of 10 interventions that are effective and can be widely deployed is based on an analysis of the current state of the field and the portfolio of grants currently funded by IES. The hurdle to establishing the effectiveness in raising student achievement of interventions in teacher quality is quite high relative to interventions in reading, mathematics, and science. A reading, mathematics or science intervention is a curriculum or practice that directly engages students. Effects on student achievement can be expected and measured in a matter of months. In contrast, measuring the impact on student achievement of, for example, a professional development program for teachers, requires one or more studies that demonstrate that teacher knowledge and behavior change, followed by studies that examine the impact on students of those changes in teachers. It simply takes longer to do this and the outcomes are less certain because interventions that change teachers do not always change students. In this context, the goal of IES researchers generating 10 effective teacher quality interventions by 2013 is ambitious. This goal is challenging but potentially obtainable given that IES has funded 25 grants that are intended to demonstrate the effectiveness of an intervention in teacher quality and the annual measure from 2007 identified 3 interventions that had already produced statistically significant positive findings that met the evidence standards of the WWC.

Measure: Minimum number of individuals who have completed IES-supported pre- or post-doctoral research training programs and are actively engaged in education research in 2013-2014.

Explanation:To expand the national capacity to carry out research that generates evidence on the effectiveness of education programs and practices, IES is committed to training at least 200 researchers who become actively engaged in conducting education research by 2013-2014. IES also funds individual post-doctoral students, but its major investment is in training grants for pre-doctoral interdisciplinary training programs in the education sciences. The first IES-supported predoctoral programs for research training in the education sciences were developed and began recruiting fellows in 2004. The long-term target of 200 researchers is derived from the number of pre-docs and post-docs that IES intends to fund, adjusted by the expected leakage in terms of students in training who do not finish their degrees or who choose a career other than education research. IES intends to fund approximately 450 such individuals through 2012. Approximately 300 fellows will be in the labor market by 2014 and about two-thirds of those will be actively engaged in education research. Based on figures supplied by the American Psychological Association, 209 Ph.D.'s in educational psychology were awarded nationally in 2000, but only 16 of those doctorate recipients were employed in a research-related career a year later. The long-term target of 200 researchers requires a significantly higher percentage of fellows supported with IES funds to be actively engaged in education research than is produced by existing education research doctoral programs. The IES target is not only ambitious compared to the yield of active education researchers produced by academia currently, but it also represents a significant step toward addressing the capacity shortfall in the education research field. Only 10-15 percent of the research-grant applications submitted to the National Center for Education Research each year are of high enough quality to receive funding. Over the 4-year period from 2003 to 2006, approximately 250 applications were funded, indicating that the national pool of qualified education scientists is limited. If 200 highly trained researchers join the ranks of education researchers by 2013, the overall capacity of the field to conduct high quality education research will have increased significantly.

Measure: The percentage of decision makers surveyed in 2013-2014 who indicate that they consult the What Works Clearinghouse prior to making decision(s) on reading, writing, math, science, or teacher quality interventions.

Explanation:IES measures the long-term success of this program not only by examining the extent to which its program investments produce interventions that are effective in improving student achievement and teacher effectiveness, but by examining the extent to which education decision makers seek evidence regarding the effectiveness of interventions before they adopt or use interventions that affect large numbers of students. In a 1997-98 survey, the percentage of principals in Title I schools who reported that the Department's dissemination vehicles, such as the Comprehensive Assistance Centers, Regional Labs, Parent Information Resource Centers, and ERIC, were very helpful ranged from less than one percent to three percent. IES' goal for the utility of the What Works Clearinghouse is 8 times greater than the baseline data from dissemination vehicles that have had much longer to establish relationships with educators. The recent report by Arthur Levine, Educating Researchers, surveyed 1800 school principals and reported that almost all depended on trade associations and newspapers as their sources of information on research. IES's goal is ambitious in this context. In 2013-2014, IES will conduct a survey of a random sample of education policymakers and administrators to determine whether they consult the What Works Clearinghouse before making decisions about the adoption or implementation of interventions in reading, writing, mathematics, science, and teacher quality.

Measure: The minimum number of IES-supported interventions with evidence of efficacy in improving student outcomes in reading or writing.

Explanation:The data for this annual measure are based on What Works Clearinghouse (WWC) reviews of initial findings on interventions from IES research grants, such as findings that will have been presented as papers at a convention or working papers provided to IES by its grantees. WWC principal investigators will review these initial reports from IES-supported projects in reading or writing education and rate them using the WWC published standards to determine whether the evidence from these research grants meets evidence standards of the WWC and demonstrates a statistically significant positive effect in improving achievement outcomes in reading or writing. The WWC defines "interventions" as "programs (for example, Accelerated Schools), products (for example, a textbook or a particular curriculum), practices (for example, mixed-age grouping), and policies (for example, class size reduction) that can be adopted by multiple schools and districts." It defines "meets evidence standards" as randomized controlled trials that do not have problems with randomization, attrition, or disruption, and regression discontinuity designs that do not have problems with attrition or disruption. The WWC's definition for a "statistically significant positive effect" is detailed and takes into account the statistical treatment of multiple measures of the outcome of interest and clustering (e.g., classrooms have been randomly assigned to receive or not receive the intervention whereas outcomes are assessed at the level of individual students clustered within those classrooms). Developing reading interventions that will improve reading achievement over and above current reading practices and that can be implemented by regular teachers with high fidelity is a difficult task. We have ample experience showing that when reading interventions are tested in rigorous evaluations that meet the WWC criteria it is more likely that there will be little or no difference between intervention adn comparison groups. It will take time to develop potent reading and writing interventions and then evaluate them to determine if they are in fact more powerful for increasing learning gains than typical instruction. In this context, the annual targets are ambitious.

Measure: The minimum number of IES-supported interventions with evidence of efficacy in improving student outcomes in mathematics or science.

Explanation:The data for this annual measure are based on What Works Clearinghouse (WWC) reviews of initial findings on interventions from IES research grants, such as findings that will have been presented as papers at a convention or working papers provided to IES by its grantees. WWC principal investigators will review these initial reports from IES-supported projects in reading or writing education and rate them using the WWC published standards to determine whether the evidence from these research grants meets evidence standards of the WWC and demonstrates a statistically significant positive effect in improving achievement outcomes in reading or writing.: . The WWC defines "interventions" as "programs (for example, Accelerated Schools), products (for example, a textbook or a particular curriculum), practices (for example, mixed-age grouping), and policies (for example, class size reduction) that can be adopted by multiple schools and districts." It defines "meets evidence standards" as randomized controlled trials that do not have problems with randomization, attrition, or disruption, and regression discontinuity designs that do not have problems with attrition or disruption. The WWC's definition for a "statistically significant positive effect" is detailed and takes into account the statistical treatment of multiple measures of the outcome of interest and clustering (e.g., classrooms have been randomly assigned to receive or not receive the intervention whereas outcomes are assessed at the level of individual students clustered within those classrooms). The WWC review of elementary school math curricula currently identifies one curriculum as showing potentially positive effects; the review of middle school math curricula identifies just 6 curricula as having positive or potentially positive effects. When interventions are evaluated through rigorous evaluations that meet WWC criteria, only a small percentage result in positive effects. Given that the majority of grants that are funded are to develop interventions and a smaller number are designed to evaluate interventions, it will take time to generate interventions that have been rigorously evaluated and obtained positive effects. In this context, the annual targets are ambitious.

Measure: The minimum number of IES-supported interventions with evidence of efficacy in enhancing teacher characteristics with demonstrated positive effects on student outcomes.

Explanation:The data for this annual measure are based on What Works Clearinghouse (WWC) reviews of initial findings on interventions from IES research grants, such as findings that will have been presented as papers at a convention or working papers provided to IES by its grantees. WWC principal investigators will review these initial reports from IES-supported projects in reading or writing education and rate them using the WWC published standards to determine whether the evidence from these research grants meets evidence standards of the WWC and demonstrates a statistically significant positive effect in improving achievement outcomes in reading or writing.: . The WWC defines "interventions" as "programs (for example, Accelerated Schools), products (for example, a textbook or a particular curriculum), practices (for example, mixed-age grouping), and policies (for example, class size reduction) that can be adopted by multiple schools and districts." It defines "meets evidence standards" as randomized controlled trials that do not have problems with randomization, attrition, or disruption, and regression discontinuity designs that do not have problems with attrition or disruption. The WWC's definition for a "statistically significant positive effect" is detailed and takes into account the statistical treatment of multiple measures of the outcome of interest and clustering (e.g., classrooms have been randomly assigned to receive or not receive the intervention whereas outcomes are assessed at the level of individual students clustered within those classrooms). The hurdle to establishing the effectiveness in raising student achievement of interventions in teacher quality is quite high relative to interventions in reading, math, and science. A reading, math, or science intervention is a curriculum or practice that engages students directly. Effects on student achievement can be expected and measured in a matter of months. In contrast, measuring the impact on students achievement of, for example, a professional development program for teachers requires one or more studies that demonstrate that teacher knowledge and behavior change, followed by studies that examine the impact on students of those changes in teachers. It simply takes longer to do this and the outcomes are less certain because interventions that change teachers do not always change students. In this context, the goal of IES researchers meeting the annual targets for teacher quality interventions with positive effects is ambitious.

Measure: The minimum number of individuals who have been or are being trained in IES-funded research training programs.

Explanation:The number of individuals who receive fellowship support as participants in IES-funded pre- and post-doctoral research training programs will be obtained from grantee reports contained in the official grant files.

Measure: The minimum number of graduates of IES-supported research training programs who are employed in research positions.

Explanation:Beginning in 2009, directors of the research training programs will be asked to obtain and provide information to IES about the current employment of the individuals who have completed their programs. Specifically, program graduates will be asked to indicate whether they are employed in research positions.

Measure: The average number of research grants administered per each research scientist employed in the National Center for Education Research.

Explanation:The major function of IES grant making is to award funds for activities that are of high quality and that fit IES' priorities. IES' principal efficiency measure is the ratio of research staff to research grants. In 2001, the Department's predecessor research organization employed 69 staff in its five national research institutes. Those staff administered 89 active research grants. In 2006, 13 staff in the IES National Center for Education Research administered 265 active research grants with support from four staff in the IES Standards and Review and Grants Administration Staff offices. In 2007, 13 staff in the Center administered 417 active grants, again with support from four staff in Standards and Review and Grants Administration.

Is the program purpose clear?

Explanation: The mission of the Institute of Education Sciences is to expand knowledge and provide information on the condition of education, practices that improve academic achievement, and the effectiveness of Federal and other education programs. In particular, its Research, Development and Dissemination program supports rigorous research to find effective strategies that address the nation's most pressing education needs, from early childhood to adult education.

Evidence: Education Sciences Reform Act of 2002

Does the program address a specific and existing problem, interest, or need?

Explanation: The education field needs scientifically based evidence of effectiveness in order to make informed decisions before adopting programs and practices that will affect significant numbers of students. The reauthorizations of the Elementary and Secondary Education Act and the Individuals with Disabilities Education Act both emphasized the need for scientifically based research on education.

Evidence: Education Sciences Reform Act of 2002; Elementary and Secondary Education Act, as amended by the No Child Left Behind Act; and the Individuals with Disabilities Education Act, as amended.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Research on education is supported by other Federal agencies, including the National Science Foundation and the National Institute for Child Health and Human Development; other programs within Education, including the Research in Special Education program; and by private institutions, including foundations and institutions of higher education. However, the Research, Development, and Dissemination program is the only Federal education research program which has the broad mandate to support research on all academic subjects, age ranges and grade levels, and types of students. It is uniquely designed to support the significant investments in research needed to find solutions to the range of problems facing States and districts. The research supported by the Institute of Education Sciences is also unique among Federal agencies because of its focus on developing and implementing classroom applications of research breakthroughs in other areas of science, such as cognition.

Evidence: Education Sciences Reform Act of 2002

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The design of the RD&D authority reflects significant reforms enacted with the passage of the Education Sciences Reform Act (ESRA) that addressed structural flaws in the previous education research authority identified by the National Research Council and others. The previous authorization required that funds be expended through certain mechanisms and on certain topics, regardless of the needs of consumers of education research. The ESRA gives IES greater flexibility to use a variety of funding mechanisms to address pressing research needs, while requiring higher standards for research methodology and peer review that ensure that investments are designed to provide solutions to problems faced by educators. The ESRA also consolidated most of the Department's research and evaluation activities within IES. The program employs a mix of grants and contracts that is consistent with other Federal research agencies. We are not aware of any analyses of other mechanisms for supporting education research that have been conducted for this or other Federal research agencies.

Evidence: Education Sciences Reform Act of 2002; Educational Research Development, Dissemination, and Improvement Act of 1994.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The RD&D program is effectively targeted to ensure that program funds are used to address the documented shortage of rigorous research, development, and dissemination on education issues facing the nation. The ESRA required higher standards for research methodology and peer review to ensure that research and development funds are invested ininvested in rigorous studies and other activities that are designed to produce evidence of effectiveness on topics that are relevant to the needs of classroom educators and policymakers. The program has also reformed its dissemination vehicles to ensure that they provide evidence based information that is responsive to the needs of the education field. These reforms were based on the flaws identified in the previous education research authority by the National Research Council.

Evidence: Education Sciences Reform Act of 2002

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: IES has committed to measuring the long-term performance of the Research, Development and Dissemination program in three ways: 1) the number of education interventions (programs or approaches) developed and/or evaluated with IES funding that are reported by the What Works Clearinghouse (WWC) to be effective; 2) the number of persons who participated in IES-funded pre- and post-doctoral research training programs who are actively engaged in education research, and 3) the proportion of education decision makers who consult the WWC before making decisions on education interventions. The WWC reviews research studies and evidence related to education interventions and reports on the scientific quality of the studies and the nature and strength of the evidence related to the effectiveness of particular interventions. The WWC produces intervention reports on specific interventions, topic reports on classes of interventions, and practice guides that provide specific evidence-based recommendations for education practice. These three sets of measures are particularly important because they reflect the challenges faced by IES to support rigorous and effective research, to develop a higher quality pool of researchers and to ensure that effective research is used by policy makers. The National Board for Education Sciences (NBES), the body which advises IES on its priorities, procedures, and activities, approved these measures and targets in its May 2007 meeting. The 15 NBES members are appointment by the President, with the advice and consent of the Senate.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database; Records of the Standards and Review Office of IES.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: IES is committed to producing or identifying a significant number of effective interventions for improving teacher effectiveness and student achievement in reading, writing, math, and science by 2013-2014. The specific targets are 15 interventions for reading or writing, 12 interventions for math or science, and 10 interventions for teacher quality. IES was established in fiscal year 2003. Developing or identifying almost 40 effective interventions in 10 years is extremely ambitious. This is borne out by an examination of the work of the WWC. To date, the WWC examined over 38,000 education research studies going back 20 years and identified research on only 52 interventions that was of high enough quality to determine whether the intervention studied might be effective in producing the desired outcome(s). Of the 52 interventions that were the subject of research that met WWC standards, only 38 were found to be effective or potentially effective. Another source of evidence to support the ambitiousness of IES' targets is the Coalition for Evidence-based Policy's review of Science, Technology, Engineering, and Mathematics (STEM) education evaluations for the American Competitiveness Council. That review identified only 10 evaluations of STEM programs across all Federal agencies that would meet WWC standards, of which only 4 found meaningful positive effects on educational outcomes. There are clearly too few interventions of demonstrated effectiveness available to educators. To expand the national capacity to carry out research that generates evidence on the effectiveness of education programs and practices, IES is committed to training at least 200 researchers who become actively engaged in conducting education research by 2013-2014. The first IES-supported programs for research training in the education sciences were developed and began recruiting fellows in 2004. Only 10-15 percent of the research-grant applications submitted to the National Center for Education Research each year are of high enough quality to receive funding. Over the 4-year period from 2003 to 2006, approximately 250 applications were funded, indicating that the national pool of qualified education scientists is limited. If 200 highly trained researchers join the ranks of education researchers by 2013, the overall capacity to conduct high quality education research will have increased significantly. IES measures the long-term success of this program not only by examining the extent to which its program investments produce interventions that are effective in improving student achievement and teacher effectiveness, but also by examining the extent to which education decision makers seek evidence regarding the effectiveness of interventions before they adopt or use interventions that affect large numbers of students. In a 1997-98 survey of the percentage of principals in Title I schools who reported that the Department's dissemination vehicles, such as the Comprehensive Assistance Centers, Regional Labs, Parent Information Resource Centers, and ERIC, were very helpful ranged from less than one percent to three percent. IES' goal for the utility of the What Works Clearinghouse is 8 times greater than the baseline data from dissemination vehicles that have had much longer to establish relationships with educators.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database; The IES Plan for Addressing Priorities; School-Level Implementation of Standards-Based Reform: Findings from the Follow-Up Public School Survey of Principals (U.S. Department of Education, 1998); What Works Clearinghouse (http://www.whatworks.ed.gov); Report of the Academic Competitiveness Council (http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf)

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: IES has set annual goals for and annually measures the number of its supported studies in reading, writing, mathematics, science, and teacher quality that meet WWC standards and that are generating preliminary evidence of the effectiveness of the intervention being developed and/or evaluated. The WWC confirms that the studies meet its standards and that evidence of effectiveness has been produced. IES also has established annual goals and collects data each year on the number of pre- and post-doctoral fellows enrolled in the research training programs it supports, and on the numbers of researchers graduating from those programs.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database;

Does the program have baselines and ambitious targets for its annual measures?

Explanation: IES has baseline data and annual targets for its annual measures of the number of effective interventions for improving teacher quality and student achievement in reading, writing, mathematics, and science; increasing the supply of rigorously trained education researchers; and increasing decision makers' desire for evidence and use of the WWC for obtaining information on the effectiveness of education interventions. These targets represent significant annual increases above the baseline for each indicator, especially considering the fact that the first round of grants for each area of inquiry are either just ending now or are not yet complete.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: IES funding announcements require that applicants identify a research goal based on the stage at which their research falls within a logical sequence that includes the development or identification of interventions for which evidence of effectiveness is produced and commit to working toward the long-term goals of IES research programs. IES staff contributions toward the long-term goals of the program are measured annually through their EDPAS performance agreements. IES has been less successful in promoting the coordination, development, and dissemination of scientifically valid research in education within the Department and Federal government, a responsibility authorized under the Education Sciences Reform Act. Its success in this area depends on the cooperation of other offices and agencies, and the Department needs to develop better mechanisms to ensure that IES has an opportunity to review practice guides, technical assistance, and other materials that provide guidance or should provide guidance on the application of scientifically-based research to States and districts.

Evidence: Request for Applications for FY 2008 research competitions (http://ies.ed.gov/funding/pdf/2008305.pdf); Request for Applications for FY 2008 National Research and Development Center competition (http://ies.ed.gov/funding/pdf/2008305_randd.pdf); Request for Applications for FY 2008 Research Training Grants (pre- and post-doctoral programs) competition (http://ies.ed.gov/funding/pdf/2008305_training.pdf).

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The National Board for Education Sciences (NBES) is administering a contract for an evaluation of the effectiveness of IES in carrying out its priorities and mission. In the Statement of Work developed prior to the award of the contract, NBES identified the following evaluation questions that are relevant to the Research, Development, and Dissemination program: Is IES funding research that is policy-relevant and rigorous? Is IES developing or identifying a substantial number of programs, practices, policies, and approaches that enhance academic achievement and that can be widely deployed? Is IES identifying what does not work and what is problematic or inefficient, and thereby encouraging innovation and further research? Is IES gaining fundamental understanding of the processes that underlie variations in the effectiveness of education programs, practices, policies, and approaches? Is IES developing web-based delivery systems that are useful and useable? Are IES's products and services used to improve results for children in the target areas? Is IES raising the quality of research funded or conducted by the Department? Is IES increasing the relevance of the IES research in order to meet needs of IES customers? Is IES increasing the number of scientists from a variety of disciplines that have entered the field of education research and are actively involved in conducting high-quality scientific research in education?

Evidence: Credit for this question, if any, will be determined once NBES provides an approved evaluation plan for this contract.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Although the justifications describe the research programs to be supported under the request, resource needs for a particular competition or research topic are not specified in the budget request and justifications are presented at the program level. Instead of establishing funding levels for particular competitions or research topics in advance, IES believes it is critical to only fund applications that meet its standards for quality even if this means funding fewer grants in a given year and working with the field to enhance the quality of applications in the next year.

Evidence: Congressional Budget Justifications; OCIO Grant Awards Database; IES website

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The National Board for Education Sciences has approved research priorities for IES that are used to inform budget requests and the development of programs of research. These priorities were used to inform the development of the IES Plan for Addressing Long-term Research Priorities, as required under Sec. 133(a)(2) of the Education Sciences Reform Act (ESRA). IES assesses its progress relative to the long-term research priorites annually based on 3 criteria: opportunities; quantity/quality of applications received; and yield in terms of findings. "Opportunities" are the degree to which competitions for research funding are aligned with the priorities. "Quantity/quality" is determined based on the number of applications and the ratings for these applications as determined by expert panels based on their methodological rigor and relevance. Yield is measured through counts of citations of funded research in scholarly publications and in the general media. One of the most significant obstacles for strategic planning at IES is the inadequate capacity of the field to conduct research that uses rigorous methodology to address questions that are relevant to classroom practice. Over the 4-year period from 2003 to 2006, IES only received 250 applications that met the standards for quality and relevance in the ESRA. These applications represent only 10-15 percent of the research-grant applications submitted to the National Center for Education Research. IES had the funds to support more grants but did not receive enough quality applications. The problem is not unique to IES. In a recent report, Educating Researchers, Arthur Levine, former president of Teacher's College of Columbia University, focused on the need for quality education research and on the preparation of the scholars and researchers who conduct it. The report is based on a survey of all deans of U.S. education schools and a representative sample of education school faculty. The report concludes that: "Deans and faculty, even at the highest-ranked schools of education, persistently complained that their doctoral curriculums did not equip students sufficiently for the dissertation??. Almost half of education school doctoral recipients thought their curriculum lacked rigor??. To expand the national capacity to carry out research that generates evidence on the effectiveness of education programs and practices, IES supports 10 interdisciplinary predoctoral training programs that include departments such as psychology, political science, economics, education, and epidemiology and are providing intensive training in education research and statistics to about 160 pre-doctoral graduate student. The IES pre-doctoral training programs are attracting a much more qualified pool of graduate students as evidenced by the fact that the combined verbal and quantitative GRE scores of students supported by the IES pre-doctoral programs is 1341 compared to an average score for doctoral students in the schools of education that house IES pre-doctoral training programs of 1196. The report by Arthur Levine cited the education research program at Vanderbilt University, one of the IES pre-doctoral grant programs, as an exemplar of excellence.

Evidence: IES priorities (http://ies.ed.gov/director/board/priorities.asp); IES Plan for Addressing Long-term Research Priorities; Arthur Levine, "Educating Researchers", The Education Schools Project, Washington, DC, April 2007, http://www.edschools.org/EducatingResearchers/

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Coalition for Evidence-based Policy recently reviewed Science, Technology, Engineering, and Mathematics (STEM) education evaluations for the American Competitiveness Council. That review identified only 10 evaluations of STEM programs across all Federal agencies that would meet WWC standards, of which only 4 found meaningful positive effects on educational outcomes. In contrast, IES has set a target of producing 3 mathematics or science evaluations with evidence of efficiency that meets WWC standards in 2007, nearly matching the total number produced by all other agencies to date despite the fact that IES awarded its first grants in this area in 2003.

Evidence: Report of the Academic Competitiveness Council (http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf)

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The National Board for Education Sciences has approved research priorities for IES that are used to inform budget requests and the development of programs of research. Funding decisions are based on the guidance of expert panels of peer reviewers.

Evidence: IES priorities (http://ies.ed.gov/director/board/priorities.asp)

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: All grantees are required to submit annual performance data that document their progress in meeting the research, financial, and administrative requirements of their grant. The Grants Administrative Unit and Program Officers use these data to identify and resolve problems. In addition, IES uses expert panels to review the relevance and quality of investments across its research portfolio. These data are used to improve the management of the program and inform future research competitions.

Evidence: IES: Guide to Discretionary Grants Making; Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to the EDPAS system, which links employee performance to success in meeting the goals of IES and specific programs within IES. In general, managers are provided individual performance agreements where they are responsible for action steps to achieve these goals. These action steps and other items included in managers' performance agreements are designed to measure the degree to which a manager contributes to improving program performance. In addition, grantees are required to submit annual financial and performance reports which are subject to IES review and approval.

Evidence: EDPAS criteria; Annual reports in IES Grants Administration office

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: RD&D funds are available for two years, but IES successfully obligates virtually all of these funds before the end of the first year of availability in accordance with its spending plan and the Congressional budget justifications. Funds are spent for the intended purposes and actual expenditures are consistent with planned expenditures, as assessed through grant and contract monitoring. No improper uses of funds have been identified.

Evidence: Contract and grant files

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: IES uses contract and grant competitions to award all funds for research and dissemination. These competitive mechanisms allow for competitive sourcing and cost comparisons. In addition, IES has created and filled a position of Chief Information Technology Officer and resourced that office to generate intranet and extranet applications that facilitate greater efficiency and effectiveness in serving staff and customers. Examples of IT efficiencies are the recent rollout of a software application to track all equipment, replacing a paper system that occupied the full time of a staff member; the development of an intranet application to track and record all answers to external queries, e.g., press queries, about IES programs of work or reports, thereby creating efficiencies by not having to replicate background research for questions that occur more than once; and the development of a sophisticated paperless electronic system for receipt and review of grant applications. In addition, all web functions at IES are being centralized on NCES servers using active server pages which allow information such as staff lists to be generated from an underlying database rather than coded in static HTML. IES has also established strong controls over external travel with a system of budgeting with centers and an approval process for travel that requires strong justification and multiple levels of supervisor approval connected to forms which require justification of in terms of the principal goals and mission of IES. IES has substantially reduced staff from the levels of the predecessor organization, OERI, while substantially increasing its number of grants and grant competitions.

Evidence: Published RFPs and RFAs; EDPAS agreements for the Chief Technology Office and the Deputy Director for Policy and Administration; Functioning web applications

Does the program collaborate and coordinate effectively with related programs?

Explanation: IES collaborates with other Federal agencies on the following projects: 1) with NICHD on the Developing English Literacy in Spanish Speaking Children (DELSS) initiative; 2) with NICHD and HHS in the Good Start Grow Smart agenda around early childhood research and policy; and 3) with the NICHD, the Department's Office of English Language Acquisition, and the National Institute for Literacy in meetings to discuss research agendas for English Language Learners. These meetings also included the International Reading Association, National Association of Bilingual Education, and Teachers of English to Speakers of Other Languages. IES has also collaborated with private and non-profit organizations on the following projects: 1) with the W.T. Grant Foundation in identifying IES-funded researchers who could benefit from private funding to expand their program of work; 2) with the International Reading Association in a series of meetings on developing research standards; 3) with the American Educational Research Association on a graduate fellowship program; 4) with the American Psychological Association on a post-doctoral training program; 5) with the Council of the Great City Schools on priorities for research programs on urban education; 6) with the National Council of State Legislatures on the dissemination of research findings. IES also collaborates with international organizations on research issues, in particular the OECD Education Committee and the OECD Centre for Educational Research and Innovation.

Evidence: IES press releases; http://www.cal.org/delss/; http://www.aera.net/fellowships/Default.aspx?menu_id=48&id=87;http://www.apa.org/releases/pert.html; http://www.cgcs.org/research/current_projects.aspx.

Does the program use strong financial management practices?

Explanation: The Grants Administration Staff track payments and expenditures to ensure that funds are used in compliance with the grantee's approved project plan. For multiyear projects, the Grants Administration Staff work with program officers with research expertise to ensure that fiscal and performance requirements are met before continuation funding is awarded. No internal control problems have been identified in audit reports.

Evidence: IES: Guide to Discretionary Grants Making.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Since the creation of IES in 2003, the structure and management of the RD&D program has been significantly reformed. Administrative functions have been consolidated in the Grants Administration Staff to strengthen fiscal and performance oversight and allow program officers to focus on research issues. IES has established more stringent criteria for supplemental funding and no-cost extensions to ensure that grantees are completing work in a timely fashion and that additional funding is only awarded where it is warranted by scientific merit. More importantly, IES has employed highly qualified scientists to serve as program officers overseeing the substantive and scientific aspects of its grantee research activities. These program officers also interact with the field to improve the quality of applications/research proposals in annual research competitions. Previous evaluations found that the peer review process used by IES' predecessor agency was not sufficiently rigorous. Based on these evaluations, IES has completely reformed its peer review process to ensure that funding decisions are based on the advice of outside panels of qualified scientists.

Evidence: IES: Guide to Discretionary Grants Making, Memorandum of Understanding between IES and the U.S. Army Medical Research Acquisition Activity to contract for scientific peer review management and administrative support. Strengthening the Standards: Recommendations for OERI Peer Review," August and Associates, 1998; "Strengthening Peer Review in Federal Agencies That Support Education Research" National Academies Press, 2005.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All research grant funds distributed by IES are awarded through competitive processes that include independent merit review and ranking of applications. IES conducts significant outreach both within and beyond the education research community to encourage applications from those who have not previously applied for or received funding. In addition, program officers provide technical assistance to researchers to help them strengthen their applications. These program officers are not involved in the grant award process, so this technical assistance can be offered without compromising the competitive process. In addition to the annual competitions announced in the Federal Register, the IES website clearly states the procedures that are used to consider applications for research and related activities that address topics not included in the annual competitions announced in the FR. Under these procedures, the applications are subject to external peer review and the process is regarded as competitive. Contract funds are also awarded through open competitive processes.

Evidence: Grant files, Federal Register notices inviting applications, and requests for applications published on the IES website

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: IES has consolidated grant management activities into a single office. In addition, the office has developed a grants administration and management handbook that outlines the different responsibilities of grants officers, program officers, and managers in providing oversight of grantee activities and performance. Program officers oversee the scientific and substantive aspects of grantee activities and performance, and grants officers oversee fiscal and management operations. Any requests for changes are reviewed and approved by both grants and program officers.

Evidence: IES: Guide to Discretionary Grants Making

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: IES collects grantee performance data on an annual basis and updates the structured abstracts posted on its website to include publications, which can be accessed for information on findings. IES encourages grantees to make their findings available to the public by providing articles to ERIC 6 months after publication in a journal. This encouragement is consistent with current NIH policy. However, if NIH decides to make such sharing mandatory in the future, IES will follow suit. In introducing structured abstracts into ERIC (http://eric.ed.gov/ERICWebPortal/resources/html/news/eric_news_35.html) and using them for all of its own reports and research grant summaries, IES has assumed a leadership role in making research information available to the public.

Evidence: ERIC (http://eric.ed.gov/ERICWebPortal/resources/html/news/eric_news_35.html); Links to all research projects supported by the National Center for Education Research (http://ies.ed.gov/ncer/projects/).

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: In addition to competitive grants, IES uses contracts to support dissemination services under the RD&D program, including the operation of the What Works Clearinghouse, the Education Resources Information Center and various subscription and operation functions related to the National Library of Education. These contracts and purchasing agreements are competitively awarded.

Evidence: Contract files.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Achievement of IES' long-term goals is closely linked to its performance on its annual goals. For example, IES' long-term goals for IES-supported interventions that have been shown to be effective in improving student achievement require interventions with evidence of efficacy that have been adopted in multiple schools and districts and that have been shown to have positive effects in multiple studies. Toward this end, IES has established annual targets for a minimum number of IES-supported interventions with evidence of efficacy. The targets for the annual performance measures cumulate to a substantially larger number of interventions than the target for the long-term measure because the interventions that are counted in the long-term measure will be a subset of those counted annually. Annual targets include interventions that may not yet have been scaled-up to permit wide adoption and that may be based on single rather than multiple studies of efficacy. More than one round of grant-making is typically required to generate interventions that can be widely deployed. Further, the yield of scaled-up, effective interventions from the base of early-stage interventions with evidence of efficacy is far from certain. IES will know that it needs to redesign its processes to achieve its long-term goals if it is not hitting its annual targets. Data on the aligned annual measures indicate that IES has made adequate progress toward achieving its long-term goals. IES awarded its first cohort of grants in 2003 and these grants are producing impressive results. In the areas of reading and writing, mathematics and science, and teacher quality, the first cohort of grants has produced 13 interventions that meet the evidence standards of the What Works Clearinghouse and produce statistically significant positive effects. This includes 6 interventions in reading and writing, 4 in math and science, and 3 in teacher quality. Some of these research findings have already been reported in high-profile publications. For example, IES-supported research on the effects of temporal spacing of practice problems on learning of mathematics has been profiled in a cover story in Psychological Science, the flagship research journal of the Association for Psychological Science. Another team of IES-supported researchers has published in Science its positive findings from a randomized controlled trial of an algorithm-guided method for use by teachers in individualizing beginning reading instruction. IES has also demonstrated progress toward its long-term goals relating to increasing the number of individuals engaged in education research. As of 2007, IES supported 176 fellows through its interdisciplinary pre-and post-doctoral training programs in leading research institutions. Graduate entrance exam scores indicate that the pre-doctoral training programs are already attracting a much more qualified pool of graduate students to the field of education research. The combined verbal and quantitative scores for students supported by the IES pre-doctoral training program is 1341 compared to an average score for doctoral programs in the schools of education that house IES pre-doctoral training programs of 1196. A recent report, Educating Researchers, by Arthur Levine, the former president of Teacher's College of Columbia University, focuses on the dearth of rigorous education research training programs and cites the IES-supported training program at Vanderbilt University as an exemplar of excellence in the field. The fact that IES exceeded its 2007 target for the number of fellows being trained, and the finding that supported fellows have already contributed to nearly 200 research presentations at conferences across the country, indicate that IES is on track to produce 200 fellows who are actively engaged in education research by 2013-14.

Evidence: Program performance information.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The education research program exceeded its performance goals for 2007. The annual performance measures are aligned with and organized to monitor progress towards the long-term goals.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The major function of IES grant making is to award funds for activities that are of high quality and that fit IES' priorities. IES' principal efficiency measure is the ratio of research staff to research grants. In 2001, the Department's predecessor research organization employed 69 staff in its five national research institutes. Those staff administered 89 active research grants. In 2006, 13 staff in the IES National Center for Education Research administered 265 active research grants with support from four staff in the IES Standards and Review and Grants Administration Staff offices.

Evidence: Grant office files.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Coalition for Evidence-Based Policy reviewed evaluations of Federal STEM education interventions according to the Academic Competitive Council's (ACC) Hierarchy of Study Designs. (This review did not include IES research and development projects.) Across the Federal government, the Coalition found 10 rigorous impact evaluations that have produced at least preliminary findings. Of these 10, only four found the intervention being evaluated to have produced positive effects. In addition, the Coalition found 15 rigorous impact evaluations that are underway but have yet to report results. Everything else was either not rigorous (65 evaluations) or not an evaluation of impact (26 evaluations). In contrast, 13 rigorous IES studies have already produced evidence of positive effects and IES estimates that 23 additional studies will have produced such evidence by 2008. The 13 IES studies with strong designs and positive results were produced under an annual appropriation of about $162 million, contrasted with approximately $2 billion of annual investments across the Federal government in programs that produced four studies with rigorous designs and positive impacts. All of the IES studies are randomized trials that fall at the top of the hierarchy of research designs identified by the ACC, whereas the four studies identified by the ACC as having rigorous and positive impacts included quasi-experimental designs that would have met the WWC standards with reservations, and some employed flawed statistical approaches that would have required the WWC to re-analyze the data.

Evidence: Report of the Academic Competitiveness Council (http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf)

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In July 2007, the National Board for Education Sciences selected a contractor to conduct an independent evaluation of IES, focusing in particular on IES' research and evaluation functions. It will also review the standards used by the What Works Clearinghouse to evaluate program effectiveness, comparing the clearinghouse to similar standards-based functions in other government agencies. The Board is expecting the evaluation to be completed by May 2008.

Evidence: National Board of Education Sciences contract.