Detailed Information on the
Mentoring Program Assessment

Provide training and technical assistance to grantees on the design, management, and evaluation of mentoring programs to strengthen their program implementation and thereby improve program outcomes.

Complete the national evaluation of the program.

Develop a publication to disseminate the results of the evaluation nationwide to school districts and community-based organizations.

Revise the GPRA performance measures for the program and continue to collect and report grantee data on outcomes for the program.

Work with Congress to eliminate funding for this duplicative program.

Measure: The percentage of student/mentor matches that are sustained for a period of twelve months. (2004 cohort)

Explanation:

Measure: The percentage of mentored students who demonstrate improvement in core academic subjects as measured by grade point average after twelve months. (2004 cohort)

Explanation:

Measure: The percentage of mentored students who have unexcused absences from school. (2004 cohort)

Explanation:

Measure: The cost per student mentored for each student-mentor match that is sustained for a period of twelve months. (2004 cohort)

Explanation:

Measure: The percentage of student/mentor matches that are sustained for a period of twelve months. (2005 cohort)

Explanation:

Measure: The percentage of mentored students who demonstrate improvement in core academic subjects as measured by grade point average after twelve months. (2005 cohort)

Explanation:

Measure: The percentage of mentored students who have unexcused absences from school. (2005 cohort)

Explanation:

Measure: The cost per student mentored for each student-mentor match that is sustained for a period of twelve months. (2005 cohort)

Explanation:

Measure: The percentage of student/mentor matches that are sustained for a period of nine months. (2007 Cohort) (New measure, added February 2008)

Explanation: 

Measure: The percentage of mentored students who demonstrate improvement in core academic subjects as measured by grade point average after twelve months. (2007 cohort) (New measure, added February 2008)

Explanation: 

Measure: The average number of unexcused absences from school per mentored student. (2007 cohort) (New measure, added February 2008)

Explanation: 

Measure: The cost per student mentored for each student-mentor match that is sustained for a period of nine months. (2007 cohort) (New measure, added February 2008)

Explanation: 

Is the program purpose clear?

Explanation: The purpose of the program is to provide children with the greatest need with the support and guidance of a mentor in order to improve their academic achievement; improve their interpersonal relationships with peers, teachers, other adults, and family members; reduce their dropout rate; and reduce their rates of juvenile delinquency and involvement in gangs. A mentor may be a responsible adult, a postsecondary school student, or a secondary school student who is able to establish a supportive relationship with the child and provide academic assistance and exposure to new experiences that enhance the ability of the child to become a responsible adult.

Evidence: Evidence: Title IV, Part A, Subpart 2, Section 4130 of the Elementary and Secondary Education Act (ESEA) of 1965, as amended by the No Child Left Behind Act of 2001 (Authorizing Legislation). Federal Register Notice, May 28, 2004. Information and Application Procedures for Fiscal Year 2004.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program seeks to address the serious problem of a lack of supportive adults at critical junctures in the lives of children at risk, which may increase the likelihood that these children engage in illegal use of drugs and alcohol, violence, use of dangerous weapons, promiscuous behavior, and other criminal, harmful, and potentially harmful activity. The program's school-based approach helps it target children with the greatest need. Students are referred to services by school staff who believe those students are at high risk for criminal or delinquent activities, academic failure, or dropping out of school, or if they lack positive role models. In addition, because research indicates that the transition from childhood to adolescence is a particularly critical developmental time in a young person's life, the program targets children in grades 4-8. There is tremendous interest in this program. In 2002 (the first year that this program was offered) the Department (ED) received approximately 1,300 applications for funding. During the next competition in 2004, ED received over 1,500 applications.

Evidence: Exposure to risk factors in the relative absence of protective factors dramatically increases the likelihood that a young person will engage in problem behaviors. Children often initiate harmful behaviors, such as using alcohol, tobacco, and other drugs, in the middle school years, and one consequence of this early initiation is that they are more likely to develop future patterns of harmful behavior. According to the Final Report of the White House Task Force for Disadvantaged Youth, while many trends in youth risk factors show improvement, risk-taking behaviors are still among the top causes of adolescent morbidity and mortality. The report states that more than 2.6 million teens use illicit substances each month. 29 percent of youth, ages 12 to 20, reported drinking alcohol in the month prior to the survey. In addition, one-third of high school students report having sexual relations in the previous three months, while 46 percent of high school students have experienced sexual relations, putting themselves at risk for pregnancy, sexually transmitted diseases, and emotional trauma. Over 135,000 unmarried girls under the age of 18 gave birth in 2001, long before they were ready to be responsible, married parents. There is encouraging evidence that mentoring can reduce the incidence of substance use, academic failure, and delinquency among youth and have great influence on their lives. In the largest rigorous evaluation of a mentoring program, Public/Private Ventures (P/PV) conducted an experimental evaluation of eight mentoring programs. The study considered six areas that mentoring might affect: social activities, academic performance, attitudes and behavior, relationships with others, self-concept, and social/cultural enrichment. P/PV found that mentored youth were less likely to engage in violence and drug use and more likely to attend school. Students with mentors also reported greater confidence in their ability to perform academically and had healthier social relationships. [Sources: National Mentoring Partnership (http://www.mentoring.org), Final Report on the White House Task Force on Disadvantaged Youth (http://www.acf.hhs.gov/programs/fysb/content/docs/white_house_task_force.pdf), PP/V Study (http://www.ppv.org/ppv/publications/assets/111_publication.pdf).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Although the program is the primary Federal program that is focused exclusively on school-based mentoring, it is only one of several major national efforts that focus on providing mentors for youth, even in a school setting.

Evidence: The Federal program supports only a small percentage of mentoring services nationally. There are more than 4,200 mentoring programs nationwide, and only 254 projects (about 6 percent) receive mentoring grant funds from the Department. Only about 1 percent of the 3 million youth across the country who are in a formal, one-to-one mentoring relationship with a caring adult do so by participating in a mentoring project that receives funds under this Federal grant program. By comparison, Big Brothers/Big Sisters (which is the largest non-profit mentoring program in the country) has 470 agencies in 5,000 communities and in 2004 served more than 225,000 youth ages five through 18. In addition, a significant number of school-based mentoring programs were in existence prior to the creation of the Department's program and the school-based mentoring approach was increasingly popular. According to a 2004 study by Public/ Private Ventures, the number of school based matches in 1999 was 27,000 and in 2002, 90,000. Also, Safe and Drug-Free Schools and Communities Act formula grant funds may be used for mentoring activities. [Source: ED program grant files; Big Brothers/Big Sisters (http://www.bbbs.org); MENTOR/National Mentoring Partnership (http://www.mentoring.org/leaders/about_mentoring/statistics_and_research.php); Public/Private Ventures, "School-Based Mentoring: A Closer Look" (http://www.ppv.org/ppv/publications/assets/180_publication.pdf).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: This program operates as intended by Congress and there is no evidence indicating that another approach would be more effective or that the structure of the program is seriously flawed. The program operates under a fair and open competitive process and efficiently awards grants to school districts and community-based organizations to support their efforts to provide mentors for children with the greatest need. The program does have some relatively minor design flaws, however. Other than a general statement that programs should expose youth to new experiences that they might not otherwise encounter on their own, the authorizing legislation provides little discussion of the specific content of the planned mentoring experience or of the specific activities and strategies that programs should use to facilitate the intended benefits for youth. In addition, certain requirements in the statute are somewhat contradictory and therefore make it difficult for programs to successfully address them. For example, the statute mandates ED to consider in the making of awards the degree to which programs will serve children in the 4th through 8th grades; however, it also requires the consideration of opportunities for job training and postsecondary education for these same students. The Administration has addressed these and other issues as best as possible through rulemaking.

Evidence: Title IV, Part A, Subpart 2, Section 4130 of the Elementary and Secondary Education Act (ESEA) of 1965, as amended by the No Child Left Behind Act of 2001 (Authorizing Legislation).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Through rulemaking the Department has required that program grants to eligible entities be made only to support school-based mentoring programs and activities that serve children with the greatest need in one or more grades 4 through 8 living in rural areas, high-crime areas, or troubled home environments, or who attend schools with violence problems. Per the program statute, children with the greatest need is defined as children who who lack strong positive role models or are at risk of educational failure, dropping out of school, involvement in criminal or delinquent activities. Because research indicates that the transition from childhood to adolescence is a particularly critical developmental time in a young person's life, the program targets children in grades 4-8. In addition, the program's school-based approach helps it target children with the greatest need. Students are referred to services by school staff who believe those students are at high risk for criminal or delinquent activities, academic failure, or dropping out of school, or if they lack positive role models. Grantees are required to link children with mentors who have been trained, screened using appropriate reference checks, and are interested in working with children with the greatest need. The Department has also, through the competitive process, focused assistance on applicants with superior program design, structure, and management capability and has encouraged partnerships between LEAs or CBOs. CBOs bring their expertise and experience to these partnerships and provide expanded access to mentors, while schools can provide access to students who CBOs may not be able to reach, additional management assistance, and a safe, secure environment for both mentors and youth.

Evidence: Authorizing Legislation, 2004 Mentoring Program Grant Application, and Peer Reviewer Guide.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has established three key performance measures for assessing the effectiveness of the Mentoring Program, all of which directly relate to research-based strategies and effective practice. (1) The percentage of student/mentor matches that are sustained for a period of twelve months. Research indicates that mentoring relationships that last 12 months or more are associated with the most positive benefits for youth. (2) The percentage of mentored students who demonstrate improvement in core academic subjects as measured by grade point average after 12 months. Academic achievement is a key predictor of future socioeconomic status and success. (3) The percentage of mentored students who have unexcused absences from school. There is a recognized link between truancy and delinquency and promoting better attendance increases the likelihood of school success. The grant length is three years; therefore, the program's performance measures serve as both annual and long-term performance measures. These measures constitute the Department's indicators of success for this program.

Evidence: Federal Register Notice, May 28, 2004; 2004 Mentoring Program Grant Application.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Department has established the following key performance targets for assessing the effectiveness of the Mentoring Programs grant program: (1) The percentage of student/mentor matches that are sustained for a period of twelve months will increase 25% by 2007; (2) The percentage of mentored students who demonstrate improvement in core academic subjects as measured by grade point average after 12 months will increase 5% by 2007; (3) The percentage of mentored students who have unexcused absences from school will decrease 10% by 2006 and 30% by 2007. Beginning with the FY 2004 cohort, the Department is collecting data for the above three annual performance measures. For the first two measures, no target is established in the first year after award because grantees will need to have operated for a minimum of 12 months in order to produce any matches that meet the criteria established. Baseline data for these measures became available in fall 2006.

Evidence: Federal Register Notice, May 28, 2004; 2004 Mentoring Program Grant Application; measures tab in the PART; annual performance reports.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program's annual performance measures are the same as the long-term measures. The grant length is three years. Each year, grantees will provide information to the Department on the measures, and ED expects to see improvement among them from year to year and over the life of each cohort of grants.

Evidence: Federal Register Notice, May 28, 2004; 2004 Mentoring Program Grant Application; measures tab in the PART; annual performance reports.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The Department has established the following key performance targets for assessing the effectiveness of the Mentoring Programs grant program: (1) The percentage of student/mentor matches that are sustained for a period of twelve months will increase 25% by 2007; (2) The percentage of mentored students who demonstrate improvement in core academic subjects as measured by grade point average after 12 months will increase 5% by 2007; (3) The percentage of mentored students who have unexcused absences from school will decrease 10% by 2006 and 30% by 2007. Beginning with the FY 2004 cohort (performance data for the 2002 cohort of mentoring grants are not comparable), ED is collecting data for these measures and has established targets for each performance measure.

Evidence: Federal Register Notice, May 28, 2004; 2004 Mentoring Program Grant Application; measures tab in the PART; annual performance reports.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Grantees are required to collect and report data to the Department about progress toward the performance measures established by the Department for this program. Consequently, applicants for a grant under this program are advised to give careful consideration to these outcomes in conceptualizing the design, implementation, and evaluation of their proposed project. After making an award, program officers contact all grant recipients to discuss performance measures, and to ensure that evaluation plans account for these measures. Performance measures are also discussed at a mandatory new grantee meeting. All grant recipients are required to provide annual reports that demonstrate substantial progress toward meeting the performance targets within their program. In addition, the program contractor provides training and technical assistance to grantees aimed at strengthening program design and implementation to improve program outcomes.

Evidence: In 2004, to ensure that all program requirements were met and that only programs of sufficient quality were awarded grants, each application was reviewed according to the selection criteria published in the Federal Register announcement for the program. Grantees are aware of the performance measures, and Office of Safe and Drug Free Schools (OSDFS) has developed guidance to ensure that grantees understand what is required. Training and technical assistance are also provided on the annual reporting requirements and, for the most part, reported data are of good quality. These data are used to measure the grantee's performance and the overall effectiveness of the program. Federal Register Notice, May 28, 2004.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: ED has initiated an independent national evaluation of this program to assess the impact of the program and to provide information for program improvement.

Evidence: ED, through the Institute of Education Sciences, is conducting an independent national evaluation of the Department's Student Mentoring Program in order to assess the impact of the program and to provide information for program improvement. IES has contracted with Abt Associates, an independent research organization, to conduct the evaluation. The impact evaluation uses a randomized control trial with a sample size of approximately 2,600 students. Approximately half of this sample will be randomly assigned to be matched with a mentor. Thirty-three of ED's mentoring grantees are participating in the evaluation. Data are being collected for the student sample on school engagement, academic performance, dropping out, the quality of interpersonal relationships, and involvement with high-risk and delinquent behaviors. To measure program impact, student surveys and student school records will be collected both at baseline and at the end of the school year. Data on the nature of mentoring program services will be collected through a survey of the program grantees and mentors to provide context for the impact findings and to provide program improvement information to OSDFS. Findings will be available in 2008.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Funding levels are not tied to grantees achieving specific performance (outcome) goals. However, the 2006 and 2007 budget requests to phase out the program over two years by funding continuation awards only were based on grantee performance (in terms of grantees making substantial progress toward achieving their identified goals and objectives for their projects) and grantee demonstrated need (for carry-over and continuation funds). Also the Department's budget submissions show the full cost (including S &E) for all programs.

Evidence: Performance monitoring, annual reports.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Beginning in fiscal year 2004, ED added three key performance measures to assess the effectiveness of the program. Grantees are aware of these new measures and are required to provide data related to these measures annually. The Department uses these data in the design, development and implementation of training and technical assistance. Data are also used during performance monitoring to evaluate the performance of a grantee -- a program that reports poor data may also require additional monitoring actions. Also, through rulemaking the Department: (1) in 2002, modified the program to have more impact by focusing exclusively on children with the greatest need in grades 4-8, as research indicates that this is a particularly critical developmental time in a young person's life; (2) in 2004, revised the program requirements to encourage applicants to propose partnerships between LEAs and CBOs for a more collaborative approach would result in diverse and effective mentoring programs rooted in the school community and able to call upon multiple sources of community support; and (3) in 2005, established an absolute priority to focus the program exclusively on school-based mentoring, so that the mentoring activities are closely coordinated with schools, including involving teachers, counselors, and other school staff in the identification and referral of students, and that projects are focused on improved academic achievement, reduced student referrals for disciplinary reasons, increased bonding to school, and positive youth development. Administration policy calls for completing the phase out of the program in 2007, after which the Department would complete and disseminate nationwide the results of the impact evaluation, which will provide the field with information for program improvement.

Evidence: 2004 Mentoring Program Federal Register Notice and Grant Application, Mentoring Program Training and Technical Assistance Contract, National Education Longitudinal Study of 1988.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Timely and credible performance information is collected through annual performance reports and monitoring activities. This information is used by program staff to identify any implementation and performance issues and to develop corrective measures or provide technical assistance to improve program performance. Information collected by the Department from grantees is also used to measure how well the program is meeting prescribed performance indicators. If the annual report discloses poor performance toward meeting the performance targets, then one result may be to place special conditions on the grantee, such as delaying continued funding until grant monitors have some evidence that progress is being made.

Evidence: ED staff regularly monitor each grantee's performance. Information obtained from the sources mentioned above is used to determine continuation funding and measure program effectiveness. Grantees are held accountable for cost, schedule, and performance results through monitoring, and the Department has developed a "watch list" that identifies and records information about grantees that exhibit potentially significant problems. This watch list is used to closely monitor grantees that are experiencing problems that could lead to a high-risk status designation. The three areas of risk that are of greatest concern are: performance, financial, and management. For each of these areas, ED provides both indicators and monitoring actions. This proactive tool helps us address and resolve problems more effectively and efficiently. Performance reports are public records and are freely available to anyone under the Freedom of Information Act.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The program performance agreement for the Assistant Deputy Secretary establishes expectations for this (and other OSDFS programs) to be implemented according to agreed upon schedules. For example, the Assistant Deputy Secretary is accountable for meeting set schedules in awarding both new and continuation awards. The Assistant Deputy Secretary is also accountable for developing a reporting tool to identify and track actions on high-risk grantees and for developing monitoring plans. Performance agreements for individual grant monitors for this program establish standards for grant program implementation, including the development of a monitoring strategy to track and evaluate grantees' progress in achieving results. These agreements also require program monitors to apply appropriate tools (audit reports, site visits, performance reports, and performance calls) to identify and address poor performance or high-risk grantees. Through these activities, grantees are held accountable for cost, schedule, and performance results. This information is used to determine continuation funding, and contributes significantly to Department efforts to strengthen this program by encouraging the development of more comprehensive programs. In addition, OSDFS has developed a "watch list" that identifies and records information about grantees that exhibit potentially significant problems. This watch list is used to closely monitor grantees that are experiencing problems that could lead to a high-risk status designation. The three areas of risk that are of greatest concern are performance, financial, and management. For each of these areas, ED provides both indicators and monitoring actions. This proactive tool helps program managers address and resolve problems more effectively and efficiently.

Evidence: EDPAS agreements, OSDFS Monitoring Plan, Watch List Criteria.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Program funds are routinely obligated in a timely manner (consistent with established schedules). The funds are accurately reported immediately in the Department's Grants Administration and Payment System (GAPS) when the Department records the obligation of funds to each recipient in the form of a grant award notification. A thorough analysis of an applicant's budget is conducted prior to the awarding of funds. During this analysis, staff ensure that budgeted items are necessary, reasonable, and allowable, and support the objectives of the program. Following award of the grant, overall grant expenditures are reported to the Department via GAPS, and staff monitor budget expenditures closely to ensure funds are being used appropriately and for the intended purpose. Although project monitors sometimes identify unallowable expenditures, in general the Department finds that most funds are used for intended purposes.

Evidence: ED financial management system printouts; individual budget analysis worksheets and notes.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ED has taken steps to develop and implement an efficiency measure for this program but has yet to collect baseline data and establish targets. The efficiency measure will look at the cost per student mentored for each student-mentor match that lasts a year. The program also addresses efficiency and cost effectiveness in other ways. Because the grants supported under this program vary significantly in scope and content, it is not feasible to develop a standard cost formula that would be appropriate for all grantees. However, as part of pre-award budget reviews and negotiations, staff use professional judgment and experience to assess cost efficiency. Budget reviews are conducted by a limited number of staff, working as a team in order to ensure a consistent result across applicants. Staff carefully review both direct and indirect cost requests and require applicants to provide proof of negotiated restricted indirect cost rates. As a part of staff review, applicants must exclude "distorting" costs from their request for indirect costs. In addition, to assist with the grant competition, the Department awarded a performance-based contract for grantee conference planning and administrative management. Through this contract, an online peer review process was developed that greatly improved efficiency and resulted in cost savings of approximately 20 percent to 30 percent annually. Application review procedures are reviewed annually and with feedback from both staff and peer reviewers, refinements are made to the on-line system to improve ease of use and to ensure it provides an efficient and effective means of conducting peer reviews. Recent enhancements to the system include the addition of on-line recruitment of peer reviewers and electronic dissemination of technical review forms to all applicants. These new capabilities and others have had a significant impact on ED's ability to recruit qualified reviewers and to perform the tasks necessary during a peer review. In the first year of operation, the acceptance rate for persons agreeing to be peer reviewers went from 10 percent to approximately 90 percent. This has ensured a large pool of highly qualified candidates from which to select peer reviewers and has significantly reduced the cost of recruiting those reviewers. The cost of producing and sending reviewer comments to unsuccessful applicants was also reduced considerably. An additional benefit of forwarding this information electronically is that applicants receive their comments more quickly. The Department also awarded a performance-based contract to provide training and technical assistance for grantees to receive assistance, as appropriate, in the management and implementation of their projects, and to ensure their capacity to sustain their efforts over time. During the procurement process for this contract, a review panel evaluated each technical proposal and recommended the one that proved most advantageous to the Department from both a quality and cost standpoint.

Evidence: Individual budget analysis reviews, Peer review contract, Mentoring Training and Technical Assistance Contract, performance measures tab in PART.

Does the program collaborate and coordinate effectively with related programs?

Explanation: During the initial development of the program, the Department worked closely with USA Freedom Corps and the White House Office of Faith-Based and Community Initiatives as a part of its efforts to expand volunteerism and increase opportunities for community service. In addition, ED staff consulted program staff at OJJDP to examine their technical assistance and national evaluation efforts in mentoring. As a result of these collaborations, ED was able to apply the lessons learned from previous Federal mentoring efforts to the ED program, especially in the areas of the training and technical assistance. ED also used this information to strengthen the emphasis on program design and management (a major factor in ineffective mentoring programs) in the criteria used to evaluate applications. Additionally, ED staff requested and received from OJJDP a comprehensive list of program sites funded under the Juvenile Mentoring Program (JUMP) in order to avoid duplication of effort in program funding and support. As a member of the Coordinating Council on Juvenile Justice and Delinquency Prevention, which was established to examine how programs can be coordinated among federal, state, and local governments to better serve at-risk youth, ED has worked with other agencies to examine Federal efforts in mentoring and to investigate possible areas of collaboration. In addition, OSDFS has collaborated with the Department's Center for Faith and Community-Based Initiatives to provide technical assistance workshops for potential applicants interested in applying for funding. These workshops were conducted at meetings across the country in addition to "virtual" meetings using Internet streaming technology available at the Department. Finally, ED has recently begun coordinating its efforts in mentoring with the Corporation for National and Community Service. The Corporation is working with OSDFS and other agencies to explore and develop effective approaches for using volunteers to enhance the overall effectiveness of national and community service programs. OSDFS also continues to ensure that overlapping or duplicative awards are not made by comparing the proposed slates of grantees to OJJDP's new award list.

Evidence: ED and OJJDP grant award slates; program files.

Does the program use strong financial management practices?

Explanation: This and other OSDFS programs benefit from the generally improved financial management practices employed by the Department. For example, recent audits have not identified material weaknesses in the Department's financial management practices. Other Department enhancements provide a strong framework for effective financial management; these enhancements include monitoring excessive grantee drawdowns, identification of high-risk grantees, development of mandatory monitoring plans, enhanced oversight of indirect cost claims by grantees, and emphasis on risk assessment,. Additionally, during performance monitoring program staff check for any potential audit findings identified in the Federal Audit Clearinghouse.

Evidence: ED audits, GPOS Handbook, GAPS system enhancements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Through rulemaking in 2004 ED developed and added three key performance measures to assess the effectiveness of the program. As a result of a GAO audit of the mentoring program, which found that staff were not making use of information submitted to the Federal Audit Clearinghouse, a comprehensive training of grant monitors was undertaken to ensure that staff are able to access the information. A requirement was added to the OSDFS office monitoring procedures directing staff to review audit findings at least annually. In addition, the GAO audit recommended that ED establish a formal process to facilitate information sharing among grant recipients. In response to this recommendation, OSDFS has developed an electronic listserv for grantees to promote communication across all grant sites. OSDFS has also funded a national training and technical assistance center for grantees. Through this center, grantees receive assistance with (1) training to ensure best practice and high-quality, evidence-based programs, (2) identifying gaps and weaknesses in their program design, (3) helping grantees collaborate with other organizations, and (4) planning strategically for program sustainability.

Evidence: GAO audit Student Mentoring Programs, OSDFS Watch List, Monitoring Plan, staff training.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All applications are reviewed according to the requirements of the Education Department General Administrative Regulations (EDGAR), Part 75. All reviewers conduct an independent off-site review and post their comments and scores to a secure web site. During the 2004 grant competition, the Department used five selection criteria to evaluate applications: Need for Project (10 points), Quality of Project Design (30 points), Quality of the Management Plan (35 points), Quality of Project Personnel (10 points), and Quality of Project Evaluation (15 points). In addition, partnerships between LEAs or CBOs were strongly encouraged by including a competitive preference bonus of five additional points added to the selection criteria for the grant competition. Peer reviewers are given specific guidance on how to evaluate the quality of an applicant's response to each criterion to ensure ratings accurately reflects the merits of each application reviewed. In addition, OSDFS staff facilitates and monitors the work of each review panel and ensures that the reviewers' judgments regarding strengths and weaknesses of an application are clearly substantiated by the score given. In addition, peer reviewers are required to undergo training on the priority and scoring criteria for the mentoring program as a condition of being selected for the review. Members of the OSDFS staff oversee the operation of peer review panels, facilitate conference calls to discuss each application, and ensure reviewers apply the scoring criteria consistently and appropriately.

Evidence: Peer review contract requirements, EDGAR, grant competition technical review forms, and peer reviewer training materials.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Department gains knowledge about grantee activities through monitoring visits, conferences, grantee workshops, and the development and review of annual performance reports. Throughout the performance period of the grant, grant monitors review and analyze budget information reported on the annual performance reports to ensure satisfactory progress towards project goals and to ensure the grantee is meeting its stated goals and objectives. Performance agreements for individual grant monitors for this program establish standards for grant program implementation, including the development of a monitoring strategy to track and evaluate grantees' progress in achieving results. These agreements also require program monitors to apply appropriate tools (site visits, review and analysis of performance reports, and twice yearly performance calls) to proactively identify and address poor performance or high-risk grantees. In addition, program monitors are expected to regularly monitor a grantee's drawdown activity to determine whether any cause for concern exists. Using the Department's Grants Administration and Payment System, monitors are able to view a grantee's expenditure of funds. Grantees are contacted if excessive drawdowns occur, or if grant funds are not used within a reasonable period of time. If the program monitor determines that funds have been used inappropriately or ineffectively, there are several courses of action, including site visits, the return of Federal funds, or the placement of special conditions (e.g., grantee payment on a reimbursement system). The Department has also worked to create a system of internal controls designed to enhance program management. OSDFS has developed a "watch list" for each of its program that identifies and records information about grantees that exhibit potentially significant problems. In constructing the list, staff consider grantee progress along a number of dimensions including financial issues, program management, and performance. The goal of the list is to help focus attention and resources on "troubled" grantees and provide technical assistance designed to prevent more serious problems and potential designation of grantees as "high risk". This watch list approach works in conjunction with the OSDFS monitoring plan that uses a variety of monitoring strategies including performance and mid-point calls, review, and analysis of performance reports, and a limited number of on-site visits. According to the most recent watch list, there are a total of 45 programs on the list.

Evidence: OSDFS 2004 Monitoring Plan and Watch List.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: ED collects information through performance reporting and monitoring to measure how well it is meeting the prescribed indicators (i.e., how well the Department is achieving its objectives). Through these data collection efforts, ED is able to evaluate its effectiveness in reaching the goals and objectives of the Department's Strategic Plan. GPRA measures for the program are included in ED''s Program Performance Management Database. Information about program measures is available on the Department's website; that information will be supplemented with information about targets and eventually aggregate performance information when it becomes available. While there is no current process to make individual grantee performance data available to the public in a transparent and meaningful manner, the office is considering a pilot effort to develop a template and posting such information on the Department's website.

Evidence: Grant files, grantee correspondence, electronic reporting.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The maximum program length for this program is three years. As such, the program's performance measures are both the annual and long-term performance measures. The Department is collecting data for the program performance measures beginning with the FY 2004 cohort. Data on the long-term meaures for the 2004 cohort of grantees under the program will not be available until 2008. For the first two measures, the target for 2007 is established two years after award because grantees will need to have operated for a minimum of 12 months in order to produce any matches that meet the criteria established. As a result, baseline data became available in fall 2006. For the third performance measure, which looks at the proportion of mentored students with unexcused absences from school, the Department did not meet its 2006 target for the 2004 cohort of grantees.

Evidence: Measures tab in PART.

Does the program (including program partners) achieve its annual performance goals?

Explanation: As for the long term measures, the Department is collecting data for the program performance measures beginning with the FY 2004 cohort. For the first two measures, no target is established in the first year after award because grantees will need to have operated for a minimum of 12 months in order to produce any matches that meet the criteria established. As a result, baseline data became available for these measures in fall 2006. For the third performance measure, which measures the proportion of mentored students with unexcused absences from school, the Department did not meet its 2006 target for the 2004 cohort of grantees.

Evidence: Measures tab in PART; annual performance reports.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Department has developed and is implementing an efficiency measure for this program (cost per student mentored for each student-mentor match that lasts a year) for which the Department has 2006 baseline data for the 2004 cohort of grantees and has established future year targets. However, the Department does not yet have data to measure progress against those targets. The program also addresses efficiency and cost effectiveness in other ways. ED entered into a performance-based contract to assist the program staff with the grant competition, grantee conference planning, and administrative management. Under this performance-based contract, the contractor has incentives to perform timely and accurately for the Department on all its contract tasks or be penalized. Through this contract, OSDFS executed an online peer review process that greatly improved efficiency and resulted in cost savings. Overall, this contract has saved approximately 20-30% in program execution costs. These savings are based on eliminating travel costs for almost 400 reviewers and the costs of two weeks of meeting space. In addition, beginning with the fiscal year 2004 grant competition, this program has encouraged and accepted electronic applications. Of the 1,569 applicants that applied, just under 20% submitted an electronic application. Providing an electronic application process increases efficiencies in time management and productivity and supports the Administration's initiative on electronic government.

Evidence: Peer review contract, costs assessed for each competition review.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Currently there is no research or data that suggest that youth who participate in school-based mentoring programs funded by the Department obtain a different level of benefits than those that participate in community-based or other types of mentoring programs. However, an impact evaluation of Education's school-based mentoring program is currently under way and the results of this evaluation will allow the Department to compare its mentoring program to mentoring programs funded by other entities.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: ED is funding an independent evaluator to conduct an impact study to assess the outcomes of the program, but the findings of the evaluation will not become available until 2008.

Evidence: National Evaluation Contract.