ExpectMore.gov


Detailed Information on the
TRIO McNair Postbaccalaureate Achievement Assessment

Program Code 10003335
Program Title TRIO McNair Postbaccalaureate Achievement
Department Name Department of Education
Agency/Bureau Name Department of Education
Program Type(s) Competitive Grant Program
Assessment Year 2006
Assessment Rating Moderately Effective
Assessment Section Scores
Section Score
Program Purpose & Design 100%
Strategic Planning 88%
Program Management 80%
Program Results/Accountability 58%
Program Funding Level
(in millions)
FY2007 $42
FY2008 $44
FY2009 $44

Ongoing Program Improvement Plans

Year Began Improvement Plan Status Comments
2007

Calculating a third year of grantee-level performance data and program-level efficiency data, making these data available to the public on the web, and examining whether revising the graduate enrollment measure is warranted.

No action taken The third year of data will be for the 2006-07 school year.
2008

Calculating grantee-level graduate school persistence rate data and making these data available to the public on the web.

No action taken Program-level results for 2007-08 will be available in December 2009. Grantee-level tables for 2007-08 will be available by April 30, 2010.
2008

Taking steps to better link rewards for past performance with the achievement of key program goals.

No action taken For the FY 2011 competition, the Department will implement strategies to ensure that past performance is fairly and consistently assessed when assigning points for prior experience. This will include using the new standard program objectives and the Annual Performance Report data to calculate prior experience points.

Completed Program Improvement Plans

Year Began Improvement Plan Status Comments
2006

Standardizing objectives for grantees to ensure that projects can be held to a consistent standard of performance and that national performance information will be available for the program.

Completed
2006

Developing targets for the program's efficiency measure.

Completed The program staff have developed and shared with OMB a strategy for setting efficiency measure targets.
2006

Calculating grantee-level performance data and program-level efficiency data and displaying these data for the public .

Completed The TRIO McNair graduate school enrollment data for 2004-05 is available on the web at http://www.ed.gov/programs/triomcnair/table-1.doc
2007

Complete the evaluation of the McNair program and examine the study findings for implications for program improvement.

Completed The Department released the final report from the McNair evaluation in March 2008.
2007

Calculate a second year of grantee-level performance data and program-level efficiency data and make these data available to the public on the web.

Completed Grantee-level data for 2004-05 and 2005-06 are available on the web at http://www.ed.gov/programs/triomcnair/performance.html.

Program Performance Measures

Term Type  
Long-term/Annual Outcome

Measure: Percentage of TRIO McNair participants enrolling in graduate school within a year of receiving their bachelor's degree.


Explanation:Percentage of participants who enroll in graduate school within a year of receiving their bachelor's degree. The large increase from FY 2004 to 2005 is the result of more complete and timely reporting (enhanced by additional fields in the Annual Performance Report). It also may reflect a shift toward immediate enrollment.

Year Target Actual
1999 N/A 35
2000 35 35
2001 35 40
2002 35 39
2003 36 36
2004 36 45.3
2005 36 56.8
2006 37 56.2
2007 39 data lag [Dec 2008]
2008 39.5
2009 39.5
2010 40
2011 40
2012 41
Long-term/Annual Outcome

Measure: Percentage of TRIO McNair participants persisting in graduate school.


Explanation:Percentage of participants who persist from the first to the second year of graduate school.

Year Target Actual
1999 N/A 48
2000 48 75
2001 N/A 66
2002 48 65
2003 75 78
2004 75 77.7
2005 70 80
2006 79 80.6
2007 79 data lag [Dec 2008]
2008 79.5
2009 79.5
2010 80
2011 80
2012 81
Annual Efficiency

Measure: The federal cost of each McNair program baccalaureate recipient who enrolls in graduate school within three years.


Explanation:This measure uses the federal appropriation for the fiscal year in which the cohort of baccalaureate recipients was established, adjusted for those projects that were not funded in any one of the subsequent three years. The appropriation is divided by the number of students in the cohort of baccalaureate recipients who have enrolled in graduate school at anytime during the subsequent three years. The targets for 2008-13 were established by estimating the expected increase in federal appropriation and a 0.5% biennial increase in the graduate school enrollment rate.

Year Target Actual
2005 NA 40,623
2006 NA 41,777
2007 NA data lag [Oct 2008]
2008 39,000
2009 39,000
2010 38,000
2011 38,000
2012 38,000

Questions/Answers (Detailed Assessment)

Section 1 - Program Purpose & Design
Number Question Answer Score
1.1

Is the program purpose clear?

Explanation: The program is designed to provide support services to low-income, first-generation college students to prepare them for doctoral programs.

Evidence: Section 402E of the Higher Education Act (HEA) states that the purpose is "to provide disadvantaged college students with effective preparation for doctoral study."

YES 20%
1.2

Does the program address a specific and existing problem, interest, or need?

Explanation: Data indicate that low-income, first-generation college students do not enroll in doctoral programs and obtain doctorates at the same rate as students who are less disadvantaged. Program services are designed to address and eliminate this persistent problem.

Evidence: Limited data are available on the precise size of the gap in doctoral program participation between disadvantaged students and their peers. However, The Survey of Earned Doctorates (Hoffer, et al, 2002) reported that only 35% of doctorate recipients are first-generation students (neither parent had received a bachelor's degree), even though first-generation students comprise more than two-thirds of the pre-college population according to 2004 Census data. Additionally, the Census data show that only 0.5% of individuals from underrepresented groups have earned a doctorate compared with 1.3% of the total population over the age of 25.

YES 20%
1.3

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: McNair is designed to provide support services to students with a demonstrated need for assistance and encourage them to pursue doctoral degrees. It is unique because it prepares low-income, first generation college students for both graduate and doctoral programs while they are undergraduates.

Evidence: Although some institutions have programs to encourage students from underrepresented groups to pursue graduate degrees, McNair appears to be alone, from a Federal perspective, in providing a comprehensive set of services including research opportunities, internships, tutoring, counseling, and financing to help disadvantaged students prepare for doctoral study. Other Federal programs, including the Department of Education's (ED) Javits Fellowship and Graduate Assistance in Areas of National Need programs and the National Science Foundation's Graduate Research Fellowships, target students who have already received an undergraduate degree.

YES 20%
1.4

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: McNair has no major design flaws. Furthermore, there is no evidence that other approaches, like leveraging community resources, are more effective in providing support services and improving enrollment in doctoral programs.

Evidence: Performance data suggest that the program's design is effective. In 2004, 45% of McNair participants immediately enrolled in graduate school after completing their undergraduate degrees. Of the McNair participants enrolled in graduate school, 78% were persisting toward doctoral degrees. The McNair study suggests that the program's overall graduate school enrollment rate may be as high as 77%, including participants who delay enrollment by one or more years.

YES 20%
1.5

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: McNair is well targeted to disadvantaged students who are underrepresented in graduate education. To receive funding, grantees must provide assurance in their applications that participants will be low-income, first-generation college students or from a group that is underrepresented in graduate education. Additionally, ED urges projects to serve undergraduate students rather than graduate students, to ensure resources are targeted to students who have both potential for doctoral study and the need for support services.

Evidence: The statute requires projects to assure that at least two-thirds of participants are low-income, first-generation college students. The remaining participants must be from a group that is underrepresented in graduate education. McNair's profile report (AIR, 2005) indicates that more than 70% of participants are low-income, first-generation college students. 2005 data also indicate that 99% of participants are undergraduates [Note: In a limited number or instances, McNair grantees are allowed to serve graduate students to promote their retention].

YES 20%
Section 1 - Program Purpose & Design Score 100%
Section 2 - Strategic Planning
Number Question Answer Score
2.1

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The primary goal of McNair is to prepare students for doctoral programs. Therefore, ED established long-term measures to increase the graduate school enrollment and persistence rates of participants. Successful enrollment and persistence in graduate school are key indicators that students have been well prepared for doctoral programs.

Evidence: The GPRA measures track the percentage of McNair participants who enroll in graduate programs within a year of baccalaureate graduation and the percentage of students in graduate programs who persist from the first year of graduate school to the second.

YES 12%
2.2

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: In 2005, based on annual trend data through 2003, ambitious targets and timeframes were established through 2011. The graduate school enrollment target exceeds the national average and the persistence target is roughly equivalent to the national average. ED has begun to reexamine graduate school enrollment data for prior years, in light of data fluctuations and study findings. As more data become available, ED plans to determine whether the current measure (tracking immediate enrollment) and targets could be improved, especially by tracking the graduate school enrollment rate a few years after participants complete their undergraduate degrees.

Evidence: The graduate school enrollment target is 38% by 2011 and the persistence target is 80% by 2011. According to NCES's Baccalaureate and Beyond Study (B&B: 2000/01), 21.1% of students who earned a bachelor's degree in 1999 enrolled in graduate school by 2001. According to the National Postsecondary Student Aid Study (2004), 81.5% of doctoral students whose income is in the lowest 10% of the nation persisted to the second year, while 77.5% of doctoral students whose parents had not achieved a bachelor's degree persisted to the second year.

YES 12%
2.3

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The annual goals are the same as the long-term goals. Annual performance data track progress against short-term targets while also tracking progress against the long-term goals.

Evidence: The GPRA measures track the percentage of participants who enroll in graduate school within a year of completing the baccalaureate degree and the percentage of participants who persist from their first to second year of graduate school.

YES 12%
2.4

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Using data from grantee performance reports, ED established baselines and targets for incremental improvement for its annual measures of the graduate school enrollment and persistence rates of McNair participants. The program serves a relatively small number of students (about 4,200) and actual performance data fluctuate from year-to-year. Establishing targets based on any one year's data, therefore, is problematic. For the percentage of participants who enroll in graduate school within a year of receiving their bachelor's degree, figures for the past 5 years have fluctuated from 35 to 45.3, rising 5 percentage points from 2000 to 2001, then dropping in the next 2 years, followed by a large rise in 2004. The median percentage over the last 5 years is 39, and provides a more reliable basis for revising outyear targets than any one year. Based partially on this figure, in 2006, the targets for 2007 through 2011 were increased by 2 percentage points. A similar problem exists for the measure of the percentage of participants who persist from the first to the second year of graduate school. The outyear target of 80 percent, however, appears to be a reasonable and ambitious target given that it would mean that 4 out of every 5 students continued in graduate school after the first year. As more data become available, ED plans to determine whether the current graduate school enrollment measure and targets could be improved, especially by tracking the graduate school enrollment rate a few years after participants complete their undergraduate degrees.

Evidence: Annual performance targets are included in the program performance plan.

YES 12%
2.5

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All McNair projects work toward the annual and long-term goals of the program. The program measures have been shared with grantees through application packages and closing date notices. Annual performance reports are required of all grantees, and their performance is measured on the basis of how well they meet the program goals.

Evidence: The program regulations (34 CFR 647) clearly articulate the program goals and indicate that grant awards, continuation funding, and prior experience points are awarded partly on the basis of how well projects succeed in helping students enroll in graduate school and attain a doctoral degree.

YES 12%
2.6

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: An independent evaluation to examine Ph.D. completion and employment outcomes of McNair participants was initiated in 2002. Since the McNair program was first funded in 1989 and participants can be expected to take 7-11 years to complete doctoral programs, this is the first major study of the program. The TRIO evaluation plan includes a follow-up study to examine the most effective and efficient McNair projects.

Evidence: Decision Information Resources, Inc began an independent evaluation in 2002 to examine the extent to which students participating in McNair earn Ph.D. degrees and become faculty members. The study relies on a stratified random sample of all McNair participants from 1989-2000. A final report is expected in summer 2006.

YES 12%
2.7

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. However, ED has satisfied the second part of this question in that the ED's budget submissions show the full cost of the program (including S&E). Additionally, ED's integrated budget and performance plan includes the program's annual and long-term goals.

Evidence: Budget requests for McNair have not been directly linked with program performance information.

NO 0%
2.8

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program performance measures for McNair have been held constant while targets have been made more ambitious. ED is examining the fluctuation in graduate school enrollment data and whether data and targets should be revised to track enrollment several years after participants complete their undergraduate degrees. ED developed a long-term evaluation strategy for the TRIO programs, including the McNair program. Additionally, ED is examining ways to better link McNair budgetary decisions with performance information.

Evidence: The current performance measures have been in place since 1999 and new annual and long-term targets were developed in 2005 based on improving performance between 2000-2003. ED is analyzing and comparing annual performance data with the McNair study findings to better understand differences between the data and identify opportunities to further improve performance measurement. The TRIO evaluation plan includes a follow-up study to examine the most effective and efficient McNair projects.

YES 12%
Section 2 - Strategic Planning Score 88%
Section 3 - Program Management
Number Question Answer Score
3.1

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ED requires grantees to submit regular and timely performance reports. The information collected is used to improve program performance and to provide technical assistance to grantees to assist them in meeting their proposed goals and objectives. Changes are being made in the 2007 McNair competition to prevent applicants from providing misleading goals so that projects can be held to a consistent standard of performance. TRIO strengthened the verification requirements for McNair performance reporting and focused on financial management deficiencies raised in audit findings for other TRIO programs, by using OPE's new electronic grant monitoring system. The requirement to submit student-level data should help ensure data quality by requiring grantees to clearly track indvidual-level performance for program monitoring visits. Finally, TRIO developed a quarterly newsletter to improve communication with grantees.

Evidence: Data analysis revealed that a number of McNair projects historically received excessive levels of funding, despite performance levels similar to other McNair projects. To improve overall performance, all McNair projects were required to ensure their average cost per student did not exceed $10,000 following the 2003 competition. Since 2005, project objectives for all new grant competitions in TRIO have been standardized to ensure that data are comparable. The Department will develop standardized objectives for McNair this summer. TRIO has held teleconferences, shared information, and conducted site visits with McNair projects to focus on inaccurate indirect cost reporting, failure to adequately expend grant funds in a timely fashion, and failure to properly document and serve the required number of eligible students. E-Monitoring enables program staff to track and monitor key aspects of McNair grants. Seven editions of the newsletter have been published. The newsletter has highlighted the PART process and the value of program efficiency measures. Up to 15 prior experience points can be granted to current grantees on the basis of performance data.

YES 10%
3.2

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to EDPAS, which links employee performance to relevant Strategic Plan goals and action steps and is designed to measure the degree to which a manager contributes to improving program performance. The Director of TRIO, program managers, and program specialists are all held accountable for the performance of projects or the program. Additionally, funding decisions for current grant recipients are based on prior performance.

Evidence: The EDPAS standards for TRIO program managers require that they develop strategies for implementing GPRA and Strategic Plan initiatives related to TRIO. TRIO program specialists are held accountable for assessing project performance, calculating prior experience points, and monitoring the progress of projects in achieving program goals and objectives. McNair grantees receive continuation funding on the basis of their reported progress in achieving the program goals, and they may receive up to 15 points for their prior performance when applying for a new grant.

YES 10%
3.3

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are obligated in a timely manner. Further, to ensure that funds are properly managed, ED monitors the financial records of McNair grantees using GAPS and quarterly reports on excessive drawdown or inactivity. In addition, ED strengthened its financial management practices by implementing e-Monitoring to help prevent potential fraud, waste, and abuse.

Evidence: McNair grants are typically obligated by June of each year, far before the agency average. Additionally, new competitions are held 6 to 8 months ahead of schedule to ensure that new projects have adequate time to begin providing services on the project start date. Staff monitor drawdown of funds in McNair by reviewing grantees' financial reports electronically. ED's e-Monitoring was implemented in 2004. A memo explaining the consequences of excessive drawdowns was sent to all McNair grantees. In addition, staff have begun monitoring grants that are not spending funds as quickly as anticipated. Grantees must submit a written request before any accounts are reopened after the close of a grant cycle, and TRIO does not allow returned funds to be used for current grant activities instead.

YES 10%
3.4

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: TRIO developed a common efficiency measure, the annual cost per successful outcome, to assess the cost effectiveness of each TRIO program and project on an annual basis. In June 2005, ED began implementing the efficiency measure for the TRIO Student Support Services program. The implementation strategy includes communicating with grantees, calculating efficiency data, setting targets for improved efficiency, and identifying and sharing best practices. ED has begun implementing this strategy for McNair and plans to have baseline efficiency data and a target later this summer.

Evidence: The annual cost per successful outcome for McNair should be available this summer. TRIO's fall newsletter discussed the efficiency measure strategy for the pilot effort. Up to 15 prior experience points may be awarded to all eligible applicants during a competitive cycle.

NO 0%
3.5

Does the program collaborate and coordinate effectively with related programs?

Explanation: TRIO program managers and staff effectively coordinate with a number of related programs at ED. For example, TRIO staff urge grantees to coordinate with other Federal and non-Federal projects to create a pipeline of services through graduate school. Some projects share project directors to oversee all programs with coordinators providing day-to-day management. TRIO managers also administer the Child Care Access Means Parents In School program and work closely with managers from the the Gaining Early Awareness and Readiness for Undergraduate Programs, Institutional Development and Undergraduate Education Service, Javits Fellowships, and Graduate Assistance in Areas of National Need programs.

Evidence: McNair projects are often linked with TRIO Student Support Services, Javits, and GAANN projects, including a number of institutions that are the recipients of multiple grants. The Winter 2005 newsletter is one example of program coordination as it included critical information for applicants and grantees of all the TRIO programs.

YES 10%
3.6

Does the program use strong financial management practices?

Explanation: The TRIO program office has not been revealed to have internal control weaknesses. In response to an audit report that indicated ED's failure to adequately monitor grantees for their compliance with grant administration requirements, OPE implemented an e-Monitoring system enabling program staff to better monitor financial management and mitigate against potential fraud, waste, and abuse.

Evidence: The OIG audit of TRIO's financial controls found no evidence of erroneous payments or other such material weaknesses. All McNair funded grantee institutions are documented in GAPS and e-Monitoring systems.

YES 10%
3.7

Has the program taken meaningful steps to address its management deficiencies?

Explanation: TRIO continues to improve program management. In response to a series of OIG audits relating primarily to individual grantees in other TRIO programs, TRIO developed a robust grant monitoring strategy for all their programs, including McNair. OPE's e-Monitoring system also has improved TRIO's ability to manage and monitor funds. An efficiency measure was recently developed for McNair, and TRIO will focus on ways to use performance data to improve the effectiveness and efficiency of McNair once data analyses are complete. ED will make these data available to grantees for feedback, both in aggregated and disaggregated formats, hopefully by the end of 2006.

Evidence: The TRIO program office developed a detailed monitoring plan that emphasizes conducting on-site visits to newly funded and high-risk projects (those with evidence of mismanagement, constant turnover in leadership, etc.). A full-time TRIO staff member is now dedicated to project oversight. ED has taken steps to improve the knowledge of staff conducting site visits through training initiatives and uses site visits selectively along with e-monitoring to assure that grantees receive sufficient technical assistance and perform the required activities to achieve their stated project goals. In addtion, based on performance data regarding use of funds, all McNair projects were required to reduce their average cost per student to $10,000 after the 2003 competition. Also, ED plans to publish McNair's efficiency data in the Winter 2006 newsletter.

YES 10%
3.CO1

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All McNair grants are awarded competitively through competitions announced in the Federal Register and on ED's website. Extensive information about the program, including information on ED grant procedures, is available on the website to enable potential grantees to learn about program requirements and accomplishments and to understand application procedures. In addition, the TRIO program office provides outreach and technical assistance to potential grantees. A clear, competitive process is implemented based on an approved Technical Review Plan, which outlines, step-by-step, the procedures to be used in the review process. Competitions are announced in the Federal Register. Independent panels of experts are used to score and rank all applications. The statute and regulations provide up to 15 bonus points for prior experience, a significant advantage, but TRIO has tightened the process for awarding those points to ensure that the competitive preference given to existing grantees is based on demonstrated performance. The TRIO program office provides outreach and technical assistance to potential grantees. Following HEA reauthorization, TRIO also plans to pursue regulatory changes to better link prior experience points with achievement of the key program outcomes.

Evidence: A notice inviting applications for the 2003 competition was published in the Federal Register in September 2002. Due to TRIO's efforts to tighten the process for awarding prior experience points, McNair grantees received fewer prior experience points in the 2003 competition; 65% of grantees received 10 or more points in 2003, down from 87% in 1999. This reflects a continuing trend across all the TRIO programs to improve the competitive grant process. Most recently, 50% of Student Support Services grantees received the maximum 15 prior experience points in 2005, down from 74% in 2001. 100% of grants are subject to review. Weblinks: http://www.ed.gov/about/offices/list/ope/trio/index.html and http://www.ed.gov/fund/grant/about/edpicks.jhtml?src=ln

YES 10%
3.CO2

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: New procedures were implemented for improving the monitoring of expenditures. GAPS, e-Monitoring, performance reports, financial status reports, and other internal reports are used to ensure that all related grant financial management practices are followed. Performance reports are reviewed to ensure that the information requested provides quality data and evidence of the success of the program, as well as of success in the individual projects.

Evidence: In addition to increasing efforts at on-site monitoring, the TRIO program office continues to review all reports and make follow-up calls to clarify questions and concerns, especially regarding changes to the scope of projects. Oversight includes documentation of grantees' use of funds, email communications, and project directors' meetings. A full-time TRIO staff member is dedicated to project oversight.

YES 10%
3.CO3

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The TRIO program office collects and compiles data from the electronically-submitted annual performance reports and produces a program profile report biennially. The reports address the GPRA performance measures and are available on ED's website. TRIO recently put disaggregated efficiency data on its website for the Student Support Services program, the first step in ED's plan to release performance data for all the TRIO programs. TRIO continues working to increase the timeliness of making data available and plans to release grantee-level performance data for McNair later this summer.

Evidence: Program profile reports prepared by a contractor are available on TRIO's website at http://www.ed.gov/programs/triomcnair/resources.html. The next report is scheduled for release in 2007. The reports include complete performance data aggregated at the program level. Data also are available by type of institution and, in some cases, by region or state.

NO 0%
Section 3 - Program Management Score 80%
Section 4 - Program Results/Accountability
Number Question Answer Score
4.1

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: In late 2004, ED established long-term targets for increasing the graduate school enrollment and persistence rates of low-income, first-generation McNair participants by 2010. At the time, these targets were based on actual data from 1999-2003, which fluctuated significantly. In 2005, the long-term targets were extended to 2011 and, for the persistence measure, made more ambitious. Although fluctuations in performance have made it difficult to establish ambitious long-term targets and measure success on an annual basis, the performance of McNair on its long-term goals is improving significantly over time. Moreover, McNair has exceeded or is on pace to exceed the current long-term targets.

Evidence: Actual data reported in the program performance plan show that McNair had a 45.3% graduate school enrollment rate in 2004, exceeding the 38% target for 2011. Additionally, the data show that McNair had a 77.7% graduate school persistence rate in 2004, which is on pace to meet or exceed the 80% target for 2011. ED has begun to reexamine graduate school enrollment data for prior years, in light of data fluctuations and study findings. As more data become available, ED plans to determine whether the current measure (tracking immediate enrollment) and targets could be improved, especially by tracking the graduate school enrollment rate a few years after participants complete their undergraduate degrees.

LARGE EXTENT 17%
4.2

Does the program (including program partners) achieve its annual performance goals?

Explanation: McNair met or exceeded all of its annual graduate school enrollment and persistence targets for 2000-2004, which had been set to maintain current performance levels.

Evidence: Actual data reported in the program performance plan fluctuates from year-to-year. However, McNair met or exceeded all of its annual targets through 2004 and performance is improving over time. Most recently, the graduate school enrollment rates for 2003 and 2004 were 36% and 45.3%, which met or exceeded the 36% target for both years. The graduate school persistence rates for 2003 and 2004 were 78% and 77.7%, which exceeded the 75% target for both years. The fluctuations in enrollment rates may be due to changing economic conditions and the challenges faced by disadvantaged students attempting to pursue doctoral degrees. The significant increase in persistence in 2003 is largely due to a change in the way the data are measured; ED shifted from a cumulative measure of persistence to a one-year rate that assesses the percentage of McNair recipients who remained enrolled in graduate school for a second year. Annual targets for 2006 and beyond were set to increase performance by 0.5 percentage points biennially. As more data become available, ED plans to determine whether improvements could be made to the graduate school enrollment targets and the way the data are calculated.

YES 25%
4.3

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ED is developing the McNair efficiency measure of cost per successful outcome. Efficiency data will be made available once data analyses are complete. ED has established a strategy for improving efficiency based on sharing data and best practices.

Evidence: The average cost per successful annual outcome baseline data and a preliminary target should be available later this summer.

NO 0%
4.4

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Most schools and many organizations offer financial assistance and other support to low-income students once they have been accepted into a graduate or doctoral program. ED's Javits Fellowships and Graduate Assistance in Areas of National Need programs target graduate and doctoral students, and the National Science Foundation provides similar support for graduate study. However, ED is not aware of any Federal or other large-scale programs that provide support to help prepare disadvantaged undergraduate students for graduate or doctoral programs.

Evidence: Programs that provide support to undergraduates in preparation for graduate or doctoral study, including the Alfred P. Sloane Foundation Minority PhD and Mellon Mays Undergraduate Fellowship programs, are small in comparison to McNair. Furthermore, most of these programs tend to be very specific in the areas of study that they support, whereas the McNair program provides preparation for doctoral study in any field.

NA 0%
4.5

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The independent evaluation of the educational and employment outcomes of former McNair participants is in its final stages; [Note: the report is due in summer 2006]. The study determines the extent to which McNair participants enrolled in graduate school and earned graduate and doctoral degrees. Findings from the study indicate that McNair participants are enrolling in graduate school at comparatively higher rates than the national average for bachelor's degree recipients.

Evidence: The study findings suggest that a substantially higher percentage of McNair alumni enroll in graduate school compared to the national average for bachelor's degree recipients: approximately 27% of undergraduates enroll in graduate school nationally, compared to 77% of McNair participants, who are low-income and first-generation college students. Looking beyond the primary goals of the program, the study also found that 5.8% of participants who have had at least 7 years to prepare for and enroll in doctoral programs earned doctoral degrees and 7.9% earned professional degrees (for example, M.D., J.D., or PharmD.). The most common factors cited as contributing to participant success were receiving research opportunities at the undergraduate level and having a faculty mentor. Comparing the outcomes of McNair participants with other undergraduates is difficult because few data are available nationally on the number and type of individuals who intend to pursue and complete doctoral programs. Additionally, given the relative size and age of the McNair program compared to the other TRIO programs, a more rigorous comparison study has not been feasible. However, McNair's outcomes are promising given that the Survey of Earned Doctorates (2005) indicates the average length of time between a student recieving a bachelor's degree and a doctorate is 10 years. ED's ongoing analysis of annual graduate school enrollment data should provide additional insights into the overall performance of the program.

LARGE EXTENT 17%
Section 4 - Program Results/Accountability Score 58%


Last updated: 09062008.2006SPR