Detailed Information on the
American Indian Vocational Rehabilitation Services Assessment

Improve use and transparency of project data to manage and improve the program, including posting summary analyses and key data on the web.

Develop a strategy for collecting data to support the Administration's job training common measures and establish performance targets.

Monitor key performance and fiscal indicators and provide technical assistance to ensure the timely expenditure of project funds and the achievement of project goals.

Examine reporting inconsistencies and develop guidance to grantees in time to collect 2006 data.

Implement an outcome efficiency measure to assess how project costs are linked to achieving employment outcomes.

Measure: Percentage of all eligible individuals who exit the program after receiving services under an individualized plan for employment (IPE) that achieve an employment outcome.

Explanation:Numerator is all eligible individuals (American Indians with disabilities as defined in 34CFR371) who exited the program in the reporting perid after receiving services and achieved an employment outcome. The denominator is the total number of eligible individuals who exited the program in the reporting period after receiving VR services under an IPE.

Measure: Number of American Indians with disabilities receiving services under an individualized plan for employment.

Explanation:The purpose of this output measure is to assess the program's performance in building the capacity to serve American Indians with disabilities.

Measure: Percentage of AIVRS projects that demonstrate an average annual cost per employment outcome of no more than $35,000.

Explanation:Cost per employment outcome is calculated by dividing total federal grant funds by the total number of individuals who achieved an employment outcome.

Measure: Percentage of AIVRS projects that demonstrate an average annual cost per participant of no more than $10,000.

Explanation:A project's average cost per participant is calculated by dividing the amount of project's Federal grant by the number of individuals who received services under an individualized Plan for Employment during the reporting period.

Measure: Job Training Common Measure: Cost per Participant

Explanation:Total federal grant funds, divided by the number of individuals served under an individualized plan for employment by projects operating in that fiscal year.

Measure: Job Training Common Measure: Entered Employment - Percentage employed in the first quarter after program exit.

Explanation:Numerator: Of those who are not employed at registration, the number of adults who have entered employment by the end of the first quarter after exit. Denominator: Of those who are not employed at registration, the number of adults who exit during the quarter.

Measure: Job Training Common Measure: Retention in Employment - Percentage of those employed in the first quarter after exit that were still employed in the second and third quarter after program exit.

Explanation:Numerator: Of those who are employed in the first quarter after exit, the number of adults who are employed in the second and third quarter after exit. Denominator: Those who are employed in the first quarter after exit.

Measure: Job Training Common Measure: Increase in Earnings - Percentage change in earnings: (i) pre-registration to post program; and (ii) first quarter after exit to third quarter.

Explanation:Numerator 1:Participant's earnings first quarter after program exit minus participant's earnings two quarters prior to registration. Numerator 2:Participants earnings third quarter after program exit minus participant's earnings first quarter after program exit. Denominator 1: Participant's earnings two quarters prior to registration. Denominator 2: Participant's earnings first quarter after program exit.

Is the program purpose clear?

Explanation: The program provides vocational rehabilitation services to American Indians with disabilities who reside on or near Federal or State reservations, consistent with their individual strengths, resources, priorities, concerns, abilities, capabilities, interests, and informed choice, so that they may prepare for and engage in gainful employment. The purpose is clearly defined in regulations, however the statute does not clearly identify the intended outcome of the program (i.e., to prepare individuals with disabilities for gainful employment.)

Evidence: 34 CFR part 371.1. The Senate Bill reauthorizing and amending the Rehabilitation Act includes language that would clarify the purpose of the program.

Does the program address a specific and existing problem, interest or need?

Explanation: The program addresses the specific need to increase the employment of American Indians with disabilities by building the capacity of American Indian Tribes to develop and implement vocational rehabilitation programs that are delivered in a culturally-relevant individualized manner.

Evidence: Disability rates are higher than average among Americans Indians, and rates are reported to be particularly high for those on or near reservations. American Indian Vocational Rehabilitation Services (AIVRS) projects' face considerable challenges in providing vocational rehabilitation (VR) services due to geographic, economic, and cultural factors. In most cases, projects reside in rural areas with limited resources for service provision and limited job opportunities. AIVRS' service areas have very high unemployment rates, even compared to the surrounding rural areas. AIVRS serve consumers who have: disabilities that are difficult to ameliorate, cultural barriers to employment off of the reservation (e.g., language and cultural values), and potential discrimination in employment. Executive Summary, Evaluation of the American Indian VR Services Program (June 2002) www.ed.gov/policy/speced/leg/rehab/eval-studies.html#aivrs

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program provides grants to governing bodies of American Indian tribes located on Federal and State reservations (and consortia of such governing bodies) that pay 90 percent of the costs of VR services for American Indians with disabilities who reside on or near such reservations. Although this population can receive services under the larger State VR program, historically they have not chosen to do so. Itinerant visits to the reservations by State VR counselors providing services to AIVRS clients have not been successful. No other tribal programs provide VR services to this population. There are no other Federal (e.g. Department of Labor or Bureau of Indian Affairs), State or local programs that provide VR services to American Indians on reservations.

Evidence: Tribal governments are uniquely positioned to provide VR services to American Indians on or near Indian reservations. Projects are staffed by members of the tribal community who speak the native language. The remoteness of reservations, language and other cultural differences have been traditional barriers in serving American Indians living on or near reservations. Executive Summary, Evaluation of the American Indian VR Services Program (June 2002) www.ed.gov/policy/speced/leg/rehab/eval-studies.html#aivrs

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence that another approach or mechanism would be more efficient/effective in achieving the intended purpose. The AIVRS evaluation found that projects generate appropriate levels of successful employment outcomes at reasonable costs. However, the evaluation also found that program stability continues to be a concern of American Indian tribes operating these projects. Uniquely designed, the program is a hybrid of a State-administered program and a discretionary program. The grantees are tribal governments that administer a program similar to the State VR Services program. However, the projects are awarded as discretionary grants and must re-compete for a grant every five years. The statute requires that priority consideration be given to applications of previously funded AIVRS projects. The Department has carried out this priority by awarding ten extra points to existing projects. Due to poor grant-writing skills, successful current-funded projects still may not rank high enough to received continued support. An alternate method of making awards to existing successful projects could possibly improve program efficiency/effectiveness.

Evidence: Given the challenging environments in which they work, AIVRS considers the rates of employment outcomes (participants placed in employment) by projects as very good. The rates compare favorably to rates for American Indians served by State VR agencies, who often live in areas with more positive economic environments. Although the latest AIVRS evaluation found reasonable costs per consumer, it noted that better outcomes and greater cost-effectiveness positively related to project staff with more years of VR experience and projects with more years of federal funding. [Evaluation of the American Indian VR Services Program (June 2002)] The Senate Bill reauthorizing and amending the Rehabilitation Act includes language that would provide the Commissioner the authority to renew 5-year grants for additional 5-year periods if it is determined that the grantee has demonstrated acceptable past performance and has submitted a plan that the Commissioner approves, identifying future performance criteria, goals, and objectives.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Resources are targeted to American Indians with disabilities who reside on or near reservations and services provided on the reservation. Projects are located on the reservation and operated by tribal governments who conduct more effective outreach than State VR agencies and can target resources to individuals who are likely to benefit from VR services. Services are provided under an individualized plan for employment (IPE) that specifies the employment goal or expected outcome and the services and resources need to achieve the employment goal.

Evidence: Section 121(a) of the Rehabilitation Act and 34CFR part 371.2 and 371.4.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: A long-term Government Performance and Results Act (GPRA) measure has been established that focuses on the primary outcome of the program (employment). The measure was included in the grant application package for new fiscal year 2004 awards under this program. As part of the Job Training Common Measures initiative, the program has adopted new long-term measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, than its current measures and (2) facilitate comparisons with similar programs. The Department of Education (ED) is conducting a study to understand how to implement the common measures.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html Federal Register, March 3, 2004 Vol. 42, pages 10009- 10011.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has established reasonable quantifiable targets and timeframes for it's long-term measure, but has not set baselines nor ambitious targets for the job training common measures. For the established long-term measure, targets are set based on program data analysis and expected project outcomes. Targets take into consideration factors such as grantee experience. For example, new grantees are expected to produce fewer outcomes in the first year of the grant. The targets appear somewhat conservative because the Department only recently implemented a standard reporting system and is still assessing granteee data reliability. The long-term performance measure is computed by comparing the number of individuals who exited the program after achieving an employment outcome with all those that exited the program in the reporting period (i.e. whose service record was closed during the fiscal year). For some of the projects, the number of individuals whose service records are closed each year is significantly less than expected, perhaps due to cultural mores (e.g., closing a consumer's record of services may be interpreted as giving up on a consumer). Culturally sensitive guidance on when to close the record of service needs to be developed to ensure more accurate program data. The performance goal and targets may need to be modified in the future to reflect improved practices in the closure of service records.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: AIVRS has annual and long-term targets, and timely annual data. All three measures are discrete and quantifiable. The outcome measure demonstrates the program's progress in meeting the long term-goal. As part of the Job Training Common Measures initiative, the program has adopted new annual measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, and (2) facilitate comparisons with similar programs. ED is conducting a study to understand how to implement the common measures. Also see annuals goals in 4.2.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines were established and reasonable targets set to ensure continued program improvement. Targets are set based on analysis of program data and anticipated project outputs. Project expectations increase over time as project staff gain knowledge and experience and build project capacity. Targets for the job training common measures will be developed after baseline data is collected.

Evidence: The performance targets are typically established two (2) years prior to the availability of the actual data. For some of the annual measures, this appears to be an underestimation of targeted performance, instead of projections based on past performance. Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Each grantee must annually report its performance on these measures through the AIVRS Annual Performance Reporting System. The primary purpose, goals, and measures of perfomance of the AIVRS and the State VR Services are essentially the same. In instances where an individual receives services from both the AIVRS project and VR State agency, the entities would be working towards the same goal.

Evidence: AIVRS Annual Reporting Form, Office of Management and Budget (OMB) Number 1820-0655 Grant application

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Development Associates, an independent contractor, conducted the first comprehensive program evaluation in 2002. They used surveys, site visits, focus groups, interviews and service records to examine consumer characteristics, services provided, outcomes, and management of the program in order to provide technical assistance and information for program improvement. In addition the evaluation analyzed the relationship between characteristics, services received, and employment outcomes of American Indians served under the VR State Grants program and those served under the AIVRS program. Future evaluations should focus on program effectiveness employing the most rigorous evaluation design that is appropriate and feasible for this program.

Evidence: Evaluation of the American Indian VR Services Program (June 2002) www.ed.gov/policy/speced/leg/rehab/eval-studies.html#aivrs

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Rehabilitation Act requires that not less than 1.0 percent or more than 1.5 percent of the funds appropriated for the VR State grants program be set-aside for grants under section 121 (AIVRS program). The set-aside is based on the Commissioner's request with input from ED's Budget Office. The budget request also identifies annual salary and expense costs allocated to the program.

Evidence: Rehabilitation Act, Section 110 (c) Within statutory limits, the budget policy has been to increase the capacity of the AIVRS program to reach additional American Indians with disabilities. Output targets also reflect this policy.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program recently established a long-term measure and implemented a data collection and reporting system in FY 2003. In response to identified problems, RSA has also increased the provision of technical assistance and the size of new awards. As part of the Job Training Common Measures initiative, the program will implement new long-term measures that (1) will indicate participants' employment and earnings outcomes, as well as program efficiency, and (2) facilitate comparisons with similar programs. RSA is conducting a study of common measures to assist in implementation and assess the capacity of program grantees to collect and report these data.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html; AIVRS Annual Reporting Form, OMB Number 1820-0655; Assisting Grantees with Common Measures ED01CO0052/0011

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: In 2003, the Department implemented an on-line data collection instrument for AIVRS (AIVRS Annual Reporting Form). Analysis of the data was very limited due to technical problems grantees encountered in reporting electronically. These problems were corrected in Fall 2004. Program staff will use the data to manage and improve the program. Program staff also receive valuable information from grantees through monthly conference calls to monitor and provide technical assistance. However, outcomes might be inflated since grantees may not apply consistent standards for closing the service records for individuals who have not obtained employment as the larger VR program. In addition, there is very limited information on the types of outcomes obtained. The program will examine reporting inconsistencies and develop guidance to grantees in time to collect FY 2006 data.

Evidence: AIVRS Annual Reporting Form, OMB Number 1820-0655; Conference call agendas; and, Annual Monitoring Plan Guidance

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, the Department has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps - hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its Senior Executive staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. ??The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Federal funds are obligated in a timely manner and funds have never lapsed. This program is covered under the Single Audit Act. Recipients that receive an aggregate of $500,000 or more in federal funds are required to submit to ED an annual independent audit. The purpose of the audit is to demonstrate that the entity has a financial system in place and that federal funds are spent and accounted for properly, in accordance with OMB Circular A-133.

Evidence: All Federal funds appear as obligated on year end fiscal reports. Office of the Chief Financial Officer (OCFO)/Grants Policy notifies program staff of excess drawdowns by grantees. Several times a year, program staff review the financial information in the Grant Administration and Payment System (GAPS). GAPS is used by the Department to track the financial activities of a grant from initial obligation of funds by ED, draw down of funds by grantee, and final settlement of grant. In addition, GAPS maintains demographic information on the grantees. There have been no substantive audit findings in this area. Education Department General Adminsitrative Regulations 34 CFR 80.26.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program's performance plan will be revised to include the common efficiency measure for job training programs. Baseline data is being collected on the cost per participant. AIVRS will develop an efficiency measure to gauge grantee performance by Fall 2004.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: At the Federal level, collaboration is limited. However, tribal projects regularly coordinate with related Federal, State, and local programs, including tribal health, education and employment programs, and the State VR program.

Evidence: Tribal governments and State VR agencies often have cost sharing agreements with respect to the provision of rehabilitation services.

Does the program use strong financial management practices?

Explanation: Auditors have reported no internal control weaknesses. The Department collects and makes available information on grantee obligations and expenditures that is used by Program Offices for monitoring purposes. AIVRS grantees have not appeared on the Department's "excessive drawdown report." The GAPS Drawdown Report indicates those grants that have drawn unusually large proportion of grant funds in any of the first 3 quarters of the grant's current budget period. Program staff then follow up with grantees and are responsible for ensuring that excess cash balances are resolved by the grantee within two weeks after being notified.

Evidence: OCFO/Grants Policy notifies program staff of excessive drawdowns by grantees. Program staff conduct fiscal reviews using information contained in GAPS at six and nine months and prior to making continuation awards. Education Department General Adminsitrative Regulations 34 CFR 80.26.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: RSA has taken steps to address program management deficiencies identified by program staff, grantees, and the program evaluation, including weaknesses in program coordination and the provision of technical assistance to applicants and grantees.

Evidence: The Department identified program implementation inconsistencies. As a result, RSA established a cross-cutting team to address identified deficiencies in the consistency of its policies, technical assistance, and monitoring. The team has also worked to identify outcome measures. Team notes are taken to ensure more consistency in providing consistent guidance to AIVRS projects by staff. There is a plan to hire a program coordinator.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All new awards are based on a competitive process that includes a panel of external peer reviewers, with competitive preference given to existing grantees (see 1.4).

Evidence: 34 CFR 371; Federal Register: March 3, 2004 (Vol. 69, #42) Also see 1.4.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program conducts monthly teleconferences, accessible to all grantees, to discuss issues related to service provision, program requirements, and upcoming activities. On an individual basis, program staff work with individual grantees to follow-up on issues identified in annual reports, request for technical assistance, or issues identifed in the monthly telelconferences. Program staff conduct at least two fiscal reviews per grantee annually. Regional office staff conduct on-site visits to grantees deemed in need. For fiscal issues, also see 3.3 and 3.6.

Evidence: Minutes from monthly teleconferences and notes related to the provision of technical assistance to the group or individual grantees are maintained in the program office. AIVRS Annual Reporting Form

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: At this time, grantee performance information is not transparent nor in a readily available format. However, a web-based system for grantee reporting has been developed and aggregate internal reports are being developed. Program staff plan to identify key data and information that could be posted to the web. The Evaluation of the AIVRS program, which includes performance data, is posted on the Deparment's website.

Evidence: Evaluation of the American Indian VR Services Program (June 2002) www.ed.gov/policy/speced/leg/rehab/eval-studies.html#aivrs

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Department regularly collects timely performance information from program grantees and appears to be on-track to achieve its long term target. However the Department has not yet collected data on AIVRS other long-term measures, the job training common measures.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Department regularly collects timely performance information from program grantees and appears to be on-track to achieve its annual performance targets.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The 2005 Program Performance Plan does not include an efficiency measure. However, the Department plans to include the common efficiency measure for job training programs (cost per participant) in the 2006 Annual Plan. Baseline data is being collected.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: AIVRS grantees' outcome performance compares favorably with other VR employment programs. However, to date, the Department has been unable to provide data on the job training common measures, which will allow for a better comparison. The answer to this question could change to "Yes" when the Department provides the necessary Job Training Common Measures data.

Evidence: The 2002 employment outcome rate for the AIVRS program was 64%, compared to the State VR rate of 60%. Even if the projects were to close more cases, it is likely that the employment outcome rate would remain comparable to the other VR employment programs. Also see 2.2. Evaluation of the American Indian VR Services Program (June 2002) www.ed.gov/policy/speced/leg/rehab/eval-studies.html#aivrs

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Department of Education recently released an independent evaluation (that employed non-experimental direct analysis) of the AIVRS program. This study tracked AIVRS grantees operating in FY 2001. The evaluation provided comprehensive information on the AIVRS program including: characteristics and demographics of the persons served; the services provided; cost-effectiveness of established AIVRS programs; the grantees' vocational practices; and identification of best practices. The evaluation also found that projects generated appropriate levels of successful employment outcomes at reasonable costs.

Evidence: Evaluation of the American Indian VR Services Program (June 2002) www.ed.gov/policy/speced/leg/rehab/eval-studies.html#aivrs