Detailed Information on the
Vocational Rehabilitation Demonstration and Training Programs Assessment

Develop long term measures; examine current annual measures to determine whether they should be maintained or revised in line with the comprehensive plan.

Develop procedures for the review of grantee data, progress and final reports, to improve program performance.

Resolve outstanding issues with the web-based data collection instrument so that it's operating correctly for both grantees and RSA.

Develop comprehensive plan (including a plan for program evaluation) that will establish procedures for identifying multi-year initiatives and annual priority areas.

Measure: The percentage of individuals who were provided employment services through projects and who were placed into employment.

Explanation:Targets were established for the first time in FY 2005. The data reflect information provided by 26 grantees funded through the Special Demonstrations program that use the web-based reporting system and that provide direct services leading to employment outcomes. The percentage of consumers placed in 2004 is based on 1,018 placements and 2,830 served.

Measure: The percentage of referrals from VR State agencies to projects.

Explanation:Targets established for FYs 2001 through 2004 were based on data that has since been corrected. The data now reflect information provided by 26 projects funded through the Special Demonstrations program that use the web-based reporting system and that provide direct services leading to employment outcomes. The FY 2004 data is based on 2,830 individuals served and 890 referred from VR.

Measure: The percentage of referrals to State VR agencies from projects.

Explanation:Targets established for FYs 2001 through 2004 were based on data that has since been corrected. The data now reflect information provided by 26 projects funded through the Special Demonstrations program that use the web-based reporting system and that provide direct services leading to employment outcomes. The FY 2004 data is based on 2,830 individuals served and 261 referred to VR.

Measure: The percentage of projects that met their goals and objectives as established in their original applications. (New Measure, added February 2007)

Explanation:This efficiency measure is designed to determine whether the grantees are providing the services for which they were awarded funding through the competitive process. Determinations will be made through an internal review by RSA staff of final reports, grant files, and applications. RSA will identify whether projects met their original goals and objectives, or goals and objectives as modified during the first year.

Is the program purpose clear?

Explanation: The purpose of the Demonstration and Training Programs, authorized by §303(a) through (d) of the Rehabilitation Act, is to conduct special projects and demonstrations that expand and improve the provision of rehabilitation and other services and to provide training and information to individuals with disabilities to develop skills necessary for the rehabilitation system and process. Sections 303(a), (c), and (d) of the Act authorize demonstration projects to increase client choice, a parent information and training program, and a Braille training program. The main focus of Demonstration and Training Programs, and this PART, is the §303(b) Special Demonstration Programs.

Evidence: Rehabilitation Act, §303(a) through (d). Implementing regulations for Special Demonstration Programs are at 34 CFR 373.

Does the program address a specific and existing problem, interest, or need?

Explanation: By 2020, the Census Bureau predicts that 1 in every 6 Americans??nearly 54 million people, will have a disability. The program addresses the specific need to expand and improve the provision of rehabilitation and other services for individuals with disabilities. State-reported data collected by the Rehabilitation Services Administration (RSA) for the Vocational Rehabilitation (VR) State Grants program indicates that in fiscal year 2003 only 66 percent of all VR State agencies assisted 55.8 percent of individuals receiving services to achieve employment. Special Demonstration Programs address the need for VR State agencies and Indpendent Living (IL) programs to examine new research findings, experiment with new practices, and to implement models of service delivery found to increase employment and IL outcomes for individuals with disabilities. For example, supported employment (e.g., on-going support services provided by VR State agencies for up to 18 months for individuals who, because of their disabilities, need intensive support in order to perform work) began as an innovative practice in the VR field that was demonstrated through grants under section 303 of the Rehabilitation Act. As we now know, supported employment services are routine and provided by all VR State agencies. Special Demonstration Programs are necessary for the VR system to appropriately respond to the latest service delivery needs and to the changing needs of its consumers.

Evidence: Data for the VR State grants program, Independent Living programs, and other programs authorized under the Act show that not all goals and targets have been met. Improvements in the provision of rehabilitation and other services authorized under the Rehabilitation Act are required.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Special Demonstration programs is the only Federal program with the flexibility to support a broad range of activities aimed at increasing employment and IL outcomes for individuals with disabilities and at furthering the purposes of the Rehabilitation Act. This flexibility allows program staff to minimize redundancies and duplication of effort, by planning for and coordinating priority areas with other federal agencies. In addition, the program is designed to promote collaborative working relationships among funded projects and rehabilitation service providers, local and State agencies, community based organizations, one-stop centers, employers, and business groups. Projects are required to report on these relationships annually. Statutorily, some overlap exists between RSA and the Office of Special Education's (OSEP) Braille Programs. However, RSA only funds two such projects and OSEP does not currently fund any.

Evidence: Rehabilitation Act, §303(b) and 34 CFR 373. The menu-driven statute and implementing regulations allow maximum flexibility in targeting funds for particular types of projects (such as model demonstrations, technical assistance, system change, special studies, etc.) and to particular entities (such as State VR agencies, community rehabilitation programs, public or non-profit agencies and organizations, and if appropriate for-profit organizations, etc.)

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program's authority has no limitations and can be used to conduct any activities that support the Rehabilitation Act.

Evidence: The program is designed to expand and improve the provision of rehabilitation and other services authorized under the Rehabilitation Act.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Under the authority for this program, the Departmnet of Education (ED) can use funds to effectively target any service need of individuals with disabilities (or particular groups within this population) to increase their employment, community integration, or independent living.

Evidence: Rehabilitation Act, §303(b) and 34 CFR 373. The program's authority has no limitations and can be used to conduct any activities that support the Rehabilitation Act.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: No long-term performance measures have been developed to date, but RSA will develop long-term measures from either their forthcoming long-range plan or their annual measures. While these measures tell RSA something about the quality and success of the projects it is funding in relation to the goals of the VR State Grants program, they do not provide much information about accomplishing the overall goals of the program. These goals should be reflected in a long-range plan for the program, and the priority areas identified in such a plan should be the focus of one or more long-term measures.

Evidence: RSA spending plan for this program (annual basis); and ED's PPMD.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: While RSA has established reasonable targets for its current annual measures it has not established a long-term measure that focuses on what RSA hopes to achieve over time.

Evidence: ED's PPMD; the web based Annual Progress Reporting Form For Special Demonstration Grants

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: RSA's Special Demonstration Programs has 3 annual performance measures -- (1) the percentage of individuals who were provided employment services through projects and who were placed into employment; (2) the percentage of referrals to State VR agencies from projects; and (3) the percentage of referrals from State VR agencies to projects. The second and third measures attempt to assess the strength of the relationship established between projects and VR service providers. Although the program does not yet have long-term measures, the annual measures provide data that can be used to determine whether the Special Demonstration Programs funds projects that support the purpose of the program. Specifically, whether projects have been instrumental in expanding and improving the provision of rehabilitation services that lead to an employment outcome.

Evidence: ED's PPMD; the web based Annual Progress Reporting Form For Special Demonstration Grants

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baseline for the annual measures were established using fiscal year 2001 data. RSA has established annual targets for its performance indicators. However, targets established for the two measures related to referrals to and from VR for fiscal years 2001 through 2004 were based on data that has since been corrected. Targets set for these fiscal years do not correspond to actual performance, and do not appear ambitious. RSA will use a three-year average to set targets for future years. The target for fiscal year 2006 and 2007 is 34 percent, which was reached by averaging the actual performance over the past 3 years (FYs 2002 though 2004). Also see discussion in 2.2.

Evidence: ED's PPMD.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: During grantee teleconferences, in responses to requests for technical assistance, and in director's conferences, RSA emphasizes the importance of performance measures (specifically employment outcomes) under the Special Demonstration Programs. In fact, RSA asks all grantees to identify employment outcomes resulting from project activities (including those projects that did not propose to directly place disabled individuals into competitive employment).

Evidence: RSA teleconferences; RSA's web-based data collection instrument.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: While discrete project evaluations have been conducted, RSA has not developed a comprehensive program evaluation. See also see the response in 2.8 regarding program evaluation.

Evidence:  

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests provide information on activities supported with program funds and include a discussion on the data reported by grantees related to the performance indicators. However, RSA does not have an overall comprehensive plan (other than the required spending plan) that details what specific projects would be funded or not funded with budget changes and RSA has not systematically evaluated how its budget structure reflects program goals.

Evidence: RSA's spending plan for Special Demonstrations; OSERS Budget Request; and Congressional Justifications.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program lacks any sort of strategic process for identifying and establishing priority areas. As described above, the program's strength is in its flexibility to target funds in any direction that supports the purposes of the Act, but a long-range plan must be developed to determine where best to direct these funds and how they will target identified priority service needs. RSA will develop a comprehensive long-range plan (including a plan for program evaluation) that will establish procedures for identifying multi-year initiatives and annual priority areas. Priorities will be established based on: relevant requirements in the Strategic Performance Plan for the VR program; evaluation, data analyses, and research findings; results from Program Improvement activities; and consistency with the Department's goals and objectives. In addition, the comprehensive plan will establish timelines by which long-term performance measures will be established. The collection of data through a uniform web-based instrument was a significant positive step to address the fundamental lack of data with these grantees, and the development of an annual spending plan greatly assists in annual financial planning.

Evidence: Lack of comprehensive long-range plan for funding priorities.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Although RSA collects timely performance information, the program does not systematically use the information to improve program performance.

Evidence: In FY 2001 the Rehabilitation Services Administration implemented a web-based data collection instrument that collects data on grantee program activities. The instrument allows for the annual evaluation of: 1) the grantees' progress in achieving the objectives in their approved applications; 2) the effectiveness of the projects in meeting the purposes of the program; and 3) the effect of the projects on the participants being served by the project. Program staff uses this data to determine if grantees are making substantial progress prior to awarding continuation awards.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to the EDPAS system which links employee performance to success in meeting the goals of the Department's Strategic Plan. In general, managers are provided individual performance agreements where they are given responsibility for achieving relevant action steps outlined in the Strategic Plan. These action steps and other items included in managers' performance agreements are designed to measure the degree to which a manager contributes to improving program performance. Specifically, the Chief of the unit that administers this program has EDPAS criteria holding him accountable for the program. The criteria for the Chief are as follows: 1. Demonstrates program knowledge and expertise in assigned areas; assures the timely and accurate collection of performance indicator data/GPRA data and makes appropriate future program decisions based on these outcomes. 2. Ensures a culture of accountability through effective oversight and management of employees' assigned projects; assures staff conduct comprehensive teleconferences with all grantees to assess whether grantees are "on target" with stated goals, objectives and accomplishments and, if not, appropriate action is taken to address these deficiencies. 3. Assures staff participate in the Department's E-government by implementing initiatives such as E-applications, E-grants and E-performance reports as appropriate. In addition, the Chief is responsible to assure that all grantees within the branch are provided with appropriate and timely technical assistance to enhance services to individuals with disabilities.

Evidence: Internal records; EDPAS

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Grant funds are obligated in a timely manner and in accordance with established program schedules, including an annual spending plan. Through general oversight and technical assistance activities, grantees are reminded that they must comply with all post-award requirements for financial management set forth in the Education Department General Administrative Regulations (EDGAR) sections 74.20 through 74.28 (Financial and Program Management) and sections 80.20 through 80.26 (Financial Administration). In addition, the program is covered by the Single Audit Act, which requires the submission of an annual independent audit by grant recipients that receive an aggregate of $500,000 or more in Federal funds. These audits examine whether or not grantees spend and account for Federal funds properly, and in accordance with appropriate OMB cost principles, program statute and regulations.

Evidence: RSA's annual spending plan for Special Demonstrations; EDGAR parts 34 CFR 74.20 through 74.28 and 80.20 through 80.26. Office of the Chief Financial Officer (OCFO)/Grants Policy notifies program staff of excessive draw downs by grantees. Several times a year, program staff review the financial information in the Grant Administration and Payment System (GAPS). GAPS is used by the Department to track the financial activities of a grant from initial obligation of funds by ED, draw down of funds by grantee, and final settlement of grant. In addition, GAPS maintains demographic information on the grantees.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has not yet instituted an efficiency measure or procedures to measure and improve cost efficiency in program execution. However, as part of the President's Management Agenda, the Department is implementing "One-ED" -- an agency-wide initiative to re-evaluate the efficiency of every significant business function, including the development of unit measures and the consideration of competitive sourcing and IT improvements.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program has collaborated and coordinated efforts with other related programs in the past. For example, the Systems Change priority was established through collaboration with other Federal agencies. In addition RSA has entered into, or was the recipient of, interagency agreement funds that show on-going collaboration and joint funding by other Federal agencies such as the Departments of Labor, HHS, and Transportation. Increased collaboration of this nature is likely to occur as RSA identifies priority areas that may interest other Federal agencies. When appropriate, RSA will seek to collaborate with other agencies in relation to the development of funding priorities and program improvement.

Evidence: RSA's transportation demonstrations; ED/HHS jointed funded Systems Change grants; and ED's contribution to Labor's mentoring projects.

Does the program use strong financial management practices?

Explanation: Departmental guidelines for financial management are followed under this program. At the Federal level a system of checks and balances exists for this program that guards against any potential misuse of funds and reduces the possibility of errors in awarding grants. At the grantee level, projects are monitored for excessive drawdown activity using the Department's GAP system at six and nine month intervals, and prior to the obligation of a continuation award. Grantees with excessive drawdowns are required to resolve the issue within two weeks after being notified. Grantees with audit findings, and who violate the financial management requirements set forth in EDGAR, are required to submit corrective action plans for review and approval by the program office that identifies a date by which the issue will be resolved. Program will enforce any of the options available under EDGAR 74.62 (Enforcement) and EDGAR 80.12 (Special grant or subgrant conditions for "high-risk" grantees) as appropriate. In addition to monitoring excessive drawdowns, program staff monitor regular grantee drawdown activity and balances to ensure that outstanding balances do not remain unspent by the end of a budget period. Unspent balances, and any incomplete activities associated with those balances, are addressed with the grantee for appropriate resolution.

Evidence: Office of the Chief Financial Officer (OCFO)/Grants Policy notifies program staff of excessive draw downs by grantees; EDGAR.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: RSA does not have a system for identifying and correcting program management deficiencies and has not taken a systematic approach to making necessary corrections within an agreed upon timeframe. However, RSA has by established a monitoring plan with grant procedures for use in administering its programs. Although grant monitoring is a grant management responsibility, the establishment of the monitoring plan created consistency among RSA program staff with performance timelines for monitoring and oversight and the strengthening of the financial grant review process. In addition, all monitoring plans and staff oversight duties are being reviewed as RSA abolishes its regional offices. Duties (including the monitoring of grants) previously assigned to regional staff will be shifted to central office staff. The realignment of staff and duties may greatly reduce management inconsistencies.

Evidence: ED's elimination of RSA's regional offices; the Special Demonstration program's web-based annual reporting instrument; and RSA's monitoring plan.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Projects awarded direct funding through Congressional earmarks have become an annual practice in this program. In fiscal year 2005, $6,973,760 (27 percent) of the Demonstration and Training program was directed to support 20 specific projects. However, the Rehabilitation Act requires that all new awards must be based on a competitive process that includes a panel of external peer reviewers. The regulations for this program (34 CFR 373.11) require that RSA only fund those applications in response to competitions announced in the Federal Register.

Evidence: Annual appropriations bills and reports; Rehabilitation Act (§301(b)) states that all grants and contracts awarded under title III must be made on a competitive basis; Special Demonstration program regulations in 34 CFR 373.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Grantees are held accountable for costs and for performance results through the analysis of data collected during conference calls, via the web based annual performance report, technical assistance sessions at the annual project director conferences, and during teleconferences focusing on GAPS fiscal reviews. Program staff work towards helping ensure grantee success by: 1) advising grantees on issues applicable to appropriate OMB Cost Principles; 2) advising grantees on issues applicable to program regulations and program statute; 3) advising grantees on issues applicable to the Education Department General Administrative Regulations (EDGAR); and 4) working closely with high-risk grantees. Those grantees experiencing problems, and those that have been designated "high-risk" are required to submit improvement plans and can have grants reduced or discontinued for serious or persistent failures to comply or meet performance targets. Monthly teleconferences accessible to all grantees to discuss issues related to service provision, program requirements and upcoming activities are also held. However, participation is not required, and the numbers of grantees participating varies depending on the topic(s) being discussed during the teleconference. In the re-structured RSA, state teams will be required to know all resources and grants in a given state and facilitate ways to coordinate them. RSA will include reviews of discretionary grants on site visits to state agencies as appropriate and as resources allow.

Evidence: Grant files, internal records, minutes from teleconferences, and requests for technical assistance.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: At this time, grantee performance information is not transparent nor in a readily available format. However, a web-based system for grantee reporting has been developed. The existing web based system will not be accessible to the public. It is a grantee reporting system that is only accessible to grantees and program staff. However, RSA can extract data from that system and transfer it to the Department's internet web site for access by the general public.

Evidence: The web based Annual Progress Reporting Form For Special Demonstration Grants

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Department regularly collects timely annual performance information from program grantees and grantees may meet fiscal year 2006 targets. However, RSA has not established long-term goals that are based on a systematic priority-setting process or a long-range plan.

Evidence: Lack of multi-year plan; and ED's PPMD: the web based Annual Progress Reporting Form For Special Demonstration Grants.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Data for the employment measure fluctuates from year to year. In fiscal year 2001, 14.20 percent of individuals were reported as placed into employment; 27.86 percent in fiscal year 2002; 38.62 percent in fiscal year 2003; and 35.97 percent in fiscal year 2004. Targets established for the two measures related to referrals to and from VR for fiscal years 2001 through 2004 were based on data that has since been corrected. Therefore, targets set for these fiscal years do not correspond to actual performance,and cannot be used to deteremine whether achievement of these goals has been accomplished. The data now reflect information provided by 26 projects funded through the Special Demonstrations program that use the web-based reporting system and that provide direct services leading to employment outcomes.

Evidence: ED's PPMD: the web based Annual Progress Reporting Form For Special Demonstration Grants.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: RSA has not developed an efficiency measure for this program.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Special Demonstrations program is the only Federal program authorized with this flexibility and with the ability to support this range of activities, but it could be compared to other demonstration programs in government. However, no systematic evidence has been collected to compare RSA's demonstration activities to other Federal demonstrations.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: No independent evaluations have been conducted.

Evidence: