Detailed Information on the
Projects with Industry for People with Disabilities Assessment

Implement a plan to improve grantee data collection and reporting.

Working with Congress to eliminate the program.

Revise measures to be comparable with other job training programs and develop a strategy for collecting data to support the Administration??s Job Training common measures

Improve use and transparency of project data to manage the program, including posting summary analyses and key data on the web.

Develop and implement a plan to meet the program??s statutory requirement for on-site compliance reviews.

Measure: The percentage of individuals served who are placed in competitive employment.

Explanation:The numerator is all persons placed in competitive employment during the reporting period. Beginning with FY 2005, the denominator is the total number of persons served by the PWI project during the reporting period."Persons served" means an individual for whom services have been initiated with the objective that those services will result in a placement in competitive employment.

Measure: Average increase in weekly earnings of participants placed into competitive employment

Explanation:The increase in average weekly earnings is calculated by subtracting earnings for the week prior to entry into the PWI project from the earnings for the first week of employment

Measure: Percentage of previously unemployed individuals served who were placed in competitive employment.

Explanation:Prreviously unemployed is defined as individuals who were continously unemployed for at least six months at the time of project entry.

Measure: The percentage of participants exiting the program who are placed in competitive employment

Explanation:The numerator is all participants placed in competitive employment during the reporting period. The denominator is the number of participants who exited the program during the reporting period. Data collection was revised to include necessary data elements. Baseline data will be collected in FYs 2005 and 2006.

Measure: The percentage of projects whose annual cost per placement is no more than $11,000.

Explanation:Cost per placment is calculated by dividing total federal grant funds by the total number of individuals placed in employment during the reporting period.

Measure: Job Training Common Measure: Annual cost per participant

Explanation:Annual cost per participant is calculated by dividing total federal grant funds by the total number of persons served during the reporting period. Data collection instrument was revised in 2005 to provide data on the total number of individuals served. Data will not be available until March 2006.

Measure: Job Training Common Measure: Entered Employment - Percentage employed in the first quarter after program exit.

Explanation:Numerator: Of those who are not employed at registration, the number of adults who have entered employment by the end of the first quarter after exit. Denominator: Of those who are not employed at registration, the number of adults who exit during the quarter. This measure has not been implemented because program grantees do not have acces to UI data system. ED is currently conducting a study to assess the capacity of these grantees to collect and report Job Training Common Measures data.

Measure: Job Training Common Measure: Retention in Employment - Percentage of those employed in the first quarter after exit that were still employed in the second and third quarter after program exit.

Explanation:Numerator: Of those who are employed in the first quarter after exit, the number of adults who are employed in the second and third quarter after exit. Denominator: Those who are employed in the first quarter after exit. This measure has not been implemented because program grantees do not have acces to UI data system. ED is currently conducting a study to assess the capacity of these grantees to collect and report Job Training Common Measures data.

Measure: Job Training Common Measure: Increase in Earnings - Percentage change in earnings: (i) pre-registration to post program; and (ii) first quarter after exit to third quarter.

Explanation:Numerator 1:Participant's earnings first quarter after program exit minus participant's earnings two quarters prior to registration. Numerator 2:Participants earnings third quarter after program exit minus participant's earnings first quarter after program exit. Denominator 1: Participant's earnings two quarters prior to registration. Denominator 2: Participant's earnings first quarter after program exit. This measure has not been implemented because program grantees do not have acces to UI data system. ED is currently conducting a study to assess the capacity of these grantees to collect and report Job Training Common Measures data.

Measure: The percentage of Projects With Industry projects who demonstrate an average annual cost per participant of no more than $4,500. (New measure, added February 2007)

Explanation:Cost per participant is calculated by dividing total annual federal grant funds by the total annual number of individuals served by the program.

Is the program purpose clear?

Explanation: Projects with Industry (PWI) aims to create and expand job opportunities for individuals with disabilities in the competitive labor market and engage private industry as partners in the rehabilitation process. However, some inconsistency exists between the statutory purpose and the program requirements, particularly as it relates to job training. For example, although one program purpose is to create job and career readiness and training programs, the provision of job training is no longer a required project activity.

Evidence: Rehabilitation Act of 1973, Title VI, Part A Program Regulations at 34 CFR 379.1

Does the program address a specific and existing problem, interest or need?

Explanation: PWI addresses the specific need to help individuals with disabilities obtain competitive employment (i.e., employment in the competitive labor market at or above minimum wage), consistent with 34 CFR 379.5(b)(2). The local PWI project provides an additional resource for individuals who cannot obtain services from their State vocational rehabilitation (VR) agency.

Evidence: In FY 2003, about 49 percent of State VR agencies were operating under an "order of selection" because they could not serve all eligible individuals. States operating under an order of selection must give priority to serving individuals with the most significant disabilities. According to the 1998 Chartbook on Work and Disability, 26 percent of individuals with significant disabilities are employed, as compared to 77% for those without significant disabilities. The employment rate was 82% for the non-disabled population. Chartbook on Work and Disability, NIDRR 1998 Rehabilitation Act of 1973, Section 101(a)(5)and VR Program Regulations at 34 CFR 361.36

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: PWI and the much larger VR State Grants serve the same target populations. The services provided by the PWI program may be provided by the VR program. PWI does differ from VR State Grants in that it primarily focuses on job placement and private sector input through the Business Advisory Council (BAC). However, two of the major functions of the BAC (local job/career identification and the corresponding needed skills) are now functions of the local workforce investment board (WIB) under the Workforce Investment Act of 1998 (WIA). In addition, some local communities have private and public job training and placement programs that offer some similar services to PWI projects. These projects are typically supported by local public funding, private and public grants, and private donations.

Evidence: An independent evaluation of the PWI program found that individuals served by the PWI program do not differ much from those served by VR agencies at the aggregate program level. Eligible individuals tend to receive training and services from the VR State agency which then refers individuals to a local PWI project for specialized placement assistance. In addition, the evaluation found that PWI projects vary dramatically in terms of the extent to which they have strong private sector involvement. The final report states that the statutory "role of the BAC may not be reasonable given the voluntary nature of BAC services, ongoing changes in BAC membership, and the infrequency with which most BACs convene." Evaluation of the Projects With Industry Program www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: About half of the individuals served by PWI grantees also receive services from the VR State Grants program. Eligible individuals tend to receive training and services from the VR State agency and specialized placement assistance from PWI, although VR Agencies can provide this service too. As a result of this duplication, the government might be able to get similar outcomes by expending fewer resources through a different mechanism. Although PWI has an impact at the local level, some questions persist as to the appropriateness of the program structure given changes in program requirements and the establishment of local WIBs and one-stop centers under WIA.

Evidence: The PWI evaluation found no statistically significant difference between PWI participants and VR consumers with respect to the percentage who obtained employment or with regard to the average earnings of those who obtained employment following program participation. In addition, the evaluation found that the design or approach evident at any one project often reflected the configuration of local resources, as well as the broader purposes pursued by the host grantee's organization. Typically, a PWI project is one of several programs operated by a host organization. The specific role of the PWI project at many, especially larger, grantee organizations is shaped by the other programs available at the host organization.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: At the program level, resources target individuals with disabilities. However, some of the beneficiaries would have access to similar services through the VR program if the PWI project did not exist. Although the programs have similar eligibility requirements, not all individuals who are eligible for VR are served because of limited resources.

Evidence: State VR agencies that are not able to serve all eligible individuals must implement an "order of selection." (See 1.2) Rehabilitation Act of 1973, Section 101(a)(5)and VR Program Regulations at 34 CFR 361.36

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: A long-term Government Performance and Results Act (GPRA) measure for this program was established as part of the FY 2005 Program Performance Plan. For FY 2006, the long-term measure will be changed to "the percentage of individuals served who are placed in competitive employment," which will more accurately reflect the statutory purpose. As part of the Job Training Common Measures initiative, the program has adopted new long-term measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, than its current measures and (2) facilitate comparisons with similar programs. ED is conducting a study to assess how to implement the common measures.

Evidence: See Measures section of PART Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Achievable targets and timeframes have been established for the long-term measure based on analysis of historical performance data. Targets are set based on program data analysis and expected project outcomes. Targets take into consideration factors such as grantee experience. For example, new grantees are expected to produce fewer outcomes in the first year of the grant. The long-term performance measure is computed by comparing the number of individuals placed in employment with the number of persons served by the project. The computation for persons served only includes individuals entering the program during the reporting period and does not include individuals receiving services who entered the program in the previous reporting period.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Annual and long-term targets, and timely annual data are available. All three measures are discrete and quantifiable. The outcome measure demonstrates the program's progress in meeting the long term-goal. As part of the Job Training Common Measures initiative, the program has adopted new long-term measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, than its current measures and (2) facilitate comparisons with similar programs. The Department is conducting a study of common measures to assist in their implementation and assess the capacity of grantees to collect and report these data.

Evidence: See Measures section of PART Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines were established in prior years and achievable targets set to ensure continued program improvement. Targets are set based on analysis of program data and anticipated project outputs. Project expectations increase over time as project staff gain knowledge and experience and build project capacity.

Evidence: Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: PWI grantees report on compliance with program standards and indicators, as defined by program regulations. In order to receive continuation funding, grantees must demonstrate compliance with the regulatory performance indicators by submitting data for the most recent project year. If a grantee does not demonstrate compliance, the grantee has an additional opportunity to demonstrate results by submitting data from the first six months of the current project year.

Evidence: Relevant program regulations are contained in 34 CFR 379.44 and Subpart F.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Research Triangle Institute (RTI), an independent contractor, recently completed a program evaluation that used consumer service records, interviews, surveys, and focus groups to examine the role and performance of the PWI program as one component of the broader set of employment-related services available to individuals with disabilities. RTI also focused on the extent to which PWI projects fulfilled their intended goal to create and expand job opportunities for individuals with disabilities at the project level. Previous evaluations of the PWI program were conducted in 1985 and 1994. A subsequent evaluation will be conducted prior to the next reauthorization of the Rehabilitation Act.

Evidence: The Final Report of the Evaluation of the Projects With Industry Program was released in December 2003. www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: In fiscal year 2003, the Administration launched an initiative to streamline job training programs and eliminate duplicative and overlapping programs. No funds were requested for PWI for FYs 2003, 2004, or 2005 because the Administration believes that the PWI program is such a program.

Evidence: Budget Requests for fiscal years 2003 through 2005.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Currently there is no strategic planning effort for this program. As part of the Job Training Common Measures initiative, the program has adopted new long-term measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, than its current measures and (2) facilitate comparisons with similar programs. The Department is conducting a study of common measures to assist in their implementation and assess the capacity of grantees to collect and report these data.

Evidence: Assisting Grantees with Common Measures ED01CO0052/0011

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Department collects timely, but not credible annual performance information from PWI grantees. This information includes data to assess grantee performance on the regulatory performance indicators. A recent evaluation of the program found that "PWI projects' data collection practices continue to undermine the program's ability to accurately measure its achievements." In addition, the data reported by grantees contains numerous errors. A web-based system for grantee reporting is currently being developed for implementation with 2004 data collection, which should help in reducing reporting errors. The program primarily uses the performance indicator data, as required by law, to make decisions about continuation awards. Program staff also use the data to follow-up with grantees that have demonstrated poor performance on specific indicators and provide technical assistance.

Evidence: PWI Compliance Indicator and Annual Evaluation Plan Reporting Form, Office of Management and Budget (OMB) Number 1820-0631; Program Regulations at 34 CFR 379 SUBPART F - "What Compliance Indicator Requirements Must a Grantee Meet to Receive Continuation Funding;" and, Final Report of the Evaluation of the Projects With Industry Program, December 2003. www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, the Department has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps -- hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its Senior Executive staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. ??The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Federal funds are obligated in a timely manner. This program is covered under the Single Audit Act. Recipients that receive an aggregate of $500,000 or more in federal funds are required to submit to ED an annual independent audit. The purpose of the audit is to demonstrate that the entity has a financial system in place and that federal funds and spent and accounted for properly, in accordance with OMB Circular A-133.

Evidence: Education Department General Administrative Regulations 34 CFR 80.26.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The PWI performance indicators include an efficiency measure (average cost per placement). Data on average cost per placement is reported by all grantees. The program is also in the process of changing its data collection to enable it to collect data for the job training common efficiency measure. Program staff follow-up with grantees who failed to meet this indicator on a monthly basis to provide technical assistance in this area.

Evidence: Performance indicators are contained in program regulations at 34CFR 379.53.

Does the program collaborate and coordinate effectively with related programs?

Explanation: At the Federal level, collaboration is limited. However, coordination and collaboration occurs at the local level with State VR agencies and with other entities such as one-stops, WIBs, welfare recipients, high schools and institutions of higher education. In some projects, representatives of these entities also serve on the BAC, which acts as a forum for such coordination.

Evidence: In past grant competitions, invitational priorities have been used to invite applicants to submit proposals for projects that collaborate with one-stop partners under the Workforce Investment Act and projects that collaborate with local school systems in serving youth who are transitioning from school to work.

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. In addition, Rehabilitative Services Administration (RSA) -- PWI's parent office -- staff monitor grantee obligations and the drawdown of funds. PWI grantees have not appeared on the Department's "excessive drawdown report." The GAPS Drawdown Report indicates those grants that have drawn unusually large proportion of grant funds in any of the first three quarters of the grant's current budget period. Program staff then follow up with grantees and are responsible for ensuring that excess cash balances are resolved by the grantee within two weeks after being notified.

Evidence: Office of the Chief Financial Officer/Grants Policy notifies program staff of excessive drawdowns by grantees. Program staff conduct fiscal reviews using information contained in the Grant Administration and Payment System (GAPS). GAPS is used by the Department to track the financial activities of a grant from initial obligation of funds by ED, draw down of funds by grantee, and final settlement of grant. In addition, GAPS maintains demographic information on the grantees. PWI Teleconference Monitoring Summary and Education Department General Adminsitrative Regulations 34 CFR 80.26.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has not developed a system of evaluating program management and correcting identified deficiencies. To address identified problems with data credibility, program staff are planning to develop guidance to improve grantee case file documentation and data collection and reporting.

Evidence: Final Report of the Evaluation of the Projects With Industry Program, December 2003. www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All new awards are made based on a competitive process that includes a panel of external peer reviewers. Continuation awards for the third and subsequent years are dependent upon the grantees' performance on the compliance performance indicators.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Commissioner of RSA is required by statute to conduct annual onsite compliance reviews of at least 15 percent of grantees. Regional and headquarter staff have had difficulty meeting this requirement. However, program staff review grantees performance on at least a quarterly basis to ensure that grantees are progressing toward achieving their project goals.

Evidence: Onsite monitoring requirements in Rehabilitation Act of 1973, Title VI, Section 611(f)(3) Revised On-Site Monitoring Procedures For PWI, 2000 PWI Teleconference Monitoring Summary

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects grantee performance data on an annual basis, but is not readily available to the public. PWI data is contained in the Annual Report to Congress on the Rehabilitation Act. However, historically there has been a significant delay in release of this Annual Report. The recent Evaluation of the PWI program, which includes performance data, is posted on the Department's web site. In addition, the Department of Education (ED) is developing a department-wide approach to improve the way programs provide performance information to the public. In 2004, ED will conduct pilots with selected programs to assess effective and efficient strategies to share meaningful and transparent information.

Evidence: Final Report of the Evaluation of the Projects With Industry Program, www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: A long term measure was only recently established. The long term measure is "the percentage of individuals served who are placed in competitive employment." The measure had previously been an annual measure. There had been an upward trend in the data until 2003 when there was a marked decrease in performance.

Evidence: See Measures section of PART Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program (including program partners) achieve its annual performance goals?

Explanation: With the exception of 2003, the program has met its performance goals for its three GPRA measures since 2000. In FY 2003, The program met the increase in earnings performance goal. However, performance declined for the other two indicators (percentage placed in employment and percentage of previously unemployed placed in employment) and the annual goal was not met. FY 2003 was the first year of operation in the grant cycle.

Evidence: An analysis of the data indicates that there is a dip in program performance at the onset of each new grant cycle, despite that fact that the majority of the current grantees were grantees during the previous grant cycle. See Measures section of PART Program Performance Plan 2005 www.ed.gov/about/reports/annual/2005plan/program.html

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The average cost per placement and average cost per served have continued to increase each year beyond the rate of inflation.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The recent PWI program evaluation states, "the outcomes of the PWI program, with respect to the percentage of persons served who exited into employment and the average hourly earnings of those individuals, are comparable to those of the VR services program." However, it is difficult to compare PWI performance with similar measures in other vocational rehabilitation employment programs because PWI's employment measure is calculated using a different denominator. In calculating the placement rate, PWI uses a similar numerator (individuals who maintain employment for 90 days). However, PWI uses the number of individuals served as the denominator where the other VR programs use individuals who exit the program. The program will be implementing the job training common measures that will provide for improved comparability across job training programs.

Evidence: Evaluation of the Projects With Industry Program, page ES- 9 www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In the aggregate level, the program generally appears to be effective in placing individuals with disabilities in competitive employment. However, the recent evaluation points out that "PWI projects' data collection practices continue to undermine the program's ability to accurately measure its achievements." In addition, there is significant variability in project performance. However, employers interviewed in the PWI study did report that they preferred referrals from PWI projects (as opposed to the VR program) because the projects engaged in more follow-up.

Evidence: Evaluation of the Projects With Industry Program, page ES- 9 www.ed.gov/policy/speced/leg/rehab/eval-studies.html#pwi