Detailed Information on the
Work Incentive Grants Assessment

Working with the Congress to terminate this program, which has successfully completed its mission of demonstrating various approaches to make One-Stop services accessible to people with disabilities.

Monitoring closely the number of individuals with disabilities served by the One-Stop Career Centers to assess access by and service to participants with disabilities.

Completing an independent evaluation of program results.

Adopting efficiency measures that are linked to performance outcomes, account for all costs, and facilitate comparisons across Department of Labor training and employment programs.

Measure: Exiters with disabilities

Explanation:The percent of participants served by the workforce investment areas receiving grant funds who are persons with disabilities One of the primary goals of the DPN grants is to improve the capacity of the workforce investment system to increase the workforce participation and career opportunities of people with disabilities. This measure indicates the effect of DPN outreach and capacity building efforts on expanding the system's services to persons with disabilities. Additionally, as a basis for comparison, DOL analyzes differences in service levels to persons with disabilities between those areas receiving grants and those areas not receiving grants. The measure is calculated by the following method: total number of persons with disabilities exiting workforce investment areas receiving DPN grants divided by the total number of exiters from those workforce areas receiving DPN grants, multiplied by 100. This measure was adopted in PY 2003. Experience shows actual performance results for this measure tend to fluctuate from year to year, often due to the inclusion of new workforce areas receiving DPN grants. A one percent target increase for PY 2007 would equate to an estimated 411 more people with disabilities exiting workforce investment areas receiving DPN grants over PY 2006 results, if the number of people exiting from those workforce areas receiving DPN grants in PY 2007 remained the same as PY 2006.

Measure: Entered Employment Rate.

Explanation:Percent of participants employed in the first quarter after exit. This is a Federal job training program common measure, which enables the Disability Program Navigator (DPN) grants to describe in a similar manner the core purposes and results of the program compared to other education, job training, and employment service programs. For example, while the target population for the Disability Program Navigator (DPN) grants is persons with disabilities, the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures how many participants got a job according to the following formula: Of those who are not employed at the date of participation, the number of adult participants who are employed in the first quarter after the exit quarter divided by the number of adult participants who exit during the quarter. This definition for entered employment was adopted in PY 2003. Experience shows actual performance results for this measure tend to fluctuate from year to year. A one percent target increase for PY 2007 would equate to an estimated 139 more people with disabilities in the workforce investment area receiving DPN grants who enter employment in the first quarter after exit in PY 2007 over PY 2006 results, if the number of people exiting in PY 2007 remained the same as PY 2006.

Measure: Employment Retention Rate.

Explanation:Percent of participants employed in the first quarter after exit still employed in the second and third quarters after exit. This is a Federal job training program common measure, which enables the Disability Program Navigator (DPN) grants to describe in a similar manner the core purposes and results of the program compared to other education, job training, and employment service programs. For example, while the target population for the Disability Program Navigator (DPN) grants is persons with disabilities, the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures how many participants retained their employment once placed in a job, according to the following formula: Of those who are employed in the first quarter after the exit quarter- The number of adult participants who are employed in both the second and third quarters after the exit quarter divided by the number of adult participants who exit during the quarter. This definition for employment retention was adopted in PY 2005, and made the measure more challenging by including an additional quarter in which a participant needs to be employed. Experience shows actual performance results for this measure tend to fluctuate from year to year. A one percent target increase from PY 2006 to PY 2007 would equate to an estimated 118 more people with disabilities in a workforce investment area receiving DPN grants who retained employment in the second and third quarters after exit in PY 2007 over PY 2006 results, if the number of people exiting in PY 2007 remained the same as PY 2006.

Measure: Average earnings.

Explanation:The average six-month earnings. This is a Federal job training program common measure, which enables the Disability Program Navigator (DPN) grants to describe in a similar manner the core purposes and results of the program compared to other education, job training, and employment service programs. For example, while the target population for the Disability Program Navigator (DPN) grants is persons with disabilities, the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures participants' average six-month earnings once placed in a job (note: the average earnings for a year can be obtained by doubling the performance measure result), according to the following formula: Of those participants who are employed in the first, second, and third quarters after the exit quarter: Total earnings in the second quarter plus total earnings in the third quarter after the exit quarter divided by the number of participants who exit during the quarter. The common measure definition for average earnings was adopted in PY 2006, while prior year results were for a different earnings measure tracking earnings change. In PY 2008, participants evaluated for the earnings measure are expected to earn 2.5% more per year than participants represented in the PY 2007 result. This means that an individual would earn $20,408 annually, $1,701 monthly, or $425 on a weekly basis.

Measure: Cost per registrant with disabilities in the workforce investment areas funded by WIG

Explanation:The average cost per registrant is calculated by dividing the total annual appropriation by the total annual number of individuals with disabilities served by Workforce Investment Act (WIA) programs in the workforce areas funded by the Work Incentive Grants.

Is the program purpose clear?

Explanation: The purpose of the Work Incentive Grants (WIG) program is to increase the labor force participation and career advancement of persons with disabilities. The program has no explicit congressional authorization. It has been funded with annual appropriations since FY 2000 under the general authority of the Wagner-Peyser Act of 1933, which authorizes a national employment system. Since the WIG program was created, it has tried to assure that the One-Stop Career Centers -- the service delivery mechanism for the workforce development system -- serves the needs of individuals with disabilities.

Evidence: Element 1 Evidence at: www.doleta.gov/disability/partevidence/ WIA section 188 regulations GAO Report TEIN No. 4-00 TEIN No. 25-99 TEIN No. 21-99 WIG Solicitations FY 00 FY 02 FY 03 FY 04 DPN Solicitations FY 03 FY 04

Does the program address a specific and existing problem, interest, or need?

Explanation: Work Incentive Grants were established to address concerns about the ability of the One-Stop Career Center system, which was established under the Workforce Investment Act of 1998, to meet the needs of people with disabilities. There are more than 54 million Americans living with a disability, representing 20 percent of the U.S. population and more than 17 percent of the labor force (2000 Census Bureau data). Almost 70 percent of persons with disabilities are not working. Most would like to work and pursue a career that would lead to self-sufficiency. WIGs help meet the first objective of the President's New Freedom Initiative, which is to increase access to assistive and universally designed technologies and universal employment and training services. WIGs also support Executive Order 13217, which directs Federal agencies in the efforts to help States comply with the Supreme Court's landmark Olmstead decision. This decision called upon States to develop "comprehensive, effectively working plans" to provide services to individuals with disabilities in the most integrated settings possible.

Evidence: Element 1 Evidence at: www.doleta.gov/disability/partevidence/ NFI Executive Order 13217 WIASRD DATA GAO Report National Academy of Social Insurance Balancing Security and Opportunity VR Act Findings WIG Solicitations FY 00 FY 02 FY 03 FY 04

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The focus of the WIG program is on improving the capacity of the workforce development system to increase the workforce participation and career advancement of people with disabilities. This program helps the One-Stop Career Centers and their services become more accessible as well as improving their capacity to integrate the full range of Federal and State services available to people with disabilities. GAO's recent report on Federal Disability Assistance identifies almost 200 Federal programs involved in disability programs and services. There is no program - apart from WIG program - designed to improve the linkages across these programs. The approach now used by the WIG program is the Disability Program Navigator (DPN), which builds upon lessons learned in prior WIG projects. DPNs are now employed in over 250 workforce investment areas. They assure that clients and recipients of many support programs that are not partners in the One-Stops become aware of, and have access to, the employment assistance provided through the One-Stop system. The Navigator facilitates linkage by One-Stop staff and customers to many of these programs at the local level and assists their ability to access services that further their employment objectives, including linking them more effectively with employers. Navigators do not duplicate or supplant services offered by agencies providing local workforce or related services, such as vocational rehabilitation counselors, SSA benefits, planning and outreach specialists, and One-Stop case managers. Instead, the Navigator supports and supplements existing programs by providing greater knowledge, ability, and capacity within the system and increasing access to programs and services available through the One-Stop Career Center system that serve individuals with disabilities. Vocational rehabilitation (VR) agencies do not perform the functions of the WIG or Navigator since they are legislatively prohibited from expending resources on anyone but VR-eligible clients. In addition, in the majority of states VR agencies ration their limited resources by operating under "order of selection" requirements, which target services on the most severely disabiled, further limiting the ability of most job seekers with disabilities to access their services. Although many state and local workforce investment areas have negotiated MOUs with VR, VR generally does not share in the infrastructure costs required to better address the needs of people with disabilities, such as procurement of assistive technology in One-Stop resource or training facilities, electronic doors or providing interpreter services. The VR counselors are often available on a limited basis at the One-Stop facilities and job seekers with disabilities may wait months before learning whether they may be eligible for their services. Grants to the One-Stop system funded by the Office of Disability Employment Policy (ODEP) are limited to a very few workforce investment areas and entail case management, counseling, and job restructuring with supported employment models. Likewise, Disabled Veteran's Outreach Program (DVOP) specialists do case management and counseling, and will not work with individuals with disabilities who are not veterans.

Evidence: DPN solicitation to states and Grant Agreement explicitly require that Disability Program Navigators are not 1) case managers, 2) vocational rehabilitation counselors, or 3) Benefits, Planning and Outreach Specialists. WIG Solicitations FY 00 FY 02 FY 03 FY 04 GAO-05-626, Federal Disability Assistance: Wide Array of Programs Needs to be Examined in Light of 21st Century Challenges, June 2005. This Report identifies 20 Federal agencies and almost 200 programs involved in disability programs and services.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The design of the WIG program as a competitive grant with a very broad purpose has allowed ETA to fund a variety of approaches for improving the ability of the workforce investment system and the One-Stop Career Centers to serve job seekers with disabilities. Regular assessments of the different approaches used by the grantees has helped ETA identify successful approaches for helping job seekers with disabilities, including the development of tools and resources that can be used across the workforce system. Using successful models developed through the WIG program, ETA has refined its approach to one that is focused on staff capacity using a Disability Program Navigator. Implementation of Navigators effectively and efficiently assures expanded access to the One-Stop Career Center system for persons with disabilities with the ultimate objective of increasing their ability to achieve successful employment outcomes and economic well-being. Grants under the WIG were first awarded in October 2000 during the early implementation phase of the WIA. Experience with these initial grants showed that a number of local workforce investment areas were using "disability resource specialists" or "navigators." With the development of the One-Stop Career Center system and partnership with Vocational Rehabilitation agencies, it became apparent that there was a need for a dedicated staff person that would bring greater awareness on disability related and workforce issues. It also became clear that many agencies that worked with people with disabilities, such as Mental Health and Developmental Disabilities agencies and providers, were not aware of, or utilizing, the One-Stop system to meet the employment goals of their clients. The systemic approach of the DPN initiative will lead to long lasting and sustainable enhancement of services for individuals with disabilities. In addition, ETA has begun to use cooperative agreements with the States to target the activities of these DPN grants in a consistent and strategic manner in line with the goals and objectives of the WIG appropriation. ETA provides criteria for the implementation of the grant, such as requiring a comprehensive accessibility plan, including physical and programmatic access of their One-Stop Career Centers, coordination strategies related to mandated and non-mandated partners, and engagement with the employer community. States that apply for the DPN cooperative agreement grant must establish a comprehensive access plan as well as other requirements that emphasize non-duplication of other functions such as VR counselors or Benefits Planning and Outreach specialists. The cooperative agreement grant process allows for increased federal involvement with the States' implementation of the DPN initiative and includes accountability for workforce outcomes of their customers with disabilities. Feedback from WIBs, One-Stop and VR staff, as well as external evaluators, strongly indicates that the Navigator is serving an unmet need in the workforce system and is central to increased integration of One-Stop programs and services as these relate to job seekers with disabilities. This further indicates a high level of effectiveness and efficiency with the limited WIG appropriation funds.

Evidence: Element 1 Evidence at: www.doleta.gov/disability/partevidence/ WIG/DPN Comparative Funding Chart WIG Process Evaluations Round I Round II Round III Selected WIG Quarterly Reports

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The flexibility of the WIG program has allowed ETA to implement a program design that strategically directs program resources to the intended beneficiary of these grants -- individuals with disabilities. The DPN initiative promotes a seamless delivery of services to help these individuals through providing information and resources that increase their likelihood of successful employment. Navigators conduct outreach to other programs and agencies that may have activities such as a local mental support groups, or non-profit community-based skill training program, to assure that One-Stop services reach the clients of these organizations. Likewise, Navigators facilitate linkages with the employer community. SSA's collaboration with ETA also supports Navigator involvement with beneficiaries of Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) programs and the systems that impact their entry or return to work efforts, thus engaging the One-Stop system with individuals with the most significant disabilities. To date, ETA has awarded DPN cooperative agreements to 17 States with more than 250 Navigators in place. Effective in 2005, all WIG funds will be directed to expansion of the DPN initiative while sustaining support to the first 17 States. ETA expects to be funding more than 500 Navigators across 33 States by December 2005. SSA provided $9 million in the first two years of the DPN initiative.

Evidence: Element 1 Evidence at: www.doleta.gov/disability/partevidence/ Add DPN modification instruction to 14 states WIG Solicitations 00 SGA 02 SGA DPN Solicitations 03 Cooperative Agreement Solicitation 04 Cooperative Agreement Solicitation WIG Annual Process Evaluation Instrument DPN Quarterly Process Evaluation Instrument DPN Fact Sheet

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Long-term measures have been established under the WIG program. These have been revised to incorporate the Administration's Common Measures for job training and employment programs and apply them to the DPN program effective PY 2005 forward. During PY 2003 and 2004, the long-term measures for the initial WIG program have been: (1) the percent of participants served by the workforce investment areas receiving grant funds who are persons with disabilities; and (2) the entered employment rate for participants with disabilities that exited the adult, dislocated worker and adult youth programs authorized under the Workforce Investment Act (WIA). DPNs will continue to be measured by the percent of WIA exiters with disabilities served and an entered employment rate measure. ETA will also be measuring employment retention and wage gain for the DPN program effective for PY 2005. Measurement of the percent of WIA exiters with disabilities is important for program oversight because of the significant declines in the numbers of persons with disabilities served by the workforce development system since 1998. ETA also monitors progress on systems change activities under the grants in order to evaluate the indirect activities and goals of these grant programs. These systems change activities include measuring (1) training of WIBs, One-Stop staff, employers and others on disability related issues, (2) development and completion of accessibility plans, and (3) coordination and collaboration with both mandated and non-mandated partners that impact the ability of individuals with disabilities to achieve successful employment.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ Common Measures WIG Performance goal DOL-06-4.1B.4 WIG Process Evaluation Instrument DPN Quarterly Process Evaluation instrument

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: ETA has established specific quantified measures and ambitious targets since the inception of the WIG program in 2000 and is incorporating the WIA common measures for the DPN initiative. The applicable long-range measures for the WIG or DPN programs are employment outcome measures (entered employment rate, employment retention and wage gain) and efficiencies (increased numbers of participants and exiters with disabilities). PY 2004 will be the first full year in which the DPN initiative will have been in effect with approximately 250 Navigators working in local workforce investment areas. Performance data for PY 2004 will be available in October 2005 through the Workforce Investment Act Standardized Data Record, or WIASRD system.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ WIG Process Evaluation Instrument DPN Process Evaluation Instrument WIG Performance goal DOL-06-4.1B.4

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The WIG/DPN program uses the same measures for annual and long-term performance. The annual goals are consistent with the goals for the long-term measures and incorporate the WIA common measures for the DPN State cooperative agreement grants. The initial WIGs have had service level goals for persons with disabilities and entered employment goals. As of PY 2005, DPNs will be measured against additional goals established under the common measures. These include: (1) the entered employment rate in those workforce areas receiving the WIG or DPN grants; (2) employment retention; (3) wage gain; and (4) efficiency measure.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ WIG Performance goal DOL-06-4.1B.4

Does the program have baselines and ambitious targets for its annual measures?

Explanation: As noted in Question 2.2, ETA established specific quantified measures and ambitious targets since the inception of the WIG program in 2000 and is incorporating the job training common measures for the DPN initiative. The following targets are established for the DPN initiative in the PY 2005 - 2008 for those States receiving a DPN cooperative agreement grant. Measure PY 2005 PY 2006 PY 2007 PY 2008 Entered Employment 66% 67% 68% 69% Employment Retention 81% 82% 83% 84% Wage Gain $1,872 $1,922 $1,972 $2,022 Efficiency $1,332 $1,292 $1,254 $1,216 The initial WIG measures focused on the percent of exiters with disabilities and entered employment rate. The DPN program adds retention, wage gain, and efficiency measures. Baselines have been established for DPN States based upon PY 2003 data when the first 14 States were just beginning to get the DPN initiatives underway. The baselines also take into account outcomes for all exiters of the WIA adult, dislocated worker and older youth programs. The measures establish applicable increases or decreases in goal levels that are ambitious. For example, a wage gain of $1,872 has been established for FY 2005 based upon the fact that while the national average for all exiters in PY 2003 was $1,267; the wage gain for exiters with disabilities was $1,772 in the 14 DPN States (data reflects a period prior to receipt of grant). The efficiency goal of $1,332 was established for the PY 2005 goal. ETA utilizes the WIASRD data in all administration budget requests submitted to OMB.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ [***add revised PY 2003 data for 14 DPN states] WIG Solicitations Process Evaluations for WIG I, II and III WIG 06 Budget Request

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: In the solicitation for the WIG program, the applicant must commit to achieving the performance levels represented in the Government Performance and Results Act (GPRA) goals and assure that effort will be made to achieve these levels of performance. These assurances are to be incorporated as a component of the response to the solicitation and are evaluated during the technical review of the application as a factor of the award criteria. If the application is successful, these assurances become a measurable aspect of the statement of work and are evaluated during monitoring and oversight activities. The applicant's committal to these performance levels extends to all sub-recipients of the WIG Funds as do all governmental requirements.

Evidence: Element 2 Evidence at: [***add modification instructions to 14 states***] WIG Solicitations 00 02 03 04

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: An external contractor, the University of Iowa, has conducted a qualitative evaluation of the activities of the WIG since its inception. An extensive qualitative and quantitative evaluation is underway related to the DPN initiative. The University of Iowa annual evaluation of the WIG has provided information regarding the impact of the WIG on: (1) systems change; (2) coordination and collaboration with mandated and non-mandated partners; (3) training to the workforce system, to Workforce Investment Boards, and employer community; and (4) implementation of accessibility reviews and corrective action, including implementation of assistive technology. The evaluation protocol for the DPN initiative includes qualitative components as well as data analysis of grant impact. The University of Iowa has completed the first phase of a telephone interview protocol with over 1,000 individuals associated with the DPN projects in the first 14 States and analyzed the results. They will conduct a second call to individuals in December 2005 (approximately a year following the first calls). The University has also conducted site visits to four in-depth evaluation States (Massachusetts, Florida, Colorado, and Wisconsin) where they are conducting interviews with State and local officials involved in the DPN initiative. Protocols for data analysis have been identified and will be conducted through a subcontract with the Jacob France Institute at the University of Baltimore, which will obtain and analyze WIA and Wagner-Peyser data in 7 DPN States, including comparative analysis of both DPN workforce investment areas and non-DPN workforce investment areas with comparable workforce characteristics. Data analysis will incorporate comparative analysis of individualized WIA records against SSA disability files if access to SSA data can be obtained.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ [***add Evaluation Plan from U. of Iowa] [***add telephone survey results] [***add telephone survey power point] WIG Process Evaluation Instrument DPN Process Evaluation Instrument

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: There is no clear relationship between the resources requested and the long-term performance goals. Different levels of resources will affect the number of grantees or the funds provided to each grantee, but the linkage between the grants and the proportion of individuals with disabilities served by the One-Stops and their employment outcomes is not known. Nonetheless, ETA's budget request does include performance information related to the WIGs and explicitly identifies annual and long-term performance measures.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ FY 06 Budget request

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: ETA has taken steps at each stage of the implementation of the WIG program to refine the program and better align it with its purpose and objectives. The flexibility of the WIG has allowed ETA to identify promising practices that are shared with the workforce system at large. Further, it has provided the opportunity to focus the program on meeting workforce system deficiencies by establishing a staff role, the Disability Program Navigator, to be fully directed to improving One-Stop Career Center services for individuals with disabilities by providing information, conducting training, coordinating with mandated and non-mandated One-Stop partners and fostering outreach to disability provider agencies and increasing employment opportunities through outreach to the employer community. Lessons learned during the initial WIG awards allowed ETA to ensure that the WIA system at the State and local level is the primary administrative entity for the WIG and DPN grants. In a few cases, the State VR agency also has significant involvement in the administration of their WIG/DPN grant. ETA has required in those instances that the WIA agency be the responsible administering entity and must host the project lead person. In order to manage the program more effectively, as well as have strong State buy-in, ETA has required designation of a State lead as the primary point of contact for monthly conference calls and consultation on multiple issues related to the State's Navigators. State project leads have been very instrumental in a remarkably successful deployment of the DPN effort. In addition, the implementation of process evaluations and a very strong training, technical assistance, and evaluation contractor, the University of Iowa's Law, Health Policy and Disability Center (LHPDC) since 2001 has allowed ETA to follow grantee progress and identify strategic deficiencies that it was able to correct or revise as the program has moved forward.

Evidence: Element 2 Evidence at: www.doleta.gov/disability/partevidence/ WIG Solicitations 00 02 03 04 DPN Solicitations 03 04

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ETA requires that both narrative and financial reports be submitted by the grantee on a quarterly basis. All reports are submitted utilizing ETA's online reporting system, the Enterprise Information Management System, or EIMS. The WIG technical assistance provider (the University of Iowa) also conducts an annual process evaluation with the WIG projects and quarterly process evaluations with the DPN States. The process evaluations include significant detail on the extent of collaboration and partnership that is occurring with both WIA mandated and non-mandated partners. The information and data collected through both the WIG and DPN process evaluations are used to inform overall program management decision making. ETA also regularly requests State and local data on service levels and outcomes for job seekers with disabilities accessing the workforce system, through the Workforce Investment Act Standardized Data Record, or WIASRD system. To assist ETA with program management and to assist with management efforts to improve performance, ETA regularly requests data on: the number of participants with disabilities, the number of program exiters, the entered employment rate, wage gain, and retention for the WIA adult, dislocated worker, and older youth programs. All of the data collected is used to gauge the performance of the individual grantees as well as the overall program strategies. This performance information was instrumental in the evolution of the initial WIG program into the current DPN initiative. ETA realized after the first year of WIG funding, as a result of the data collected, that the initial WIG strategy was not the optimal approach. Changes were made to each subsequent solicitation as a result. Funding strategies, moving from grants to cooperative agreements, were also changed as a result of this information. ETA has also undertaken, as a strategy to enhance performance of the program, regularly scheduled teleconferences between the grantee, the Federal Project Officer, and program office staff. Collecting performance information on each of the projects as well as identifying technical assistance needs and effective practices are the main topics for these calls. Information collected in these calls is summarized and shared with project staff, program office managers and staff, regional office management and Federal Project Officers and is used to effectively manage the programs, improve performance and increase efficiency.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ WIASRD Data WIG Process Evaluation Instrument Select EIMS Narratives Select EIMS SF 269 submission Select WIG Teleconference notes

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ETA has not held partners directly accountable for quantifiable performance standards for the WIG program due the nature of the grant and the fact that performance outcome data were only available after the grant ended. Also, the grants under the WIG program provide no incentives or sanctions based on performance outcomes. Nonetheless, ETA closely monitors costs and schedules for the new DPN program. ETA is taking past performance into account as it awards DPN cooperative agreement grants to States that have had WIGs in the past. It is ETA's belief that, despite past performance, the task of holding grantees more accountable will be much easier to accomplish under the DPN approach which establishes specific and finite timelines and involves direct relationship between expenditures, the salaries of Navigators, and the grants rate of expenditure (e.g., if Navigators are not hired, it is very quickly evident in expenditure reports). This process is much more predictable and performance is more readily subject to review and oversight. ETA has entered into an interagency agreement with the SSA for the DPN program and has the full administrative responsibility, including performance accountability for the initiative. ETA provides all DPN performance information to its SSA counterparts and holds formal discussions and reviews with SSA in making management decisions that affect performance of the DPN program. ETA has instituted a rigorous grants management system that utilizes resources from both the regional and National office to oversee the grants, track performance and expenditures and to monitor implementation and grant activity timelines. The ETA Regional Office's Federal Project Officers (FPO's) are responsible for tracking grant execution and expenditure data identified through quarterly reports, desk reviews and on-site monitoring. The National office is charged with the overall responsibility of managing the program, establishing and implementing policy, program oversight, and technical assistance including administrative training and instruction, and performance accountability. All accountability related information is collected through electronic systems developed by ETA for this specific purpose. Overall program performance related data is collected through the WIASRD database. Individual grant performance information is collected through ETA's EIMS. Actual grants management and oversight by both the national and regional offices is effectuated through the utilization of the new GEMS (Grants Management System) which was created for the sole purpose of facilitating FPO and ETA oversight. The GEMS system affords the opportunity to evaluate the grantees and Federal management performance through the collection and aggregation of available data. The Performance Management Plan for Senior Executives which applies to the Senior Executive responsible for the WIG/DPN initiative states in element A: "Uses performance plans and other management systems to link individual performance to outcomes and to overall organizational performance objectives." ETA managers directly responsible for the WIG/DPN initiative have included in their individual performance plans as a performance agreement result: "3. Implement accountability and reporting measures for adult programs (A) Conducts analysis of State WIA performance relative to people with disabilities as these relate to WIGS and other grantee performance. (B) Identifies and recommends applicable processes for facilitating improved performance based upon model grant program designs and practices."

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ 269 Select Monitoring reports Select Teleconference Notes

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The nature of the WIG appropriation requires that all funds be obligated in the first twelve months and that all grant/program activities be concluded by the end of the 36th month. All program funds are obligated within the assigned time frames. Expenditures are purposefully tracked to assure that they are being expended according to the plan as outlined in the scope of work. When an unacceptable rate of expenditure, i.e., when obligated balances are not commensurate with the projected timelines, is identified by either the regional FPO or the program office, interventions are begun to identify and rectify any problems that affect the program's performance.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ WIG Solicitations 00 02 03 04 DPN Solicitations 03 04 Select WIG Quarterly reports Select DPN Quarterly reports Select WIG Teleconference notes

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ETA has multiple procedures in place to measure and achieve increased efficiencies and cost effectiveness. One long established procedure is to require that all procurement actions be subject to the respective State procurement rules that include competitive sourcing and cost comparisons. These requirements extend to all partner programs and include the acquisition of information and assistive technologies. DPN grantees are being evaluated in an ongoing basis through several mechanisms including a quarterly process evaluation that evaluates the grantees activities each quarter. An annual analysis of the data collected through the process evaluation is produced at the end of each program year that informs the assessment of the grants impact and overall efficiencies and effectiveness attained. An efficiency measure has been established for the DPN program and will be in effect for PY 2005 forward. ETA has elected to determine efficiency based upon the number of individuals with disabilities exiting the WIA adult, dislocated worker and older youth programs in States or workforce investment areas receiving grant awards divided by the total grant award. ETA expects that efficiencies can be accounted for in this manner based upon the fact that there should be more individuals with disabilities accessing intensive and training services where Navigators are in place and conducting outreach to the disability community as well as improving One-Stop awareness and responsiveness. The program office has developed an online repository, onestoptoolkit.org, of information on the WIG and DPN programs. The website is maintained under contract and is updated on a regular schedule. One purpose of the website is to make information on grantee activities and products developed available to all grantees thereby avoiding duplication in most instances and to promote replication in others, all in an effort to maximize the impact and increase the efficiency of the resources used to support the WIG and DPN initiatives.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ WIG Solicitations 00 02 03 04 SPR data WIG Process Evaluations Onestoptoolkit.org URL University of Iowa URL

Does the program collaborate and coordinate effectively with related programs?

Explanation: ETA has had a long history of collaboration with partner agencies with regard to the WIG program and funding. The development of the WIG initiative was undertaken through coordination between multiple agencies with programs serving people with disabilities under the umbrella of the Task Force on the Employment of Adults with Disabilities, the White House Domestic Policy Council, and OMB. DOL/ETA had the lead role in the development of the proposal in collaboration with the Task Force. The subsequent grant solicitations have always included a major focus on coordination with WIA mandated and non-mandated partner organizations that affect the ability of people with disabilities to succeed in obtaining and retaining employment. ETA understands that coordination of the multiple programs, services and the mitigation of associated barriers related to employment are of significant concern to people with disabilities. ETA considers the WIG program, and the DPN focus, to be a major catalyst in facilitating a more comprehensive, seamless approach to addressing this concern. Similarly, ETA views the One-Stop system as the primary vehicle to addressing this concern and thus has directed the WIG and DPN approaches with this in mind. Further confirmation of this effective coordination is ETA's joint collaboration on the DPN initiative with SSA providing joint funding during the first two years of the project and a commitment to continue its involvement for two additional years. SSA is and has been involved in all aspects of DPN program development, implementation, training, and program evaluation. As a full partner in the effort, SSA has become increasingly aware of the WIA funded One-Stops and their crucial role in securing effective and self-sustaining employment for individuals with disabilities. ETA is also conducting DPN briefings with Federal partner agencies to reinforce coordination and to keep its grantees informed of issues and activities underway with Federal partners through the One-Stop Toolkit Resources of the Week. Briefings have been conducted or are being planned with the following agencies, OSERS, RSA, CMS, SMSHA, In addition to coordination related to the grants, ETA has led an Interagency Forum at the Assistant Secretary level that includes ETA, ODEP, SSA, OSERS, VETS and DPC. At the staff level there is also a long history of collaboration with Federal partners on numerous topics and issues.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ SSA Interagency Agreement WIG Solicitations 00 02 03 04 DPN Solicitations 03 04 Section 188 Regulations Ticket TEIN

Does the program use strong financial management practices?

Explanation: In the FY 2003 Performance and Accountability Report, DOL's Office of the Inspector General (OIG) identified grant accountability among the most serious management challenges facing DOL. In DOL's FY 2004 Performance and Accountability Report (November 2004), the OIG recognized that ETA has undertaken a grants management initiative, the results of which the OIG plans to review. While the OIG did not put ETA grants management on its most recent list of top management challenges, it noted that accountability over DOL awarded grants will continue merit diligent attention. Because of the continuing concerns of the OIG about ETA's grants management in general, this question has received a "NO." The OIG recognized that ETA took a significant step in April 2003 when it issued an order implementing its Grant/Contract Administration Plan. Employment and Training Order No. 1-03 is designed to improve grantee accountability, compliance, and performance. In carrying out the order, ETA has issued a Regional Accounting Policies and Procedures Manual and is making improvements to post-award grant monitoring, among other things.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ Select Notice of Awards for WIG OFAM TAG SF 269 SF 424A OIG's annual reports on "Top Management Challenges Facing the U.S. Department of Labor" are available at www.oig.dol.gov/topchallenges.htm. These reports are included in DOL's Annual Reports on Performance and Accountability, which are accessible at www.dol.gov/dol/aboutdol/main.htm. WIA quarterly expenditure reports are available quarterly at: www.doleta.gov/budget/. TEGL 16-99, Change 1 titled, "Workforce Investment Act Financial Reporting," and TEGL 16-99, titled "Workforce Investment Act Financial Reporting," provide the instructions and clarifications for the financial reporting instructions. These TEGLs are accessible at wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1433 and wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1229. Employment and Training Order No. 1-03; ETA Regional Accounting Policies & Procedures Manual, provides Federal managers guidance on grant management.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Through the reporting requirements and ensuing review of submissions by the grantees, ETA found that initial funding strategies in the first round of the WIGs were inadequate to achieve the purpose and objectives of the program. As a result, governance requirements in the subsequent solicitations were changed to require greater involvement of the actual WIA funded administrative entity. Eligible applicants for the first round WIGs included private non-profits and other agencies such as Vocational Rehabilitation. Experiences from this round led to requirements in subsequent solicitations that only entities with WIA administrative responsibilities were eligible for WIG or DPN funding.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ WIG Solicitations 00 02 03 04 DPN Solicitations Draft Regional FPO Re-designation Memo

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: ETA has used a fully competitive process for awarding grants under the WIG program that adheres to all Federal procurement rules and allows for an impartial technical review using established programmatic and administrative criteria published within the solicitation. In order to assure compliance with all rules and regulations and to assure fairness in the award process, ETA uses a panel for proposal review and acceptance. The process begins with the development of the solicitation for grant applications (SGA). The SGA contains background, purpose, eligibility, goals and objectives, ETA expectations, and selection criteria. The selection criteria are based upon the technical components of the proposal, scope of work, timelines, budget, and legal assurances with each individual criterion being assigned a specific point value. Solicitations are then published in the Federal Register and online with specific submission requirements and any required forms, including the due date of the proposal. The merit of a particular proposal is determined by a technical review panel convened for the sole purpose of reviewing and determining the acceptability of the proposals. Application scoring is based upon the criteria published with the solicitation and is solely based upon the technical merit of the response. Acceptance of a proposal is incumbent upon the applicant scoring above a predetermined score, most generally a base score of 80 is considered acceptable. Applicants are then ranked according to their individual score, with award decisions based upon the rank order, the number of acceptable proposals and the amount of funding available. The award for the initial WIGs provided funding for two years. There were no grant renewals. ETA is currently using an alternative competitive procurement device - the cooperative agreement - for the DPN initiative, which affords opportunity for greater interaction and a more comprehensive pre-award negotiation with the State-level applicants. If a proposal is found unacceptable or non-responsive, negotiations are undertaken with the applicant until the proposal meets ETA's minimum requirements as outlined in the solicitation. If a State does not respond appropriately, an award will not be made. To date, all applicants have succeeded in meeting or exceeding ETA's pre-award requirements. DPN awards are funded for a minimum of two years, subject to the available of funds, the performance of the State, and the State's continued commitment of meet the requirements of the grant. The goal of the DPN initiative is to provide 100 percent coverage in all areas served by WIA programs and services. This is being done in three stages, with each stage representing a third of the States. Determination of the eligible entities has been made based upon the performance under previous WIG awards. States that have not performed up to ETA's expectations under the WIG project have been delegated to the latter tiers and will be required when solicited to submit comprehensive State-wide accessibility plans as part of the award process. The competitive aspect of the agreement requires States to meet criteria defined by ETA. Those States that do not meet these criteria will not be awarded funds under the DPN initiative. The DPN initiative has only been in place for two years and no grant renewals have been made. ETA will begin to consider renewal of grants based upon demonstrated performance of States in meeting established GPRA goals for the WIG program.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ WIG Solicitations 00 02 03 04

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ETA has instituted a post-award grants management system and process that is composed of multiple layers of oversight conducted at both the regional and National offices. The program office supplements these established procedures and requirements through regularly scheduled teleconferences and additional grantee contact in coordination with ETA's contracted technical assistance provider, the University of Iowa Center for Law, Health Policy and Disability. WIG grantees are required to submit quarterly financial and programmatic narrative reports that are reviewed first by the regional Federal Project Officer (FPO). Once the regional FPO accepts them, the reports are reviewed by the National office. If problems or issues are identified, the National office in concert with the FPO will determine the best and most appropriate course of action. As a result of these oversight and monitoring activities, corrective actions have been required of several WIG grantees. The corrective actions consisted of simple procedural changes to more significant alterations to the scope of work. On-site monitoring is coordinated through the regional office and may include site visits from either regional office staff or National office or both. Monitoring visits are scheduled and conducted by the regional FPOs with assistance from the program office. Most regions require at least one on-site FPO visit per calendar year. Program office initiated site visits are conducted on an as needed basis with the frequency often determined by the available resources. On-site visits of grantees by the technical assistance provider are conducted in coordination with the National office on an as-needed basis. ETA also has two primary on-line reporting tools, Enterprise Management Systems (EIMS) and Grants Electronic Management System (GEMS), which provide the opportunity to view grantee performance in a real time mode. ETA uses data mining technology to aggregate relevant data from existing government databases, including daily funds draw-downs and other important data necessary to effectively manage the programs. Since the inception of the WIG program, grantees are required to participate in a process evaluation that further informs ETA of the program's performance. The process evaluation is used to identify effective strategies that have guided the overall effort.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ Select Teleconference notes FPO list Select Monitoring reports WIG Process Evaluations DPN Process Evaluations EIMS screen shots SF 269 Narrative GEMS Screen shots

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Under the Workforce Investment Act, ETA requires an annual submission of performance related information from all State-level WIA grantees. ETA compiles this information under the Workforce Investment Act Standardized Record Data-book (WIASRD). This information is published on the internet and is provided through other mediums including print. Users can access data down to the individual workforce investment area impacted by the grant and eventually will be able to drill down to the individual One-Stop. Additional information relating to the WIG/DPN effort is compiled through the both the WIG and DPN process evaluations. This information is reported out through publication on the internet and through ETA internal performance related reporting. All performance data is collected annually and is made readily available to the public through multiple mediums. ETA actively reports this data to Congress through annual reports and to the public through quarterly performance reports.

Evidence: Element 3 Evidence at: www.doleta.gov/disability/partevidence/ WIASRD Data Website U of Iowa website URL

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Data indicate that progress has been made since the inception of the WIG program in achieving an integrated and universally accessible workforce system that meets the needs of jobseekers with disabilities. Data show that increasing numbers of individuals with disabilities in the areas where WIG projects are active are achieving successful employment outcomes: they are retaining their jobs, and their wages have increased significantly when compared to individual program participants without disabilities. In PY 2003, WIG awards made in the first funding round met the entered employment goal of 70 percent but did not meet the service level goal of 6 percent of WIA program exiters being persons with disabilities. For Round I WIG projects, the entered employment rate was 72.1 percent for persons with disabilities (and higher than the 65.8 percent nationwide average) and projects served 4.4 percent of WIA exiters with disabilities (nationwide average 5.6 percent). For this same round of projects, the retention rate was 84.1 percent (nationwide average 82.7 percent) and wage gains of $1,497 (National Average $1,595). In PY 2003, Round II WIGs almost met the service level goal of 6 percent but did not meet the entered employment goal of 70 percent. Round II WIGs served 5.9 percent of those exiting the WIA adult, dislocated worker, and older youth programs, and achieved an entered employment rate of 60.8 percent. Their performance for employment retention was 80.6 percent and wage gain was $1,715. The DPN initiative has only been underway for two years and the results from PY 2003 are only a partial reflection of their impact. However, this early data is quite positive for the first 14 States funded. They meet the PY 2003 performance goals for service levels with 6 percent of their WIA program exiters being individuals with disabilities. The employment and retention data are not yet available for DPN grantees. The increased level of services for individuals with disabilities is an early indication of outreach and attention to job seekers with disabilities.

Evidence: Element 4 Evidence at: www.doleta.gov/disability/partevidence/ SPR Data (WIG I, II and III) PROCESS EVALUATION WIGs I, II, AND III SPR Data (Navigator States for PY 2004 and 2005)

Does the program (including program partners) achieve its annual performance goals?

Explanation: A primary objective of the WIGs is to improve the employment and employability of people with disabilities through enhancing the ability of One-Stop Career Centers to better serve people with disabilities. WIASRD data related to the WIGs demonstrates a mixed result in terms of achieving the performance goals in PY 2000, 2001, and 2002. For Round I WIG in PY 2001: The baselines were established as follows: 5.6 percent of people with disabilities exited WIA adult, dislocated worker, and adult youth programs; 68.4 percent of people with disabilities enter employment.

Evidence: SPR data reports for Round I WIGs covering PY 2001, 2002 and 2003; SPR data reports for Round II WIGs covering PY 2003. Process Evaluation Data on Systems Change for WIGs I, II, and III Trends of Exiters and Service Levels for People with Disabilities - JTPA and WIA Element 4 Evidence at: www.doleta.gov/disability/partevidence/ SPR/WISARD Data

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The efficiency of the WIG program can be, in part, demonstrated through a comparison of the total grant funds and the total number of exiters with disabilities in workforce investment areas that received the grant, or a cost per exiter. While this is an incomplete look at the efficiency of a program that is designed to change systems and to yield benefits over a number of years, it has the advantage of being comparable to the efficiency measure used by the job training Common Measures. In addition, efficiency measured in this way can be improved by serving more persons with disabilities with the mainstream WIA services. Utilizing this efficiency measure, Round I WIGs cost $1,873 per WIA exiter with disabilities of the two years they were in effect. By Round II of the WIG projects, efficiency had improved, with a cost of $860 per exiter with over the course of two years they were in operation. The 14 DPN States achieved efficiencies of $509 over the course of partial year in which these were in place. Data are not yet available to determine efficiencies related to WIG Rounds III or IV. The efficiency measure may overestimate the costs per exiter because the data rely on self-reporting of disabilities. However, the substantial efficiencies demonstrated by the DPNs suggest that this will prove to be the most effective approach in terms of increasing participation of job seekers with disabilities in the workforce system and as a cost-effective approach to meeting the needs of this customer group.

Evidence: Element 4 Evidence at: www.doleta.gov/disability/partevidence/ SPR Data (WIG I for PY 2001 and 2002; WIG II for PY 2002 and 2003) SGAs for WIG I and WIG II

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other systems change projects that are clearly comparable to what ETA has been doing. This is the case for both the initial WIG grants and the DPN initiative. The initial WIGs were systemic in nature and funded infrastructure development in a variety of ways. As systems change grants, the WIG grants are similar to HHS Centers for Medicaid and Medicare Services' Medicaid Infrastructure Grants, but the focus of the systems change differs so that their outcomes cannot be compared. There is also no other Federal agencies that fund staff positions in the workforce development system that are similar to Navigators. ETA, in coordination with SSA, has been clear throughout implementation of the DPN initiative that these individuals are: (1) not case managers; (2) not vocational rehabilitation counselors; and (3) not benefits, planning and outreach specialists. Navigators assist both One-Stop staff and persons with disabilities who are customers of the disability-focused programs to "navigate" through the variety of services available in a way that furthers their employment objectives.

Evidence: Element 4 Evidence at: www.doleta.gov/disability/partevidence/ RS 911 Data RSA Longitudinal study SPR Data (Round I and II WIGs for PY 2002)

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Although there have been several external reviews of the WIA system and services to job seekers with disabilities, there have not been any of sufficient scope and quality to evaluate the program's effectiveness, especially its impact on participants' employment outcomes. An independent evaluation of the DPN initiative is being conducted by the University of Iowa working with the Jacob France Institute at the University of Baltimore, but it has not yet been completed. An annual process evaluation has been conducted with each round of the WIG program. The WIG process evaluations also substantially supported that the WIGs were accomplishing their intent of building the One-Stop Career Center infrastructure to better serve job seekers with disabilities. Other external reviews of the WIA system and services to people with disabilities have been conducted by the GAO, Urban Institute, and National Council on Disability (NCD). These reviews and reports were not specific to the WIG or DPN grants. However, each has mentioned the positive impact that these grants have made, while noting the continuing need for assistance to the system. The Urban Institute study was conducted for the Ticket to Work and Work Incentive Improvement Act Advisory Panel: "Of most relevance of these initiatives are the WIGs and the DPN demonstration program??although many of the evaluations of these activities are ongoing, the studies that are available plus discussions with government officials and other experts indicate that the programs have made a discernible and positive impact on improving the capacity of the One-Stop Career Centers for serving customers with disabilities."

Evidence: Element 4 Evidence at: www.doleta.gov/disability/partevidence/ DPN Evaluation Power Point Telephone Survey Power Point Colorado Independent Evaluation WIG Process Evaluation Analysis DPN Process Evaluation Analysis DPN Evaluation Instrument Supporting Documentation for the PART