Detailed Information on the
Office of Disability Employment Policy Assessment

Maintaining a consistent set of performance indicators to measure progress toward achieving long-term goals and developing a valid performance management tracking system for collecting data.

Collecting baseline data and out-year data to measure program efficiency and effectiveness.

Conducting a rigorous evaluation to assess the impact and effectiveness of the program's policy and coordination functions.

Measure: Number of policy related documents

Explanation:ODEP develops and influences the implementation of policy to reduce barriers to employment for people with disabilities. This output measure tracks policies developed for implementation to eliminate barriers to employment for people with disabilities. Policy documents include but are not limited to legislation, regulations, and Executive Orders. ODEP is currently developing and testing intermediate outcome measures, which will measure the long-term impact of ODEP's efforts, the development of policy documents, toward eliminating barriers to employment. Baseline targets will be set for intermediate outcome measures in FY 2010.

Measure: Number of formal agreements.

Explanation:ODEP develops and influences the implementation of policy to reduce barriers to employment for people with disabilities. ODEP provides national leadership on disability employment policy by building collaborative partnerships. This measure tracks the results of ODEP's work to foster innovative, effective, and efficient partnerships and collaboration between and among public sector and private sector entities to develop and implement policy that will eliminate barriers to employment for people with disabilities. Formal agreements include memoranda of understanding (MOUs), inter- and intra-agency agreements, contracts, and alliances. ODEP is currently developing and testing intermediate outcome measures, which will measure the long-term impact of ODEP's work, establishing partnerships and collaborative efforts, toward eliminating barriers to employment. Baseline targets will be set for intermediate outcome measures in FY 2010.

Measure: Number of effective practices.

Explanation:ODEP develops and influences the implementation of policy to reduce barriers to employment for people with disabilities. ODEP undertakes research and policy development initiatives to identify and disseminate effective practices that have resulted in increased employment of people with disabilities - strategies, models or theories that have a documented record of success or validated effectiveness by independent sources. This encourages public and private employers to learn from each other and assist others in improving their practices and/or procedures related to the employment of individuals with disabilities. ODEP is currently developing and testing intermediate outcome measures, which will measure the long-term impact of ODEP's work, identifying and developing models and practices, toward eliminating barriers to employment. Baseline targets will be set for intermediate outcome measures in FY 2010.

Measure: Cost Per Policy and Practice Disseminated

Explanation:ODEP's efficiency measure is an annual measure utilized by ODEP to more efficiently use its limited funds to disseminate the greatest number of policies and practices while reducing dissemination costs. It is calculated by dividing the cost of disseminating effective practices by the number of practices disseminated.

Is the program purpose clear?

Explanation: ODEP was created to bring a heightened focus on disability employment within the Department of Labor (DOL) through policy development, technical assistance, and dissemination of best practices. Congress created ODEP in the Consolidated Appropriations Act of 2001, to succeed the expiring President's Task Force on Employment of Adults with Disabilities. The Task Force was created by Executive Order in 1998 and terminated in 2002 after submitting its final report. In its final report, the Task Force report recommended that, "??the President direct the Department of Labor to develop a proposal for consideration in the FY 2001 budget process for an Office of Disability Policy, Evaluation, and Technical Assistance (ODPET) to be headed by an Assistant Secretary of Labor??" (1999, p. 25). As a result, ODEP was established to bring a permanent focus within DOL to address the significant barriers to employment faced by individuals with disabilities and ensure coordination among DOL and other Federal agencies on matters related to or affecting people with disabilities. ODEP also supports the employment-related goals of the President's New Freedom Initiative.

Evidence: ?? "Re-Charting the Course: If Not Now, When," Second Report of the Presidential Task Force on Employment of Adults with Disabilities (November 1999) ?? ODEP Budget History Appropriations Language 2001 - 2006 (Legislative History) ?? ODEP Strategic and Performance Plan FY 2006-2010 ?? CFDA Program 17.720 (http://12.46.245.173/pls/portal30/CATALOG.PROGRAM_TEXT_RPT.show)

Does the program address a specific and existing problem, interest, or need?

Explanation: ODEP's purpose is to develop disability-related employment policy and service provision strategies to improve employment outcomes for people with disabilities. Data from the 2000 Census shows that people between the ages of 16-64 were less likely to be employed if they were disabled. Compared to the general population in this age group, the employment rate for people with disabilities was lower, at 56%. The Government Accountability Office (GAO) reports that the numerous, stove-piped service-based systems addressing the employment of people with disabilities continue to produce outcomes that have not significantly increased the employment rate of people with disabilities in decades, and recommends that Federal disability employment programs be better coordinated (2005). In addition, knowledge of employers' perspectives on recruitment, retention and promotion of people with disabilities has not been sufficiently integrated into policy analysis, formulation and implementation at the Federal level.

Evidence: ?? People with Disabilities: Strengthening the 21st Century Workforce. Final report of the Presidential Task Force on Employment of Adults with Disabilities (July 2002); ?? Disability Status: 2000. March 2003. U.S. Census Bureau. Disability and American Families: 2000. July 2005. U.S. Census Bureau. ?? Employer-focused research conducted by ODEP, through the Job Accommodation Network (2005) and the Employer Assistance and Recruiting Network (2005), documents the pervasive employer concerns that continue to create barriers to the employment of people with disabilities; ?? United States Government Accountability Office (GAO), Report no. GAO- -05-626 Federal Disability Assistance: Wide Array of Programs Needs to be Examined in Light of 21st Century Challenges (June 2005); ?? New Freedom Initiative (February 2001).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ODEP addresses a specific and existing problem. Although there are other Federal, state, and local, private and public programs working to improve employment outcomes for people with disabilities, none of these programs are designed to develop and foster implementation of comprehensive and integrated disability employment policies and strategies. ODEP develops policies and practices for Federal, State, and local service delivery systems to enhance their work administering services and supports. ODEP also develops policies and strategies for public and private employers to improve their recruitment, retention, and promotion efforts.

Evidence: ??Presidential Task Force on Employment of Adults with Disabilities Reports (1999; 2000; and 2002); ??ODEP Strategic and Performance Plan FY 2006 - 2010. ??US Department of Education, Office of Special Education and Rehabilitative Services www.ed.gov ??FY 2007 Budget, New Freedom Initiative Programs

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: ODEP was designed to effect change regarding the numerous, stove-piped service-based systems addressing the employment of people with disabilities, which have traditionally produced outcomes that have not significantly increased the employment rate of people with disabilities in decades. Until recently, ODEP had engaged in excessive grant making activities that shifted attention from the agency's original mission. The FY 2006 and FY 2007 Budget return ODEP to its original mission and provide only the resources needed to support research and disability employment policies and practices that will most effectively reach people with disabilities.

Evidence: ?? Presidential task Force on Employment of Adults with Disabilities Reports (1999; 2000; and 2002); ?? ODEP Strategic and Performance Plan FY 2006 - 2010; ?? ODEP Budget History Appropriations Language 2001 - 2006 (Legislative History)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: ODEP conducts numerous activities to effectively target resources to efforts addressing the program's purpose--to develop a coordinated and comprehensive disability employment policy that affects change within systems and among employers--thereby improving employment outcomes for the ultimate beneficiaries--people with disabilities. ODEP's direct impact is on the Federal, state, and local systems and employers who adopt and implement the results of ODEP's efforts.

Evidence: ?? Sample Solicitation for Grant Announcements (SGA), ?? Sample Requests for Quote (RFQ); ?? Sample Award document; ?? Sample MOU.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: To ensure that performance measures reflect ODEP's core mission and meaningfully reflect the purpose of the program, ODEP, with the support of the Department's Center for Program Planning and Results (CPPR), conducted a comprehensive and thorough review of ODEP's performance history, including annual performance goals and measures. This review resulted in the documentation of annual and long-term performance goals and measures in a "Strategic and Performance Plan FY 2006-2010." This plan includes the following three long-term measures: (1) the number of people with disabilities served by service delivery systems; (2) the number of universal strategies adopted by service delivery systems; and (3) the number of employers accessing services to recruit, retain, and promote people with disabilities.

Evidence: ?? DOL Annual Performance Reports 2002-2005; ?? ODEP Strategic and Performance Plan 2006-2010; ?? ODEP Budget and Performance Integration PMA Scorecards; ?? ODEP Quarterly Performance and Management Reviews.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: ODEP does not have ambitious targets or timeframes that extend beyond FY 2008. Currently, only baseline data for 2005 is available. In FY 2006, ODEP will continue to work with the Department's Center for Program Planning and Results to set ambitious targets and develop data collection methods for the long-term measures.

Evidence: ?? ODEP Strategic and Performance Plan FY-2006-2010; ?? LMI Final Project Report; ?? ODEP Budget and Performance Integration PMA Scorecards; ?? USDOL Annual Performance Reports 2002 - 2005;

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Over its five years of operation, ODEP has consistently changed its annual measures making it difficult to assess any progress towards the long-term goals. Currently, ODEP has ten output and outcome related measures, many of which are process oriented and/or not meaningful (e.g. "policy related documents disseminated"). Additional work is needed to identify a limited number of specific annual performance measures that can measure whether or not ODEP's assistance actually helped increase and improve the employment and earnings of people with disabilities.

Evidence: ?? ODEP Strategic and Performance Plan FY-2006-2010; ?? USDOL Annual Performance Reports 2002 - 2005; ?? ODEP Performance Measures History through 2006.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: In FY 2006, ODEP, with the support of the Department's Center for Program Planning and Results, will develop a valid methodology for establishing baselines and collecting annual performance data. ODEP will also work on setting ambitious yet attainable targets once baselines have been set.

Evidence: ?? ODEP Strategic and Performance Plan FY-2006-2010; ?? USDOL Annual Performance Reports 2002 - 2005;

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ODEP works closely with all of its partners to ensure they are informed of and committed to supporting ODEP in achieving its performance goals and meeting its targets. All of ODEP's key partners, whether grantees or contractors, are required to provide performance and other project data (e.g., customer satisfaction) on a regular basis that inform ODEP's annual and long-term performance goals. ODEP's initiatives and efforts, whether implemented through grants, cooperative agreements, contracts, memoranda of understanding, etc., have always been implemented to support achievement of annual GPRA goals and measures and long-term goals. For example, grantees and contractors regularly provide performance data to ODEP regarding project performance and annual and long-term goals. The quarterly reports for ODEP grantees specifically require demographic reporting on the number of project participants served; the number of project participants employed; their 3- and 6-month retention rates; and effective practices.

Evidence: ?? Sample MOU / MOA ?? Sample SGA; ?? Sample Grant or cooperative agreement award; ?? Sample Contract. ?? www.dol.gov/odep/research

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: ODEP has conducted evaluations of its activities that do not constitute rigorous evidence of program effectiveness. To date, ODEP's evaluations have focused on pilot and technical assistance projects. The evaluations identified "process-related" findings on steps taken in ODEP-funded projects to increase the workforce development system's capacity to serve people with disabilities (e.g., findings on whether the ODEP projects increased training of One Stop center staff in working with people with disabilities). These findings do not constitute rigorous evidence because they do not address whether these process changes caused changes in key program outcomes, such as: (1) an increase in the number of people with disabilities receiving services (compared to the number that would have received services in the absence of ODEP's efforts); and (2) an improvement in employment and earnings of clients with disabilities (over what they would have experienced in the absence of ODEP's efforts).

Evidence: ?? Evaluation of Selected ODEP Grant and Project Efforts: Task 1: Project Work Plan (Westat, October 2003); ?? Evaluation of Disability Employment Policy Demonstration Programs: Task 5: Initial Site Visit Report??Fiscal Year 2003 Demonstration Programs, Interim Progress Report (Westat, February 2005); ?? Evaluation of Disability Employment Policy Demonstration Programs: Task 10: Interim Report on ODEP Demonstration Programs: Accomplishments and issues Identified by the Independent Evaluation (Westat, June 2005)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Although ODEP's budget request identifies how program resources will be utilized to meet annual and long-term performance goals, the program's design does not lend itself to marginal cost analysis. ODEP's budget requests identify priority policy areas and describe the critical strategies that ODEP will employ and the specific efforts conducted to fulfill the program purpose and achieve goals. ODEP's budget narratives include a discussion on different resource levels and how those levels will lead to increases or decreases in scope and breadth of activities and efforts to develop disability employment policy.

Evidence: ?? ODEP Congressional Budget Justification Submissions (FY 2005, FY 2006, FY 2007)

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: ODEP has developed a five-year strategic plan (2006-2010) that documents for ODEP's internal and external stakeholders what is envisioned to achieve the agency's mission and fulfill its purpose. ODEP's plan details ODEP's mission and purpose and identifies the agency's annual performance goal and measures that will be used to determine progress toward the achievement of long-term service delivery systems outcome goals. The strategic plan describes how ODEP will add value to the efforts of the workforce development and other service delivery systems, their partners, and employers, and how the agency will meet the specified performance outcomes. In addition, in FY 2006 ODEP will continue to work with the Department's Center for Program and Planning Results to set ambitious targets and develop data collection methods for its long-term and annual performance measures.

Evidence: ?? ODEP Strategic and Performance Plan FY 2006-2010

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: All of ODEP's key partners, whether grantees or contractors, are required to provide performance and other project data (e.g., customer satisfaction) on a regular basis. Further, ODEP requires partners to maintain continuous and substantive communication with ODEP to ensure adequate project, program, and performance management. ODEP's data collection process is determined by the type of agreement established with partners. For example, ODEP systematically collects data, through an electronic management tool, from grantees. These data are shared with an independent evaluator for the purposes of long-term analysis and data verification. Data collection instruments are designed to assess the quality, relevance and utility of the interventions being tested. To improve data reliability, the independent evaluator reviews the data, checks for inconsistencies, and informs ODEP of any identified problems. ODEP follows up on the inconsistencies and changes are subsequently made to the data based on corrections from the sites. Data provided in the quarterly reports includes information that is used for the annual agency performance reporting. In addition to reporting to ODEP on key performance indicators, grantees also include a number of other indicators to measure progress toward meeting individual project objectives.

Evidence: ?? Sample Quarterly Report; ?? Independent evaluation Reports; ?? Sample SGA; ?? Sample Monthly Reports; ?? Sample invoice; ?? Sample Award Document (grant and contract).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ODEP monitors program partner performance on a continuous basis to ensure that project costs, schedules and performance results are acceptable. ODEP assigns an agency staff person as the Grants Officer's Technical Representative (GOTR) to each grant and cooperative agreement, and a Contracting Officers' Technical Representative (COTR), to each contract it awards. The GOTR or COTR responsible for monitoring the project works in conjunction with additional ODEP policy subject matter experts to monitor and assess cost, schedule and performance results during the life of the project. Should performance below what was required under the award, ODEP initiates procedures that prescribe a series of progressive steps that may include a formal corrective action plan to restore performance. Personnel serving as GOTRs or COTRs are required to be certified as such, and work under performance plans that incorporate clearly defined performance criteria regarding project performance management. All awards require that ODEP partners provide a budget and complete the project within a specified period of performance, and define the results anticipated upon project completion. SGA's and RFQ's inform potential partners of ODEP's mission and goals, and describe how the specific project is expected to support achievement of the mission and goals. For those awards that include option year renewals, awards state clearly that renewals are subject to project performance.

Evidence: ?? ODEP Program Management Standard Operating Procedures; ?? Sample Award Document (grant or cooperative agreements, and contract); ?? Sample Report (quarterly, monthly, financial and project); ?? Sample Invoice; ?? Sample manager and staff performance plan ?? Sample corrective action plan.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: ODEP obligates all funds annually appropriated within the fiscal year to directly support the program's purpose - to affect change within service delivery systems and among employers. Funding decisions are guided by ODEP's strategic plan and serve to assist ODEP's progress in achieving annual and long-term outcomes. As an agency serviced by the Department's Office of Financial Management Operations and Procurement and Contracts Office, ODEP follows all applicable Federal and Departmental laws, guidelines, etc. when obligating and tracking funds. All ODEP funded projects require spending reports to be submitted for review on a regular basis.

Evidence: ?? End of year Status of Funds Report (FY 2005); ?? Sample Grant and Contract award; ?? Sample SF-269; ?? Sample Invoice;

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ODEP is in the process of developing an efficiency measure, "Cost per policy and practice disseminated," for which a baseline will be set at the close of fiscal year 2007. ODEP is developing the methodology and will begin collecting data during FY 2006. Currently, ODEP demonstrates efficiency and cost effectiveness by procuring Human Resources Management, Financial Management (including contracting authority), IT Management, Competitive Sourcing, and other administrative management services from the Department's Office of Administration and Management.

Evidence: ?? ODEP FY 2007 Congressional Budget Justification submission (which includes ODEP's efficiency measure).

Does the program collaborate and coordinate effectively with related programs?

Explanation: As identified in the Congressional Justifications, ODEP collaborates and coordinates with related programs on a routine basis. ODEP has collaborated and coordinated with the Bureau of Labor Statistics, Census, SSA, NIDRR, and other federal agencies, to develop a valid and reliable method to measure the employment rate of Americans with disabilities. This has resulted in the development of a valid and reliable set of disability questions to be incorporated into the Current Population Survey. In addition, ODEP serves as DOL's representative to the Interagency Committee on Disability Research (ICDR), required under the Rehabilitation Act as amended. ODEP's collaboration has resulted in the establishment of a new ICDR Interagency Subcommittee on Employment. This subcommittee, co-chaired by ODEP, is designed to promote interagency collaboration and coordination on disability employment research. In addition, ODEP's ICDR activities have resulted in an MOU between ODEP and NIDRR formalizing coordination on disability employment research. ODEP continues to manage Disabilityinfo.gov, an award winning interagency web portal of Federal Government information about disabilities.

Evidence: ?? FY 2007 Congressional Justification ?? ICDR and ISE web site; ?? NIDRR MOU; ?? Section 188 Checklist and TEGL; ?? Mental Health Workgroup documents; ?? Disabilityinfo.gov (www.disabilityinfo.gov) materials.

Does the program use strong financial management practices?

Explanation: In compliance with the Federal Managers' Financial Integrity Act and the Federal Financial Management Improvement Act, ODEP provides quarterly certification to the Department that it exercises sound management systems and internal controls. Systems of internal controls are reviewed on a regular basis and any identified weaknesses are addressed. As an agency serviced by the Department's Office of Financial Management Operations, ODEP follows all applicable Federal and Departmental laws, guidelines, etc. regarding financial management. ODEP consistently receives a green score on the President's Management Agenda Financial Management Scorecard. ODEP has had no material control weaknesses reported. No non-compliance issues or audit findings have been reported. Grants Officer Technical Representatives (GOTR) and Contracts Officer Technical Representatives (COTR) monitor project costs through the receipt of periodic financial reports from grantees and the HHS/PMS reports, and contractors. This information is reviewed against program performance. In addition, GOTRs and COTRs monitor project costs through regular communication between DOL staff and project partners. All GOTRs and COTRs have received GOTR and/or COTR training. All grantees must comply with the financial management provisions of federal procurement regulations.

Evidence: ?? Financial Management documents (audit findings); ?? ODEP Quarterly Financial Management Certifications; ?? ODEP PMA Scorecard on Financial Management.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: According to ODEP's Division of Program Management Standard Operating Procedures for grant and cooperative agreement management and Departmental financial management guidance and policy for grants and contracts, ODEP monitors all of its projects to ensure management deficiencies are corrected in a timely manner when identified. All award recipients, whether grantees or contractors, are required to provide regular reports on project and financial management. When management deficiencies are identified, ODEP conducts a monitoring site visit to review performance and financial management. When necessary, ODEP recommends to the DOL/OASAM grant or contract officer an aggressive corrective action plan containing clear guidance and timelines for correcting the deficiencies. Additionally, ODEP has an internal strategic plan (FY 2006-FY 2010), used by management to prioritize and allocate resources to ensure ODEP's efforts support mission and performance achievement.

Evidence: ?? Sample Award Documents (grant or cooperative agreement, and contract); ?? ODEP Program Management Division Standard Operating Procedures; ?? Sample Corrective Action Plan; ?? ODEP Strategic and Performance Plan FY 2006 - 2010

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: There are limited data on which to base an assessment of ODEP's progress. The annual performance goals have been revised throughout the course of ODEP's existence making it difficult to measure the extent to which ODEP has been meeting its long-term goals.

Evidence: ?? ODEP Strategic and Performance Plan FY 2006 - 2010; ?? External evaluation reports; ?? USDOL Annual Performance Reports (FY 2002 - FY 2005).

Does the program (including program partners) achieve its annual performance goals?

Explanation: There are limited data with which to assess ODEP's progress on achieving its annual performance goals. Throughout its five years of operation, ODEP has consistently changed or updated its annual performance goals. In FY 2006, ODEP will work on setting ambitious targets and improve the data collection methods for its measures.

Evidence: ?? ODEP Performance Measures History through FY 2006; ?? USDOL Annual Performance Reports (FY 2002 - FY 2005); ?? Fiscal year 2005 Pilot Demonstration Program Result: Number Served and Entered Employment and Retention Rates chart.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ODEP developed an efficiency measure, "Cost per policy and practices disseminated," for which baseline data will be set at the close of FY 2007. Curently, ODEP demonstrates cost effectiveness by procuring Human Resources Management, Financial Management (including contracting authority), IT Management, Competitive Sourcing, and other administrative management services from the Department's Office of Administration and Management (OASAM).

Evidence: ?? FY 2006 Congressional Budget Justification; ?? FY 2007 Congressional Budget Justification.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ODEP is the only Federal agency mandated to develop comprehensive and integrated disability employment policy solutions never before tested or implemented by Federal, state, or local public or private entities, including employers. ODEP was created to fill a specific gap--the lack of any program that focused on comprehensive and integrated disability employment policy development.

Evidence: ?? NOD report ?? Task Force report ?? GAO report

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: ODEP's evaluations do not constitute rigorous evidence of program effectiveness. ODEP's evaluations have produced "process-related" findings that do not address whether the process changes resulted in key program outcomes, such as: (1) increasing the number of people with disabilities receiving services (compared to the number that would have received services in the absence of ODEP's efforts); and (2) improving employment and earnings of clients with disabilities (over what they would have experienced in the absence of ODEP's efforts).

Evidence: ?? WESTATs' evaluation reports of the Evaluation of Disability Employment Policy Demonstration Programs; ?? Outcome Analysis Project Report, DOL Annual Performance Reports; ?? ODEP Strategic and Performance Plan FY 2006 - 2010.