Detailed Information on the
Assistive Technology Alternative Financing Program Assessment

Develop a schedule so that Education can begin to analyze program data and submit the annual report to Congress on time.

Using new performance data to evaluate and improve program performance.

Replace existing annual output measure that calculates the number of loans per $1 million invested with an outcome efficiency measure that calculates the total amount loaned per $1 million invested.

Develop a data collection instrument for use by the 33 States that have received AFP funds.

Measure: The amount loaned per $1 million federal investment.

Explanation:This is a new efficiency measure for the Alternative Financing Program (AFP). The amount loaned by each state will vary greatly, depending on the type of loans states use to make alternative financing available (e.g., guaranteed loans, interest rate buy-down loans, non-guaranteed low interest loans, guaranteed and interest rate buy-down loans, and direct loans) and the difference in the amount paid from the AFP permanent account to cover administrative and program costs. The actual data represents the total amount loaned by all participating States per $1 million federal investment. A target of $750,000 for fiscal years 2006 and 2007 has been set. Fiscal year 2005 data are not yet available.

Measure: Number of loans to individuals with disabilities per $1 million Federal investment and State matching funds (measure no longer used).

Explanation:Measure does not capture performance; and has been replaced with an outcome measure that looks at the total amount loaned per federal investment.

Is the program purpose clear?

Explanation: The Alternative Financing Program (AFP) awards matching funds (50% Federal) to help States provide loans to individuals with disabilities (or their family members, guardians, advocates, and authorized representatives) to purchase assistive technology (AT) devices and/or services. AFP was first funded in FY2000.

Evidence: Assistive Technology (AT) Act of 1998 (P.L. 105-394); Title III (as in effect before the enactment of P.L.108-364).

Does the program address a specific and existing problem, interest or need?

Explanation: A July 2003 survey by the Rehabilitation Engineering and Assistive Technology Society of North America (RESNA), NIDRR's AT Technical Assistance contractor, documented that 29% of respondents had some unmet need for AT in their lives. In addition, multiple device use increases with severity. The percentage of individuals using at least three devices was about 50% higher for individuals with severe disabilities than for individuals with mild disabilities. Unmet AT needs across disability populations include in-home use and workplace environments with differences by severity of disability.

Evidence: A Secondary Analysis of the AT/IT Survey (RESNA, July 2003). This study provided information about AT needs within the general population but is not specific to the Alternative Financing Program.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The AT AFP does not duplicate other Federal AT programs, though it was added to the 1998 AT Act to complement the AT State Grants program. Within Education, the Rehabilitation Services Administration implemented a similar financing program in 2002/2003 to provide loans for individuals with disabilities to purchase computers and other equipment for the purpose of teleworking. The Access to Telework program however, has only received one year of funding to date.

Evidence: AT Act of 1998 (PL 105-394); Title III (as in effect before the enactment of P.L.108-364). The Rehabilitation Act of 1973, as amended (PL 105-220); §303(b). For descriptions of each loan program see Federal Register notice August 5, 2003--Alternative Financing Mechanism Program and Access to Telework Fund Program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Title III of the AT Act has no major flaws and States have flexibility to structure their models to provide loans. But provisions requiring a State minimum match of 50% to create a minimum AFP program of $1 million, and prohibiting States from receiving more than one grant were made inapplicable for the 2001, 2002/2003, and 2005 competitions (through appropriations language) to increase State participation rates by allowing States to apply for any size grant they could match with 25 percent State funds. A total of 31 States currently participate in the AFP.

Evidence: AT Act of 1998 (as in effect before the enactment of P.L.108-364); appropriations language requested annually in the President's Budget; contents of applications; and outcomes of competitive grant competitions. The alternative financing mechanisms may include a low interest loan fund; an interest buy-down program; a revolving loan fund; a loan guarantee or insurance program; a program operated by a partnership among private entities for the purchase, lease, or other acquisition of AT devices or AT services; or another mechanism that meets the requirements of the AFP and is approved by the Secretary.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: States and community based organizations are the program grantees because they have knowledge and experience with state and/or local concerns. Grantees establish policies and practices to address the program's purpose and target population. There is no evidence to suggest that grantees are not targeting the right individuals or that loan recipients do not need or merit the funding. However, ED needs to conduct analysis to determine if the current beneficiaries of the loans are individuals for whom traditional loans are unavailable. Specifically, does the AFP provide loans with better terms (i.e., lower interest rates, and/or extended repayment periods) to individuals that have been unable to obtain loans through public lending institutions?

Evidence: Annual performance reports provide information about program activities and beneficiaries. Specific loan data have been reported to ED by RESNA through the technical assistance grant awarded to collect AFP data.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: There are no long-term measures currently but a web-based outcomes reporting system is being developed through the AFP technical assistance grant. This system will collect information to address the program's purpose, including data on how AFP loans have helped transform the lives of people with disabilities in employment, education and independent living.

Evidence: Outcome data was reported in the Alternative Financing Mechanisms, Title III, AT Act of 1988, First Annual Report to Congress (May 2003). In 2003, the AFP and Telework Data Collection Work Group (Rehabilitation Services Administration) met to draft a logic model for AFP and Telework to guide decision-making for data collection, generate specific Telework data items, and review and modify AFP data elements as necessary.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program does not have ambitious targets.

Evidence:  

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: AFP has one annual measure -- the number of loans to individuals with disabilities per $1 million Federal investment and State matching funds. However, this information is not sufficient for program analysis and must be modified to include more detailed financial information such as the amount of funds loaned per funds invested by Federal sources and be tied to the long-term goal. The data elements to support these measures must be clearly defined in the instrument. For example, grantees will need to disaggregate funds being used to operate and administer the program, those made available for loans, those used as guarantee (and other models), and additional funding contributed by the lending institution partners.

Evidence: Forthcoming: ED will modify the annual measure currently contained in the GPRA plan, and develop a long-term goal/target.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The GPRA plan for AFP contained a target of 33 loans per $1 million invested for fiscal years 2004 and 2005. The AFP program is still too new and does not have enough reliable and consistent data to have a good baseline. ED may revise the existing target in the GPRA after developing new measures.

Evidence: Information on AFP loan activity in 2000 was published in the First Annual Report to Congress in May 2003.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The AFP does not tie grantee or contractor performance to program goals. However, ED will work with AFP partners on annual and long-term goals for the 2006 GPRA plan.

Evidence: The program does not have procedures in place to get partners to commit to, measure, and report on performance related to the program goals. There are questions on the survey tool to require AFP program managers to provide actual performance data for their grants.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There has not been an independent evaluation of the AFP. After setting long-term program goals, ED plans to develop a schedule for an independent evaluation. NIDRR had worked closely with AFP's TA provider who helped grantees in implementing their AFPs. Any evaluation will be based on reports and/or information developed by the TA provider.

Evidence: In addition to data collection and reporting activities, the AFP TA provider is responsible for assisting States with their applications for AFP grants, and with developing and implementing the AFP (AT Act, §306(a)(as in effect before the enactment of P.L.108-364). Poor AFP grantee performance may reflect poor performance by the TA provider.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The 2005 budget request (and appropriation language changes) were in response to State interest in the 2003 competition. However, this request, and previous requests, were not tied to either annual or long-term goals.

Evidence: Congressional budget justifications for fiscal years 2000 through 2005.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: NIDRR did not modify the annual measure currently contained in the GPRA plan, or develop a long-term goal/target.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AFP's 2000 loan activity and information was published in the First Annual Report to Congress in 2003. This report included aggregate data on approved loan amounts, applicants' demographic information (including disability type, income, primary language, geographic location, and employment status). There are currently two web-based reporting systems (program data and applicant data) that grantees have to use. NIDRR worked with the TA provider to improve the reliability of the data, and the consistency among the two data systems.

Evidence: AT AFP Annual Report to Congress, May 2003.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, ED has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps ' hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its SES staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. ??The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: In 2001, NIDRR lapsed about $1 million of the $14.7 million appropriations because many States did not submit acceptable applications (14 awards from 26 applications). In 2003, 26 grantees received $36M in funding (from funds appropriated in 2002 but made available through 2003). While funds were available for 2 years, awards were made on the last day of the second fiscal year.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. ??The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: To date, the Department has not established procedures for this program to measure and achieve efficiencies in program operations. However, ED is in the process of developing its competitive sourcing Green Plan, and is working to improve the efficiency of its grantmaking activities. The Department has also established a strengthened Investment Review Board to review and approve information technology purchases agency-wide.

Evidence: Department Investment Review Board materials. ED's Discretionary Grants Improvement Team (DiGIT) recommendations.

Does the program collaborate and coordinate effectively with related programs?

Explanation: For the 2003 competition, NIDRR worked with RSA (Access to Telework staff) to develop a joint regulatory priority. NIDRR coordinated with RSA on AT and Telework issues.

Evidence:  

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been identified for the AFP.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: However, NIDRR planned to improve on-going assessment of outcomes through annual program performance reports and the development of a data collection instrument. NIDRR developed a corrective action plan to address management deficiencies. The AFP has since been transferred to RSA.

Evidence: NIDRR corrective action plan.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The program was designed as a competitive program but it has not been implemented that way. Every State that applied for 2003 grants received funds. Grantees responded to a priority for funding published in the Federal Register that specified what requirements were necessary. Among other things, States were required to submit assurances in their application and to follow with their policy and procedures 6 months after the award of Federal funding. Application assurances were reviewed internally (by NIDRR rather than a panel of external peer reviewers) and all States received funds. Scores were not established nor were States put in any rank order. Because the total amount of States' requests for funds ($42M) exceeded the total amount available ($36M), each State's award was reduced by an equal percentage share.

Evidence: Federal register notice published on August 5, 2003, grantee applications, grantee policy and procedures, and ED's grant award database.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: NIDRR reviewed grantee policies and procedures and maintained detailed oversight to ensure that grantees met their required match within 12 months, as required. In addition, grantees provided NIDRR with annual progress reports. NIDRR worked to revise the data collection instrument that grantees will use to report data.

Evidence: Grantee policy and procedures.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: AFP's first annual report in 2000 was submitted to Congress in May 2003. However, this report has not been made available to the public in a meaningful manner. The second report has not been completed. Education is developing a department-wide approach to improve the way programs provide performance information to the public.

Evidence: NIDRR planned to take steps to ensure that programmatic information, including evaluation findings were accessible to the public by placing reports and updated program information on ED's website.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: NIDRR did not develop long-term performance goals.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: Because AFP was first funded in 2000 and the first targets were set for 2004, the program has not achieved it goals.

Evidence: AT GPRA plan; data collection instruments.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: 2000 and 2001 data showed 33 loans per $1 million invested. Data for 2002 is forthcoming.

Evidence: AT GPRA plan; First annual report to congress.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: We are not aware of comparable programs that provide alternative financing mechanisms to people with disabilities with which to compare the AFP.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: There has not been an independent evaluation of AFP but NIDRR collected follow-up data of individuals receiving loans or guarantees from AFP grantees. Pending data collection include: the impact of AT on those receiving loans, perceived change in quality of life as a result of AT received, program benefit, and overall satisfaction with services received from the AFP program.

Evidence: