Detailed Information on the
American Printing House for the Blind Assessment

Ensure that APH implements revised guidelines for evaluating the quality, utility, and relevance of products.

Conduct an analysis of existing performance measures based on survey data to determine if there are measures that are no longer needed.

Conduct an analysis of how Federal funds are being used at APH.

The Printing House shall implement rigorous performance measures related to the educational materials, advisory services, and educational and technical research activities at APH.

The Department will develop a formal mechanism and schedule for monitoring the federally funded programs for compliance with the Act to Promote the Education of the Blind.

The Department will take affirmative steps to assess the scope and quality of the program to ensure that it is operating effectively, addressing its statutory purpose, and achieving results.

Measure: The percentage of trustees who agree that the American Printing House's educational materials are appropriate, timely, and high quality and allow blind students to benefit more fully from their educational programs.

Explanation:The survey instrument used by APH was designed to measure the levels of customer/consumer satisfaction with each of the factors. The survey was distributed to 147 ex officio trustees, as well as to a variety of professional groups whose member work in the field of blindness, and was made available on the APH web site.

Measure: The percentage of advisory committee members who agree that the American Printing House's educational materials are appropriate, timely, and high quality and allow blind students to benefit more fully from their educational programs.

Explanation:The APH advisory committees are comprised of a subset of the Printing House's ex officio trustees. APH has two advisory committees, one for educational products that works on issues related to publications and product research and one for educational services that provides oversight regarding APH services.

Measure: The percentage of trustees who agree that the performance of students and their participation in educational programs improves as a result of the availability of educational materials provided by the American Printing House for the Blind.

Explanation:

Measure: The percentage of teachers who agree that the American Printing House's educational materials are appropriate, timely, and high quality and allow blind students to benefit more fully from their educational programs.

Explanation:

Measure: The percentage of teachers who agree that the performance of students and their participation in educational programs improves as a result of the availability of educational materials provided by the American Printing House for the Blind.

Explanation:

Measure: The percentage of consumers who agree that the American Printing House's educational materials are appropriate, timely, and high quality and allow blind students to benefit more fully from their educational programs.

Explanation:

Measure: The percentage of APH products sold that are new products

Explanation:This measure was instituted in FY 2006 to help determine the efficacy of new products developed by APH by maintaining a the balance between the percentage of APH products sold that are new products and sales of products previously developed. The intent of this measure is to maintain a minimum percentage of sales of new products as an indication that products developed by APH are relevant to consumers because they are buying the new products.

Measure: The percentage of APH product research deemed to be of high quality by an independent review panel of qualified experts or individuals with appropriate expertise to review the substantive content of the research projects.

Explanation:A panel of experts proposed by APH and approved by the Department will review the underlying research related to a random sample of new products on an annual basis to make an assessment of whether: (1) appropriate research methodologies are being used consistent with the type of product being developed, (2) data is gathered from a geographically diverse U.S. population and variety of potential user groups, (3) evaluation data is gathered from appropriately qualified individuals, (4) data is gathered from an adequate number of sources, (5) product development demonstrates use of field review data for modifications to the product prior to development, and (6) data are gathered on student/consumer outcomes related to use of the proposed product. The quality of the research related to product development is rated on a scale of 1 to 7, with 1 = Does not meet criteria and 7 = Exceeds criteria.

Measure: The percentage of APH products deemed to be of high relevance by an independent review panel of qualified experts or individuals with appropriate expertise related to the target audience.

Explanation:A panel of experts proposed by APH and approved by the Department will review the a random sample of new products on an annual basis to make an assessment of whether: (1) there is evidence of an examination of need for the product, (2) there is evidence that APH sought opinions of knowledgeble individuals to determine the need for the product, (3) APH made the decision to produce the product based on a standardized process for product selection, (4) the product addresses an identified need for individuals who are visually impaired, and (5) the product is fully accessible for the intended population. The relevance of the products is rated on a scale of 1 to 7, with 1 = Does not meet criteria and 7 = Exceeds criteria.

Measure: The percentage of APH products deemed to be of high utility by an independent review panel of qualified experts or individuals with appropriate expertise related to the target audience.

Explanation:A panel of experts proposed by APH and approved by the Department will review the a random sample of new products on an annual basis to make an assessment of whether: (1) sales of the new products indicate demand, (2) new product indicators, such as field evaluations and other feedback, substantiate that the product is meeting a need, and (3) the products are considered highly useful for individuals with visual impairments. The relevance of the products is rated on a scale of 1 to 7, with 1 = Does not meet criteria and 7 = Exceeds criteria.

Measure: Cost (in cents) per page of braille text. (New measure, added February 2007)

Explanation:In FY 2006, it cost APH 9.5 cents per page to produce braille (on the Braillo or similar computerized embosser). In FY 2005, APH printed 18,617,000 pages of braille, which represented approximately $1,769,000 and 12.2% of total sales. APH's goal is to reduce its costs by 10% by the end of FY 2011.

Measure: Cost (in cents) per page of color large type text. (New measure, added February 2007)

Explanation:In FY 2006, it cost APH 14.5 cents per page to produce color large type. In FY 2005, APH printed 21,927,000 pages of large print, which came to almost $3.2 million and represented 17.3% of its total sales. APH's goal is to reduce its costs by 10 % by the end of FY 2011.

Is the program purpose clear?

Explanation: The Act to Promote the Education of the Blind (the Act) spells out the Federal purpose for funding the American Printing House for the Blind (APH): to produce and distribute educational materials adapted for students who are legally blind and enrolled in formal educational programs below the college level. Materials produced by APH include textbooks in braille, large type, and digital formats; educational tools such as braille typewriters and computer software and hardware; teaching aides such as tests and performance measures; and other special supplies. The materials are distributed to programs serving individuals who are blind through allotments to the States. Allotments are based on an annual census conducted by the Printing House of the number of students who are legally blind in each State and are provided in the form of credits that are used to purchase educational materials from APH. State educational agencies and programs serving persons who are blind may order materials free-of-charge up to the amount of funds allocated to each State. In addition to producing educational materials in accessible versions, APH also provides training in the use of the materials and conducts research to develop new materials and accessible versions of textbooks.

Evidence: The Act to Promote the Education of the Blind of 1879, as amended.

Does the program address a specific and existing problem, interest, or need?

Explanation: Students with visual impairment need accommodations, including specialized versions of educational materials that have been adapted to meet their needs, to participate in the educational process. In 2004, APH served 56,913 individuals, including children from birth through high school and students enrolled in postsecondary training programs below the college level. Office of Special Education Programs (OSEP) records indicate that 29,240 students with visual impairments were served under the Individuals with Disabilities Education Act (IDEA) in school year 2001-2002 (the most recent year for which disaggregated child count data are available). All or most of these students would have required accessible versions of educational materials used by nondisabled students.

Evidence: APH Annual Report for 2004, APH data submissions, and the OSEP Data Analysis System.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: States are required to provide a free appropriate public education (FAPE) to all students with disabilities under IDEA and to provide reasonable accommodations under section 504 of the Rehabilitation Act of 1973. In fiscal year 2005, Federal support for special education under the IDEA Grants to States, Preschool Grants, and Grants for Infants and Families programs totaled over $11.4 billion. The State and local educational agencies that receive these funds are responsible for ensuring that all eligible children with visual impairments receive FAPE and are provided the special educational materials necessary for these students to participate in their educational programs. State and local educational agencies also have their own funding sources for education that may be used to obtain educational materials to serve students with visual impairments. Federal funding for APH, $16.9 million in fiscal year 2005, supplements funds available under IDEA and other sources. Providers such as States, local volunteer organizations, and private nonprofits such as BookShare also produce educational materials in braille, electronic text, and other accessible formats.

Evidence: Department records.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Department monitoring efforts indicate that APH operates in accordance with its governing legislation, is well managed, and involves stakeholders in decision-making regarding funds provided through the Act. However, by statute, the only entity eligible to receive Federal support through the Act to Promote the Education of the Blind is APH. There is no competition for these funds and limited Federal oversight of the program. Management and programmatic decisions are all under the purview of APH and its Board of Directors. Because of this statutory structure, funding is awarded to one specified grantee without regard to performance, and there are only minimal provisions in the authorizing statute to ensure that a high level of performance is maintained. The Department has not conducted evaluations of the effectiveness of the educational materials, advisory services, and educational and technical research programs at APH, whether the direct appropriation to APH is currently the best mechanism available to serve the target population, or if the program, as currently structured, represents the most efficient way to achieve the program purposes.

Evidence: Act to Promote the Education of the Blind and Department records.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The target population for this program is specified in statute. Department reviews of written materials and meetings with APH staff indicate that APH programs are all targeted toward the key purpose of providing specialized materials, products, and services needed for students who are visually impaired to participate in the educational process. In addition, the Department conducted a study in 1991 of APH's system of providing materials and received positive responses from States to the method by which APH distributes credits and materials. While the study was conducted some time ago, the method used to distribute credits and materials has not changed.

Evidence: The Act to Promote the Education of the Blind, Department records, and the Department's study of APH.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: In 2005, APH established a new measure related to the percentage of students who "attain identified concepts or skills" during the field testing of products in four areas - low vision, early childhood, multiple disabilities, and tactile graphics. The measure will evaluate the pre-test and post-test status of students who use different APH products. Since the purpose of providing accessible materials is to increase access to the regular curriculum for students with disabilities and to help them achieve academically, this new measure addresses student outcomes. However, this measure is marked "no" because at least two acceptable long-term outcome measures are necessary to get a "yes" for this question.

Evidence: The Government Performance and Results Act (GPRA) data submitted by APH.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The new long-term measure was just established in 2005. Targets, timeframes, and baseline data have not yet been established for the measure.

Evidence: GPRA data submitted by APH.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: APH has developed a set of performance measures that use surveys to gain information on ex officio and consumer satisfaction with the appropriateness, timeliness, and quality of products, and whether the materials allow blind students to benefit more fully from their educational programs. However, satisfaction surveys are highly subjective and the ex officio trustees and advisory committees, whose members are selected from APH's trustees, have a statutory relationship with APH whereby they are responsible for all materials provided to States from the appropriation for APH. There also is no external verification or analysis of data supplied by APH, and the teachers polled in the survey are identified by the ex officio trustees, rather than through random selection. A new measure is being added in 2005 that would assess the percentages of students who attain identified concepts or skills, which would be measured during the field-testing of APH products. This measure relates to the program's ultimate goal of helping students achieve in their academic programs. However, this measure has not been tested, and baseline data and annual targets have not been established. In addition, ED has not developed an efficiency measure for this program.

Evidence: GPRA data submitted by APH.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: APH has established baselines and targets for its existing annual measures. However, the question is scored "no" because the annual measures are not acceptable. In addition, the baselines and targets vary between 95% and 100%. The baselines, actual performance data, and targets are improbably high, are based on subjective information, and have not been validated externally. They also do not leave room for much, if any, improvement. A new measure is being added in 2005. However, this measure has not been tested, and baseline data and annual targets have not been established. Baseline data for the new measure are anticipated in 2005.

Evidence: GPRA data submitted by APH.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Materials provided by APH are only one factor affecting the progress of students who are visually impaired, which makes it difficult to assess the impact of this program on student outcomes such as the academic performance of students, their level of participation in the educational programs, and access of students who are visually impaired to the general curriculum. However, APH is committed to developing appropriate measures of outcomes associated with its programs. The Department and APH worked collaboratively in developing its performance measures, including the new long-term measure related to assessing the percentages of students who attain identified concepts or skills, which APH plans to pilot test in 2005. At the program level, APH works closely with State educational agencies, schools for the blind, organizations involved with visual impairment, and teacher training programs at institutions of higher education. This question is marked "no" because the program does not have at least two acceptable long-term outcome measures.

Evidence: GPRA data submitted by APH and Department records.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Department released a study of APH in June 1991 that focused on methods of distributing funding and products. In addition, in 1987, the Office of the Inspector General conducted a review of the appropriations awarded to APH, including a review of the costs associated with producing educational materials. These studies were limited in scope and are now outdated. There are no current studies or evaluations of the scope or effectiveness of APH's programs or how well it is addressing its statutory purpose or the needs of its service population.

Evidence: Study of APH, 1991, and Department records.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: APH provides detailed information on materials produced and its advisory services and research activities. The Department uses this information in formulating its budget requests and budget materials clearly indicate what APH activities they support. However, the budget requests are not explicitly tied to accomplishment of annual and long-term performance goals. While the Department reports on performance data submitted by APH, the Department is not able to determine the impact of particular levels of funding on the performance of APH in relation to annual and long-term performance measures.

Evidence: Department budget requests and justifications and GPRA data.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Department has identified strategic planning deficiencies related to the performance measures and is working together with APH to address these deficiencies, particularly the need for student outcome measures. The new measure being piloted in 2005 relates to percentages of students who attain identified concepts or skills is a result of this collaboration.

Evidence: GPRA data submitted by APH and Department records.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Printing House regularly provides detailed data on program operations and its use of the funds provided through the Federal appropriation. While Department budget requests are not explicitly tied to accomplishment of the annual and long-term performance goals for NTID, the Department has used the data in monitoring activities, including assessments of the relevance of research and outreach activities, and to help inform its budget requests for the program.

Evidence: GPRA data, APH annual reports, budget submission data, and Department records.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Department of Education employees are subject to the EDPAS system, which links employee performance to success in meeting the goals of the Department's strategic plan. Staff associated with this program are provided individual performance agreements that are designed to measure the degree to which the employee contributes to improving program performance. The performance of APH and its State partners is monitored through reviews of annual data submissions and periodic meetings with APH staff.

Evidence: Budget submission data and reports from APH, Department records, and GPRA data.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: In 2004, an issue arose regarding the timely obligation and expenditure of funds by APH. This issue was resolved, and all funds are currently obligated within the timeframe set by the Department, and there are no carryover funds beyond that period. All funds are spent for the intended purpose.

Evidence: APH annual reports and Department records.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: APH reports each year on new developments in technology for the manufacture or distribution of materials that may affect its efficiency, product line, or other programs. For example, in 2004, APH installed a larger server that is big enough to allow storage of all recorded books and large type masters. This improved efficiency and turnaround time for both areas due to the reduction in time to retrieve masters and load them into the reproduction system required to make cassettes or large type books. However, the Department does not have procedures to measure the efficiency and cost effectiveness of such initiatives or to monitor program execution. The Department is examining options related to developing an efficiency measure for this program, but has not yet established the measure.

Evidence: APH data submissions and reports and Department records.

Does the program collaborate and coordinate effectively with related programs?

Explanation: APH coordinates with Recording for the Blind and Dyslexic, Inc., and the Library of Congress, the two other major Federally funded programs providing educational materials for individuals with visual impairments in order to avoid duplication of effort. APH's close work with State and local educational agencies, schools for students with visual impairments, vocational rehabilitation agencies, and training programs for personnel that will be working with students with visual impairments and the feedback it receives from these entities and individuals has helped to ensure that customers are satisfied with the timeliness, quality, and appropriateness of the educational materials it provides. In addition, APH maintains close ties with all major organizations for individuals who are visually impaired. The Office of Special Institutions is placed in the Office of the Assistant Secretary of the Office of Special Education and Rehabilitative Services (OSERS). There is little collaboration or coordination between the Office of Special Institutions, other OSERS offices, and other Department offices involved in general education or the education of individuals with visual impairments. However, while complementary, the programs administered by these offices do not directly intersect or require coordination in order to operate effectively. In a related effort, APH is coordinating with Office of Special Education Programs staff and programs related to the implementation of the National Instructional Materials Accessibility Standard and National Instructional Materials Access Center to ensure that the programs are implemented appropriately.

Evidence: APH data submissions and Department records.

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. The Department has a system for identifying excessive drawdowns and can put individual grantees on probation or designate the grantee as high risk. This would require Departmental approval of all grantee drawdowns. ED uses a number of tools and practices to ensure strong financial management. ED has strengthened its financial management practices, as recommended by GAO in September of 2004, by implementing e-Monitoring on December 22, 2004. E-Monitoring is a new software tool that enables grant program staff to better monitor grants and helps prevent potential fraud, waste, and abuse. Using e-Monitoring and GAPS data, along with other reports, allows the Department to monitor the status of a grantee's performance across programs. Further, these systems are used to flag institutions determined not to have applied funds appropriately and to withhold funding, if necessary. At the program level, APH conducts an annual independent audit of its programs and incorporates audit figures related to the Federal program into its annual report. Department interactions with APH staff indicate strong financial management. For example, APH's accountants self-identified a potential problem with carryover funds in 2004 and worked closely with Department staff to reach a solution.

Evidence: APH annual reports and Department records.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Office of Special Institutions does not have a formal mechanism for monitoring APH operations or a regular schedule for program monitoring. The Department periodically meets with APH in Washington. However, these meetings primarily consist of information sharing activities. The Department has not conducted a monitoring visit at the Printing House since 1989.

Evidence: Department records.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Department staff meet periodically and communicate frequently with APH representatives. In addition, APH provides detailed responses to Department requests for program information. However, the Department does not conduct site visits to the program, track actual expenditures to verify that funds are used for their designated purpose, assess program data quality, or assess APH's compliance with the requirements of the Act to Promote the Education of the Blind.

Evidence: Department records.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Detailed performance data are collected annually and shared with Congress through the budget process. GPRA data are posted on the Department web site and provided to Congress. In addition, entries on the OSERS web site provide a link to the APH web site. The APH website contains extensive information about the Federal program. APH also publishes a monthly newsletter to the ex officio trustees and other parties related to items of interest at the Printing House, including the Federal program.

Evidence: APH reports and data submissions, Department's justification materials submitted to Congress, and the Department's website link - www.aph.org/ and GPRA data - www.ed.gov/about/reports/annual/2005plan/edlite-usc-americanprinting.html.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The long-term goal was established in 2005. However, this measure has not been tested and baseline data and annual targets have not been established. Baseline data are anticipated in 2005.

Evidence: GPRA information submitted by APH.

Does the program (including program partners) achieve its annual performance goals?

Explanation: APH has met or exceeded each of its existing performance measures, as indicated in the Department's GPRA reports. However, while the data collected primarily relate to outcome measures, it is not clear how useful the information is as objective measures of APH performance since satisfaction surveys are highly subjective. A new measure being added in 2005 would relate to the program's ultimate goal to provide specialized materials to help students who are visually impaired achieve in their academic programs. However, this measure has not been tested and baseline data and annual targets have not been established.

Evidence: APH GPRA information.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The results of APH surveys of ex officio trustees on the timeliness and quality of its educational materials are very positive. In addition, APH reports each year on new developments in technology for the manufacture or distribution of materials that may affect the Printing House's efficiency, product line, or other programs. For example, APH reported that it reorganized its tape duplication area and installed a third Braillo 400 computerized braille printer. However, APH did not report on or quantify the cost or time saved as a result of these measures, show change over time, or indicate the relative significance of the initiatives. The Department does not have procedures to measure the efficiency and cost effectiveness of such initiatives or to monitor program execution. The Department has has not yet established an efficiency measure for this program.

Evidence: APH data submitted to the Department.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Department has not conducted any analyses of APH's performance relative to other programs serving students with visual impairments. However, there are no comparable federally funded programs focused specifically on providing accessible educational materials in the same format for students with visual impairments, and no performance data on private programs are available.

Evidence: N/A

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Department released a study of APH in June 1991 that focused on methods of distributing funding and products. In addition, in 1987, the Office of the Inspector General conducted a review of the appropriations awarded to APH, including a review of the costs associated with producing educational materials. These studies were limited in scope and outdated. There are no current studies or evaluations of the scope or effectiveness of APH's programs or how well it is addressing its statutory purpose or the needs of its service population.

Evidence: Study of APH, 1991, and Department records.