Detailed Information on the
IDEA Technology and Media Services Assessment

Develop baselines and targets for the program's 2 long-term performance measures.

Develop a baseline and targets for the program's efficiency measure.

Develop a strategy for evaluating the impact and effectiveness of program activities.

Establish baselines and targets for two of the three annual program performance measures dealing with quality and usefulness of program products and services.

Measure: The percentage of technology and media projects judged to be of high relevance.

Explanation:Data for the measure will be gathered through an expert panel review of products and services provided by projects.

Measure: The percentage of technology and media projects judged to be of high quality.

Explanation:Data for the measure will be gathered through an expert panel review of products and services provided by projects.

Measure: The percentage of technology and media projects that produce findings, products, and/or services that contribute to improving results for infants, toddlers, children and youth with disabilities.

Explanation:Data for the measure will be gathered through an expert panel review of products and services provided by projects.

Measure: The percentage of technology and media projects that develop and validate technologies that incorporate evidence based materials and services.

Explanation:Data for the measure will be gathered through an expert panel review of products and services provided by projects.

Measure: The percentage of technology and media projects that make technologies that incorporate evidence-based practices available for widespread use.

Explanation:Data for the measure will be gathered through an expert panel review of products and services provided by projects.

Measure: The federal cost per unit of technology and media services for the Special Education Technology and Media Services program, by category, weighted by an expert panel quality rating. (New measure, added February 2007)

Explanation:Data will be collected through expert panel reviews of technology and media products. The Baseline for 2006 was developed using only data for Media Services activities and is expected to be revised for 2007.

Is the program purpose clear?

Explanation: The purposes of the Technology and Media Services program are clearly stated in authorizing legislation. They include promotion of the development, demonstration, and use of technology, and educational media activities. Amendments to the IDEA in 1997, and particularly in 2004, have increased the focus of activities on education, and program priorities in recent years have reinforced the link between technology and media activities.

Evidence: Individuals with Disabilities Education Act Section 674 (http://thomas.loc.gov/cgi-bin/query/F?c108:1:./temp/~c108IgCG6G:e468891:) Application notice for Access to Emerging Technologies competition http://www.ed.gov/legislation/FedRegister/announcements/2006-2/041306a.pdf

Does the program address a specific and existing problem, interest, or need?

Explanation: Technology in a variety of forms can open opportunities for children with disabilities. Some technology provides access to materials that would otherwise be unavailable to children. For example, video description and captioning provide access to educational materials for visually and hearing impaired children. Other technology can be used to directly address the instructional needs of children with specific learning problems, such as learning disabilities. Program activities range from the adaptation of technology through the actual implementation of technology based approaches.

Evidence: Center for Implementing Technology in Education (http://www.citeducation.org/) Family Center on Technology and Disability (http://www.fctd.info/) NIMAS at CAST (http://nimas.cast.org/) NIMAS is the National Instructional Materials Accessibility Standard. CAST operates a NIMAS Development Center and a NIMAS Technical Assistance Center. IDEA section 612(a)(23) now requires States to adopt the NIMAS for the purpose of providing instructional materials to blind persons or other persons with print disabilities. See http://thomas.loc.gov/cgi-bin/query/F?c108:1:./temp/~c108qT40fS:e96317: Captioned Media Program (http://www.cfv.org/)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program is designed to support other Federal, state, local, and private efforts dealing with using technology to improve results for children with disabilities. For example, the National Instructional Materials Accessibility Standard (NIMAS) Technical Assistance Center works with states, school boards, and publishers to raise awareness of the benefits of accessible materials. However, activities that adapt technology have the possibility for overlapping other Federal programs such as research conducted by the Institute of Education Science or the National Institute on Disability and Rehabilitation Research to the extent that those organizations might choose to focus resources on technology for children with disabilities. Competitions for providing accessible media are designed specifically to avoid overlap with other sources of accessible media.

Evidence: NIMAS at CAST (http://nimas.cast.org/) NIMAS is the National Instructional Materials Accessibility Standard. CAST operates a NIMAS Development Center and a NIMAS Technical Assistance Center. IDEA section 612(a)(23) now requires states to adopt the NIMAS for the purpose of providing instructional materials to blind persons or other persons with print disabilities. See http://thomas.loc.gov/cgi-bin/query/F?c108:1:./temp/~c108qT40fS:e96317: Institute of Education Sciences - National Center for Special Education Research (http://www.ed.gov/about/offices/list/ies/ncser/index.html) National Institute on Disability and Rehabilitation Research (http://www.ed.gov/about/offices/list/osers/nidrr/index.html) Application notice for Access to Emerging Technologies competition http://www.ed.gov/legislation/FedRegister/announcements/2006-2/041306a.pdf

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program does not have design flaws that prevent it from pursuing its purposes. The program authority is sufficiently broad to afford a variety of means for achieving program objectives. We have no evidence that other approaches toward pursuing these purposes would be more effective or efficient than the ones currently being used.

Evidence: Individuals with Disabilities Education Act Section 674 (http://thomas.loc.gov/cgi-bin/query/F?c108:1:./temp/~c108IgCG6G:e468891:)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Much of the funding under the program is provided for centers that are targeted on serving particular purposes. For example, the Family Center on Technology and Disability is designed specifically to provide information on assistive technology to organizations that work directly with families. Multiple awards are made for adapting technology (Steppingstones competition) in order to encourage innovation. Multiple awards are also made to support accessible media in order to promote a wider distribution of these media. In both areas where multiple awards are made, the intended purposes of the awards are clearly stated in application notices.

Evidence: Family Center on Technology and Disability (http://www.fctd.info/) Center for Implementing Technology in Education (CITEd) http://www.citeducation.org/ Steppingstones of Technology Innovation for Children with Disabilities http://www.ed.gov/legislation/FedRegister/announcements/2005-3/090105a. Application notice for Access to Emerging Technologies competition http://www.ed.gov/legislation/FedRegister/announcements/2006-2/041306a.pdf

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Two long-term measures have been developed this program. These measures focus on the extent to which the program (1) develops and validates and (2) makes available evidence-based technology. These two measures will help the Department determine how effective the projects funded by this program have been deployed into the communities that need technological innovation. The Department has identified 6 critical areas that must be addressed to improve results for children with disabilities; assessment, literacy, behavior, instructional strategies, early intervention, and inclusive practices. Baselines and goals for long-term measures will be established to focus resources on these areas.

Evidence: PARTWeb Measures. See 4th measure (projects develop and validate technologies) and 5th measure (projects that make technologies that incorporate evidence-based practices).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program does not at this time have targets for its long-term measures. Baseline data will become available in the fall of 2006 and in the fall of 2007. SUBJECT TO OMB APPROVAL OF TARGETS, THIS ANSWER MAY BE CHANGED FROM NO TO YES WHEN BASELINES AND TARGETS BECOME AVAILABLE IN THE FALL.

Evidence: PARTWeb Measures. See 4th measure (projects develop and validate technologies) and 5th measure (projects that make technologies that incorporate evidence-based practices).

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Department believes that there are 3 interrelated requirements for the program to achieve its purposes. The projects funded must be relevant (i.e. the right kinds of projects), be of high quality, and the projects' findings, products, and/or services be used to improve results. Together, these three measures will help identify the extent to which funded projects benefit the disability communities. While the program does not currently have an efficiency measure, it is in the process of developing one. TO RECEIVE A YES WE HAVE TO SHOW THAT WE ARE IN THE PROCESS OF DEVELOPING AN EFFICIENCY MEASURE. A SCHEDULE FOR THE ADOPTION OF THE MEASURE MUST BE APPROVED BY OMB.

Evidence: PARTWeb measures.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program does not at this time have baselines or targets for its annual measures. Baseline data from which targets can be set will become available in the fall of 2006. SUBJECT TO OMB APPROVAL OF TARGETS, THIS ANSWER MAY BE CHANGED FROM NO TO YES WHEN BASELINES AND TARGETS BECOME AVAILABLE IN THE FALL.

Evidence: PARTWeb Measures.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program goals are typically embedded in program priorities. In addition, about half of Technology and Media Services awards were made through cooperative agreements where awardees and OSEP staff work together to define and achieve project goals. Most of the remaining awards are made through grants to develop and adapt technology to meet the needs of children with disabilities. Most awardees, including recipients of awards to adapt technology, are required to attend annual meetings in Washington, DC, where program goals and the evaluation of program activities are key topics. Progress is reported annually and sometimes quarterly to the Department.

Evidence: Application notice for Technology Implementation Center (Award made to Center for Implementing Technology in Education (CITEd)) http://www.ed.gov/legislation/FedRegister/announcements/2004-2/062804b.pdf INSERT MEDIA DISTRIBUTION APPLICATION ANNOUNCEMENT WHEN PUBLISHED Steppingstones of Technology Innovation for Children with Disabilities http://www.ed.gov/legislation/FedRegister/announcements/2005-3/090105a. Regional Resource Center and Federal Center Network http://www.rrfcnetwork.org/ See conference information. INSERT COPY OF ANNUAL REPORT IF AVAILABLE.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: No independent evaluations of program activities in general have been conducted for the program, though project evaluations are conducted in the second year of some 5-year awards. While these "3+2" evaluations are used primarily for determining whether a project will be funded in its fourth and fifth years, valuable information is also gathered that is used in program management.

Evidence: INSERT APPLICATION NOTICE FOR 327N IF IT INCLUDES 3+2 WHEN PUBLISHED 3+2 Evaluations.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests have not been tied to annual and long-term performance goals. There are no annual and long-term performance goals for the program at this time.

Evidence: Congressional Justifications for the Technology and Media Services program. PARTWeb Measures.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has addressed and is continuing to address strategic planning deficiencies in two areas. First, it has developed and obtained approval for both annual and long-term performance measures, and is now working to collect data on these measures. Second, it has prepared a draft Comprehensive Plan for Technology and Media Services and other programs as required by section 681(a) of the IDEA.

Evidence: PARTWeb Measures. Individuals with Disabilities Education Act Section 674 (http://thomas.loc.gov/cgi-bin/query/F?c108:7:./temp/~c108xT227T:e477488:) INSERT FEDERAL REGISTER PAGE IF COMPREHENSIVE PLAN IS PUBLISHED CITE ANNUAL PERFORMANCE REPORT WHEN ISSUED

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: OSEP collects information from awardees in a timely manner through ongoing contact between the grantee and the agency project officer. This is done by regular conference calls, e-mail exchanges, or written communication with the awardee. Based on communications with awardees the Steppingstones competition was redesigned to fund research activities and development activities in two separate phases. Annual performance reports and final reports are required and are reviewed by staff, and 3+2 evaluations are conducted for some projects funded at more than $500,000 per year. Projects that include captioning and video description submit monthly reports to the project officer. Information is used to make improve corrective actions when necessary. For example, reports showing that captions are failing to appear may require corrective action to address transmission difficulties. However, baseline data for GPRA measures will not be available until the fall of 2006. A YES REQUIRES THAT "THE PROGRAM HAS COLLECTED THE BASELINE PERFORMANCE DATA NECESSARY TO SET MEANINGFUL, AMBITIOUS PERFORMANCE GOALS." THE ANSWER TO THIS QUESTION MAY BE CHANGED TO YES WHEN APPROVED GPRA BASELINES AND TARGETS ARE ESTABLISHED.

Evidence: Application notice for Television Access competition http://www.ed.gov/legislation/FedRegister/announcements/2005-2/042905b.pdfED Steppingstones of Technology Innovation for Children with Disabilities http://www.ed.gov/legislation/FedRegister/announcements/2005-3/090105a. Annual reports from projects.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to EDPAS, which links employee performance to relevant Strategic Plan goals and action steps, and is designed to measure the degree to which a manager contributes to improving program performance. Awardees are held accountable through performance reports and annual reviews. Awardees will also have their products and services reviewed in determining performance for the annual GPRA measure on quality.

Evidence: EDPAS agreements. Annual reports from projects. PARTWeb Measure. 3rd measure dealing with quality

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: OSEP successfully obligates funds by the end of each fiscal year (but mostly late in the year). Funds are spent for intended purposes (as assessed through grant and contract monitoring, and project reviews of some of the larger award recipients) . We have not identified improper uses of funds.

Evidence: Finance reports, notices of competitions, lists of funded applications. 3+2 evaluations.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Unless otherwise required by law, awards are made on a competitive basis. Since 2000, the Department has accepted applications through its e-Application system. In order to expand on that effort, it is now participating in the new government-wide Grants.gov Apply site. However, the program does not have an approved efficiency measure. SUBJECT TO OMB APPROVE OF AN INDICATOR, BASELINE, AND TARGETS, THE RESPONSE TO THIS QUESTION MAY BE REVISED TO YES.

Evidence: Individuals with Disabilities Education Act Section 674 (http://thomas.loc.gov/cgi-bin/query/F?c108:1:./temp/~c108IgCG6G:e468891:) (See section 674(a)(1) Steppingstones of Technology Innovation for Children with Disabilities http://www.ed.gov/legislation/FedRegister/announcements/2005-3/090105a. (See Application and Submission Information - a. Electronic Submission of Applications) http://e-grants.ed.gov/egWelcome.asp (See e-Application) Grants.gov Apply site (http://www.grants.gov)

Does the program collaborate and coordinate effectively with related programs?

Explanation: Awardees are required to collaborate and coordinate with other awardees and with other programs and entities as appropriate. This collaboration and coordination is of particular importance to the program in light of its vertical design, which focuses on developing technology applications and putting those applications into practice to improve results for children with disabilities. For example, a 2006 competition for a Center to support Technology Innovation for Students with Disabilities has as one of its primary purposes "to promote partnerships and collaboration among researchers, developers, vendors, and other appropriate entities" to expand the use of technology in serving children with disabilities. Representatives on the Centers advisory board must include, among others, representatives of Federal agencies and programs.

Evidence: Application Notice for Center to Support Technology Innovation for Students with Disabilities http://www.ed.gov/legislation/FedRegister/announcements/2005-4/121605e.pdf

Does the program use strong financial management practices?

Explanation: Auditors have not reported internal control weaknesses. The Department has a system for identifying excessive draw downs, and can put individual grantees on probation where draw downs need to be approved.

Evidence:

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has taken meaningful steps to address its management deficiencies. For example, the President's Commission on Special Education identified the "peer review" process as an area of weakness in current program management practices. In response, OSEP has provided internet training on the peer review process. Annual and long-term performance measures have been developed and approved by OMB. Most recently, OSEP has produced a draft Comprehensive Plan, which includes the Technology and Media Services program, as required by the Individuals with Disabilities Education Improvement Amendments of 2004. Program personnel are also in frequent contact with awardees to identify and address any problems that may arise.

Evidence: President's Commission on Excellence in Special Education report (http://www.ed.gov/inits/commissionsboards/whspecialeducation/reports/index.html) See "Section 7: Special Education Research and Dissemination of Information", "Improve the Current Grant Review Process" PARTWeb measures. Requirement for Comprehensive Plan http://thomas.loc.gov/cgi-bin/query/F?c108:7:./temp/~c108bApkrj:e477488:

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Unless otherwise required by law, awards are made on a competitive basis. However, for many years there have been appropriation earmarks that require funds to be provided to Recording for the Blind and Dyslexic, Inc. and the Greater Washington Educational Television Association. In 2006, funds earmarked to these two entities constituted 35 percent of the funds appropriated for the program. The authorizing legislation for the program also requires that funds be awarded to the American Printing House for the Blind for a National Instructional Materials Access Center.

Evidence: Department of Labor, Health and Human Services, and Education, and Related Agencies Appropriations Act, 2006, P.L. 199-149 (http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=109_cong_public_laws&docid=f:publ149.109) See Title III Page 119 STAT 2865 - 2866 Special Education Individuals with Disabilities Education Act Section 674 (http://thomas.loc.gov/cgi-bin/query/F?c108:1:./temp/~c108IgCG6G:e468891:) See sections 674(a)(1) and (e).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: OSEP reviews contractor and grantee performance through deliverables, periodic (sometimes monthly) reports, annual performance reports, and final reports. It also meets at least annually with all awardees, and, in the case of some recipients of 5- year awards, conducts a special review of project activities during the second year of the award. When necessary, project site visits are conducted.

Evidence: 3+2 Evaluations. Annual project reports.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Some aggregated data on awardees performance will be made available through GPRA measures. Data on these measures will be available in the fall of 2006. With regard to individual awards, the program collects annual, and sometimes more frequent, performance data from awardees. It also collects performance data through the awardee's final reports. However, this data is not readily accessible to the public.

Evidence: PARTWeb Measures. Annual project reports.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program does not have data showing progress toward achieving its long term-performance goals. Baseline data and targets will be available in the fall of 2006. However, data showing progress against targets will not be available until subsequent years.

Evidence: PARTWeb Measures

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual performance goals have not been established for the program. Therefore, progress in achieving goals cannot be assessed.

Evidence: PARTWeb Measures.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program does not yet have an efficiency measure.

Evidence: PARTWeb Measures

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no comparable programs to which the Technology and Media Services program can be compared.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: There has not been an independent evaluation of the program.

Evidence: