Detailed Information on the
Ready to Learn Television Assessment

Developing new performance metrics that measure the impact and quality of Ready to Learn television programming content.

Develop and collect data for at least one program efficiency measure to determine how efficiently the program accomplishes one or more of its key outcomes.

Collect baseline performance data and establish targets for new annual performance measures.

Establish long-term performance measures.

Implement planned program evaluations for each RTL grantee, to obtain reliable program outcome information.

Developing new performance metrics that measure the impact and quality of Ready to Learn television programming content.

Creating a program evaluation strategy, along with a schedule for an independent program evaluation, to obtain reliable program outcome information.

Measure: Percentage of children's television programming supported by the program deemed to be of high quality by an independent review panel of qualified experts.

Explanation:

Measure: Percentage of targeted outreach products and services supported by the program deemed to be of high quality by an independent review panel of qualified experts.

Explanation:

Measure: Dollars leveraged from non-Federal sources over 5 years per Federal dollar dedicated to core non-outreach program activities. (New measure, added February 2007)

Explanation:Targets will be established for every 5th year, and data will be reported cumulatively on an annual basis. Data are not yet available.

Is the program purpose clear?

Explanation: The mission and primary purposes of Ready-to-Learn television (RTL) are clearly delineated in the program's authorizing statute as follows: (1) support the development of educational programming for preschool and elementary school children (and their parents) and the accompanying support materials and services that promote the effective use of such programming; (2) develop programming (and digital content containing RTL-based children's programming) and resources for parents and caregivers that are specifically designed for nationwide distribution over public television stations' digital broadcasting channels and the internet; and (3) support contracts with public telecommunications and related entities to ensure that programs are widely distributed.

Evidence: Elementary and Secondary Education Act (ESEA), Title II, Part D, Subpart 3

Does the program address a specific and existing problem, interest or need?

Explanation: The RTL program is designed to address several specific, well-defined interests, including: (1) promoting early literacy and school readiness for children (particularly for disadvantaged preschool and elementary school children); (2) enabling parents, caregivers, and child educators to interact more effectively with children by promoting positive, interactive behaviors -- such as daily reading to children, and interactive television viewing through active use of the RTL skills "triangle" (View, Read, and Do). Research shows that children who read well in the early grades are far more successful in later years; and those who fall behind often stay behind when it comes to academic achievement (Snow, Burns and Griffin 1998). Given the prevalance of technology in the home, it is also critical to provide learning-centered media content that is attractive to children and their caregivers -- rather than content that is driven solely by marketing potential.

Evidence: "The State of Children's Television Report - Programming for Children Over Broadcast and Cable Television.", Annenberg Public Policy Center at University of Pennsylvania 1999; One Mission, Many Screens: A PBS/Markle Foundation Study on Distinctive Roles for Children's Public Service Media in the Digital Age. April 17, 2002. An excerpt -- "According to the Annenberg Public Policy Center "Media in the Home 2000" report, the average family with children 2-17 had almost three television sets.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The RTL program is uniquely situated to provide meaningful learning opportunites for children, parents, and caregivers in home and community settings. While the RTL target population (e.g., young children, parents, and care providers) is served by numerous other Federal, State, and local programs, the RTL program is primarily distinguished by existing evidence on the quality and design of its programming for children. Several non-Federal analyses have concluded that RTL funded programming is of higher educational quality than comparable children's programming. Also, while other Federal resources may support the production of publicly broadcast programming (primarily through the Corporation for Public Broadcasting), RTL is the only dedicated source of funding for the creation of original children's programming.

Evidence: See: "Data Collection of the Federal Performance Indicators for PBS Ready to Learn: Year 4 Summary Report." WestEd. August,2004; "One Mission, Many Screens: A PBS/Markle Foundation Study on Distinctive Roles for Children's Public Service Media in the Digital Age." April 17, 2002. ESEA, Title II, Part D, Subpart 3

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program has no major design flaws that prevent it from meeting its defined objectives and performance goals, and there's no evidence that other approaches could achieve the program's intended purposes more efficiently. However, the program model may be strengthened by developing a children's media advisory committee -- involving producers from varied platforms, advocates, researchers, journalists, artists, authors, teachers and others--to improve show content. In addition, there is some concern that local PBS affiliates air program episodes in poor time slots (e.g., during hours when target audiences are in school). Notwithstanding those minor flaws, the program model is sound and the current grantee (Public Broadcasting Services) uses Federal investments to leverage significant additional resources (e.g., program production, marketing, and distribution costs) from private production companies.

Evidence: The "One Mission, Many Screens" report suggests several ways that the program design might be improved, including ESEA, Title II, Part D, Subpart 3.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Given that program funds support both programming and outreach activities that have been found to reach a significant number of parents and children in the targeted age-groups, it is believed that RTL is well-targeted. Because RTL grantees tend to be commercial non-profit television production entities (e.g., PBS, CPB) the program is not characterized by "acceleration of activities that increase profits for a business." Recent evidence suggests that the RTL outreach activities alone are unlikely to lead to measurable improvements in student academic achievement -- insofar as this is true, the Department may wish to reconsider the design and implementation of current RTL outreach activities.

Evidence: Nielson Ratings reports on viewership for children ages 2 to 5 years-old, and 6 to 11 years-old show that during an average week in the 2001-2002 season "Between the Lions" reached approximately 2.4 million children (ages 2-11), and "Dragon Tales" reached approximately 6 million. ESEA, Title II, Part D, Subpart 3. Data collected by evaluation entities show that the RTL ancillary activities supported under this program are highly effective at reaching a range of target audiences; however, similar data also suggest that such activities have little or no measurable effect on child learning outcomes.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Department of Education (ED)and OMB are developing several long-term outcome and output measures that are likely to stem from the current PBS cooperative agreement (e.g., develop a minimum of 2 new children's shows during the 5-year award period; contribute to improvements in the learning and development of young children at risk of educational failure because of poverty, race, geography, limited English proficiency, or disability); no such goals have been formally adopted as "long-term performance measures."

Evidence: "Special Provisions Cooperative Agreement Between the US Department of Education and the Public Broadcasting Service." July 30, 2000. PR/Award #: R295A00002

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: N/A - see above.

Evidence: N/A

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program's measures illustrate progress made against RTL's major goal-- Enhancing readiness to learn of young children. The annual measures for this objective are: (1) The percentage of children aged 3-6 years old who view literacy based RTL shows demonstrating expressive vocabulary and emergent literacy skills. The program has not yet finalized an annual efficiency measure.

Evidence: Data for these measures are now collected using two mechanisms: 1) PBS' "Management Information System," an on-line data collection mechanism that all local affiliates are required to use; 2) a private evaluation firm (Mathematica Policy Research) is currently conducting a multi-year evaluation which includes data collection related to all aspects of program performance.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Ambitious baselines and annual targets have been developed for the program's annual performance measures.

Evidence: N/A

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: While the current RTL grantee, PBS, actively supports the overall annual goals of the program, long-term goals have not yet been established. PBS measures and reports on their performance as it relates to these annual measures. PBS routinely presents analyses on, and discusses the importance of the current performance measures at meetings where PBS affiliate stations are in attendence (e.g., the importance of providing accurate, reliable data is a common discussion topic).

Evidence: PBS affiliate subcontract performance clauses. Program staff are currently working with PBS to develop long-term performance measures. This question will be reconsidered once additional information is available for review as it is expected that PBS will implement the same kinds of accountability processes for the eventual long-term measures.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Several independent evaluations have been conducted to support program improvements and evaluate effectiveness and relevance. For example, PBS contracted with a private research firm, Mathematica, to conduct a 5-year evaluation of RTL activities under the current grant. The final report of this evaluation, published in Fall 2004, contains a thorough analysis of RTL service delivery (based on coordinator surveys and management information system data), program development (based on site visits), outcomes of RTL activities (based on data from parents, children, and RTL coordinators), and professional development and technical assistance needs.

Evidence: "Data Collection of the Federal Performance Indicators for PBS Ready to Learn: Year Four Summary Report." WestEd. August, 2004; "Using Television as a Teaching Tool: The Impacts of Ready to Learn Workshops on Parents, Educators, and the Children in Their Care." Mathematica. June, 2004.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. The program, at this time, does not have sufficiently valid and reliable performance information to assess (whether directly or indirectly) the impact of the Federal investment. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program (including S&E).

Evidence: ED Congressional Budget Justifications.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Program staff routinely review program management activities, and make corrections to eliminate identified deficiencies. For example, program staff are currently working to develop a limited number of long-term performance measures. ED staff also participate in Board Meetings, and work very closely with PBS to ensure that training for local affiliates targets areas where stations need to improve (e.g., reaching the most challenging low-income and migrant populations that typically benefit most from RTL outreach activities).

Evidence: Review of program files illustrates numerous strategic planning corrective actions taken as a result of ED staff working closely with grantees.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: PBS regularly collects performance data relating to key program goals using a web-based data collection mechanism (MIS - Management Information System), and routinely uses such data to adjust program management strategies. Examples of adjustments include developing and implementing specific definitions for key ancillary outreach activities and establishing uniform standards for purposes of parent training activities and adjusting broadcast times to maximize viewing amongst target audiences.

Evidence: ED staff discuss program performance with grantee via regularly scheduled (bi-weekly) conference calls. Most aspects of program performance -- including both program production and ancillary RTL program activities -- are managed using available performance data.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, the Department has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps ' hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its SES staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program successfully obligates funds by the end of each fiscal year. Funds are spent for the intended purposes; this is assessed through grant monitoring. No improper uses of funds have been identified.

Evidence: Funds are alloted, obligated, and drawn-down according to regular, pre-planned schedule. Program budget files.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: To date, the Department has not established procedures for this program to measure and achieve efficiencies in program operations. However, ED is in the process of developing its competitive sourcing Green Plan, and is working to improve the efficiency of its grantmaking activities. The Department has also established a strengthened Investment Review Board to review and approve information technology purchases agency-wide.

Evidence: Department Investment Review Board materials. ED's Discretionary Grants Improvement Team (DiGIT) recommendations.

Does the program collaborate and coordinate effectively with related programs?

Explanation: By statute, the RTL is required to collaborate with other agencies/program staff that have major training components for early childhood development (e.g., HHS, Head Start, Even Start, and State activities funded under the Child Care and Development Block Grant). Local PBS affiliates are also required to develop partnerships to ensure effective delivery of RTL ancillary activities. The existence of these activities have been confirmed by program evaluations.

Evidence: ESEA, Sec. 2431(a)(4)(B); On-going Mathematica evaluations -- see "Building Strong Ready to Learn Outreach," October 30, 2001, and "Implementing Ready to Learn Outreach: Lessons from 20 Public Television Stations," April 1, 2002 -- demonstrate that local partnerhsips (both formal and informal) lead to meaningful changes that can be documented. For example, workshop participants are routinely recruited using community partnerships. Children's books are also routinely distributed through community-based partnerships.

Does the program use strong financial management practices?

Explanation: PBS is subject to routine program audits, and all financial information available is accurate and timely. Also, on two occasions ED staff have submitted records demonstrating the program's use of strong financial management practices to Congress.

Evidence: Congressional Reports (to both House and Senate). Review of audit files.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: There are numerous examples of meaningful steps that have been taken by program staff to address management deficiencies. For example, program staff are currently working to develop long term performance and efficiency measures.

Evidence: Review of program records; meeting notes; annual performance reports; records and recommendations from mid-point reviews.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The single 5-year cooperative agreement supported under this program is awarded on a competitive basis, using a fair and open selection process that includes an independent merit review and a ranking of applications.

Evidence: Review of competition slates and program files.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Program staff demonstrate a strong relationship with the grantees and a high level of understanding of what grantees do with resources allocated to them. This understanding has been reflected in reports detailing grantee use of Federal resources previously cited above.

Evidence: Review of notes from bi-weekly conference calls scheduled with grantee; review of ED staff notes and agendas for all regularly scheduled grantee board meetings, conferences, and other program events.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Information that is available on the performance of RTL is not aggregated and disaggregated in a way that relates to the impact of the program. Education is developing a department-wide approach to improve the way programs provide performance information to the public. In 2004, Education will conduct pilots with selected programs to assess effective and efficient strategies to share meaningful and transparent information.

Evidence: PBS Kids website; ED program performance database.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program does not yet have any long-term performance goals.

Evidence: N/A

Does the program (including program partners) achieve its annual performance goals?

Explanation: Initial data collected on annual performance measures suggest that the program is meeting its annual performance targets.

Evidence: Ready to Learn Performance Measurement Data

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Department has not yet developed an efficiency measure for this program.

Evidence: N/A

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There is no reliable basis for comparing RTL to other programs. No current studies, analyses, or evaluations have attempted to make such comparisons.

Evidence: N/A

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: While past evaluations of RTL activities have generally been positive, the current Mathematica and WestEd RTL studies are not as promising. These studies suggest that the program may not be achieving intended results in key areas of implementation. For example, the Mathematica study concluded that the current workshop approach to outreach has little or no measurable effect on student outcomes and parent/caregiver behaviors.

Evidence: "Data Collection of the Federal Performance Indicators for PBS Ready to Learn: Year Four Summary Report." WestEd. August, 2004; "Using Television as a Teaching Tool: The Impacts of Ready to Learn Workshops on Parents, Educators, and the Children in Their Care." Mathematica. June, 2004.