Detailed Information on the
Audience Development for US International Broadcasting Assessment

The purpose of the BBG Audience Development program is to provide audience development support to the BBG's broadcasting entities in their efforts to promote and sustain freedom and democracy by broadcasting accurate and objective news and information about the United States and the world to audiences overseas. The Audience Development Program is focused on achieving its long-term goal of contributing to the Agency's success in reaching the target audiences. Meeting this outcome goal will be tracked by measuring the growth in BBG audience size annually, as measured by the worldwide, unduplicated number of weekly audience members. BBG's target is to increase the worldwide reach to 160 million by 2012 from 140 million in 2005.

Revising the current agency strategic plan to build on a market-based approach to broadcasting.

A key function of BBG's Audience Development Program relates to carrying out performance reviews of the broadcasting programs. BBG has developed and will continue to track measures to assess this function by focusing on (1) program quality ratings, (2) the percentage of scheduled program reviews completed each year, and (3) the level of compliance with recommendations that result from program reviews. The program quality rating directly relates to the ability to attract large audiences. Both the program review and the recommendations are directly designed to assist broadcasters in reaching the largest audiences, whether through increased technical capacity and signal strength, improved quality, more targeted programming, etc. BBG will continue to monitor these measures to make sure they are on track to meeting the annual targets related to the performance reviews.

Measure: Overall Audience Size for U.S. International Broadcasting (measured in millions of listeners/viewers) Note: Measure assesses effective support for BBG strategic decision-making based on audience and market research and performance reviews as well as effective marketing and advertising, which contribute to the Agency's success in reaching the target audiences, as measured by the overall audience size for BBG broadcasting.

Explanation:A key indicator of whether or not BBG broadcasts are reaching their intended audience is the number of people who choose to listen to or view BBG programs. Audience reach, also called weekly audience, is a measure of the regular listening/viewing audience. Regular audience is defined as all adults in a given population listening/viewing at least once a week, as determined by an audience survey that has an adequately designed sample. The effectiveness of Audience Development activities such as audience research, performance review, and marketing and advertising directly impact the development of BBG broadcast strategies and improve program quality, and thereby support growth in the overall audience size.

Measure: Percentage of VOA language programs that rate "good or better" on the program quality scale. Note: Since 2001, the BBG has improved the percentage from 86.7% to 98-100%. Targets are set to maintain this optimal level and annual monitoring will ensure continued excellence in programming.

Explanation:The program quality score is an outcome measure based in part on how the target population perceives the quality of the BBG language services. The data shown here is specific to all VOA language services and the percentage that are scored as good or better. This measure is an industry standard used to determine the target audience's assessment of the quality of media. Audience Development activities directly impact this measure by ensuring that audience research, performance review, and marketing and advertising effectively support the programming by each language service. A monitoring panel of regular listeners from the target population rates the program's accuracy, reliability, authoritativeness, objectivity, comprehensiveness, and other variables reflecting distinct statutory, policy, and mission mandates for the different stations. Another component of the program quality score is the evaluation of IBB delivery systems (engineering and transmission), marketing and program placement efforts, and other broadcasting support functions. Program content and presentation are evaluated by both internal and external reviewers to determine the overall Program Quality Score. In total, the Program Quality Score is a comprehensive, weighted score, which ranges from 1-4, with 1.0-1.3 = poor; 1.4-1.6 = poor to fair; 1.7-2.3 = fair; 2.4-2.6 = fair to good; 2.7-3.3 = good; 3.4-3.6 = good to excellent; 3.7-4.0 = excellent. Annual measures of the perceived quality of BBG programming are tracked by each language service, and directly support the long-term measures of growing the size of the audience and increasing the overall perceived reliability of the programming.

Measure: Level of Compliance with Program Review Recommendations (% of recommendations issued to Voice of America and Office of Cuba Broadcasting language programs that are implemented).

Explanation:This measure is maintained and verified by the Office of Performance Review, a function of Audience Development, and tracks all action items listed in the formal Action Plan as part of the program review process. Compliance is measured by a panel of reviewers, which includes members from the language service, supporting services (such as the Office of Engineering, VOA Central News, Office of Marketing and Program Placement) and managers. The follow-up Action Plan Review meetings are generally held three months after the final Action Plan is approved and during the meeting all items are discussed and the compliance results determined.

Measure: Verified Compliance from Broadcast Stations that Receive Cash Compensation (%of broadcast stations receiving cash compensation that are complying with BBG's broadcasting objectives). As media markets become more competitive, broadcasters learn the value of their air time, especially for stations that attract large audiences also targeted by the BBG. In order to place programming on these stations at desirable times, the BBG must often pay the market rate for air time. Some stations, in lieu of cash payments, are willing to accept payment in equipment (e.g., transmitters) and/or technical, managerial or journalistic training. All compensation is stipulated as dependent on adherence to BBG 's broadcasting objectives. Note: the measure can fluctuate from year to year as new affiliates are added and then monitored for compliance, resulting in termination actions if warranted.

Explanation:BBG broadcasting has increasingly migrated from the shortwave bands to local and regional stations, requiring a fundamental shift in placement and promotion strategies. BBG programming must now compete in a marketplace that includes innovative and diverse international and private local broadcasters. In this media environment, the on-going challenge is to marry the BBG's mission to targeted markets, developing and implementing well-researched and effective marketing strategies, program placements, and affiliate support plans for the distribution of U.S. international broadcasting. This measure monitors and ensures that affiliates are documented and verified as complying with BBG's broadcasting objectives. The measure is tracked and monitored by the Office of Marketing and Program Placement, a function of Audience Development, by a number of methods, including over-the-air monitoring by OMPP staff and other BBG and US Government Employees; and by remote monitoring via the Internet and telephone.

Measure: Response Time within 48 Hours to Affiliate Concerns (% of affiliate concerns addressed within 48 hours) Note: Since the baseline in 2005, the percentage of responses within 48 hours has risen from 80% to over 90%, with annual fluctuations primarily due to the number of remotely located affiliates with less advanced communication methods, which can cause delays.

Explanation:BBG broadcasting has increasingly migrated from the shortwave bands to local and regional stations, requiring a fundamental shift in placement and promotion strategies. BBG programming must now compete in a marketplace that includes innovative and diverse international and private local broadcasters, as well as local government controlled media and the uncertainly of local government restrictions against affiliations. In this media environment, the on-going challenge is to maintain affiliate relationships by providing effective support to affiliate concerns. This measure ensures that affiliate concerns are documented and response times are verified. Concerns can include such issues as responding when a lightening storm destroys satellite receiving equipment granted to stations as part of their broadcasting agreement, knocking programming off the air. Replacement gear needs to be quickly ordered, and when necessary, installation assistance will be provided by the BBG. The measure is tracked and monitored in the Office of Marketing and Program Placement by the Operations Director, a function of Audience Development.

Measure: Program Cost per Audience Member (total program cost divided by total number of listeners)

Explanation:This efficiency measure is the total annual cost for audience development, including 1) Audience research, 2) Performance review, and 3) Marketing and advertising divided by the unduplicated BBG audience for all media. The costs for these functions facilitate the development of research-based broadcast strategies, improvement in programming quality, and competitive audience awareness and reach in targeted markets. The ratio (measure) must be evaluated in context of the changes in either the total cost or audience size. For example, a short-term cost increase to improve research methodology or increase advertising in a critical market may not increase audience size for several years. In some cases, audience size may be kept artificially low by restrictions imposed by the host government (such as forbidding affiliations with foreign broadcasters) or increased competition, not necessarily inefficient management.

Is the program purpose clear?

Explanation: The purpose of this program is to provide audience development support to the Broadcasting Board of Governors (BBG) broadcasting entities in their efforts to promote and sustain freedom and democracy by broadcasting accurate and objective news and information about the United States and the world to audiences overseas. The primary functions of the program are 1) Audience research, 2) Performance review, and 3) Marketing and advertising. These functions facilitate the development of research-based broadcast strategies, improvement in programming quality, and competitive audience awareness and reach in targeted markets. All BBG entities, including the grantees, are involved in the Audience Development Program by either direct funding and responsibility for certain functions or participating in the coordination of centralized functions.

Evidence: ?? BBG Strategic Plan (2002-2007), updated BBG Strategic Plan (2008-2013) ?? U.S. International Broadcasting Act of 1994 ?? Foreign Affairs Reform and Restructuring Act of 1998

Does the program address a specific and existing problem, interest, or need?

Explanation: In order to carry out its mission of broadcasting accurate and objective news and information about the United States and the world to audiences overseas the BBG broadcasting entities must have market research to tailor the programming and broadcasting to reach the largest audience in the preferred media. The BBG's strategic plan to "marry the mission to the market" depends on extensive research to understand our markets in an increasingly complex media environment. Performance review analyzes the research data and evaluates the programming in terms of content and production to support the broadcasters in reaching the largest audience in target areas while adhering to high journalistic standards. The functions of marketing and advertising address the need to broaden the reach of BBG programming through audience awareness and promotion.

Evidence: ?? BBG Strategic Plan (2002-2007), updated BBG Strategic Plan (2008-2013) ?? U.S. International Broadcasting Act of 1994; ?? Foreign Affairs Reform and Restructuring Act of 199

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Regarding duplication of internal processes, BBG Audience Development includes the research and performance review offices of all broadcasting entities under the purview of the Broadcasting Board of Governors, including the Voice of America (VOA), Office of Cuba Broadcasting (OCB), Radio Free Asia (RFA), and Radio Free Europe/Radio Liberty (RFE/RL) and Middle East Broadcasting Networks (MBN). The centralized audience research is conducted by a single independent contractor (InterMedia) and the results are shared by all entities. This same research is used extensively by the performance review offices. The Office of Marketing and Program Placement (OMPP) carries out marketing and placement activities on behalf of all entities, based on BBG strategic priorities and market analysis. There is no evidence of another federal, state, or local effort that performs a similar function. Regarding duplication of private efforts, the research function is a commercial function, and is performed by an independent contractor. The BBG is a member of the Confederation of International Broadcasting Audience Research (CIBAR) and adheres to industry standards while fulfilling the unique worldwide requirements of the U.S. international broadcasting mission. The performance review function uses the research from the independent contractor to examine the effectiveness of audience reach and programming quality under common criteria and to develop work-plans to improve programming quality and impact, which is most effectively performed in-house. The marketing and advertising function is also a commercial function, but there are components of this function that are most effectively performed in-house as well because this function involves negotiating agreements and contracts on behalf of the agency and liaising with U.S. embassies overseas. The BBG does not have a methodology for comparing the cost or performance of the marketing and advertising function with the private sector (see questions 2.3 and 4.4). This PART recommends that the BBG research and develop a measure that would benchmark this function to the private sector and/or consider performing a streamlined or standard competition for this function or sub-functions within this function in order to determine the optimum level of efficiency and whether their costs are competitive with the private sector. This suggested improvement addresses a small component of this question and does not warrant a "no" answer for this question.

Evidence: ?? The U.S. International Broadcasting Act of 1994 and the Foreign Affairs Reform and Restructuring Act of 1998 state that U.S. international broadcasting should not duplicate the activities of private United States broadcasters or the activities of government-supported broadcasting entities of other democratic nations. ?? U.S. International Broadcasting Act of 1994; ?? Foreign Affairs Reform and Restructuring Act of 1998

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence that a different approach or mechanism would be more effective or efficient at meeting the programs goals. BBG Audience Development provides market research, performance reviews, and marketing and advertising to support broadcasting in all entities to reach audiences around the world. For the research function, the BBG has concluded that a single, consolidated research contractor for all broadcasting entities is better able to coordinate audience research and to deliver the required research in a consistent format, and while doing so, realize economies of scale. The performance review offices are staffed with experts in the field of programming content and production who do not work for the broadcasting offices, but can evaluate and provide recommendations for improvement as well as monitor the quality of programming over time. Marketing and advertising functions are centralized for efficiency in meeting strategic priorities such as building audience in target areas. Advertising and promotion campaigns are frequently outsourced under the management of OMPP to provide key design and development activities.

Evidence: ?? U.S. International Broadcasting Act of 1994; ?? Foreign Affairs Reform and Restructuring Act of 1998

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Audience Development reaches two groups of intended beneficiaries. The first is the assortment of internal broadcasting organizations; the second is composed of worldwide listeners. BBG beneficiaries: Through the centralized research contract all entities and broadcasters can be assured of current, reliable and statistically valid market and audience research on a periodic basis. The frequency of market and audience research is determined by a schedule coordinated with all entities, including the grantees, to meet their needs and BBG priorities. The performance review functions assist the broadcasters in examining all aspects of their programs on an annual basis- bringing together the research results, content and production evaluation and support functions, such as central news and transmission capabilities (Engineering), to develop a comprehensive analysis and roadmap for improvement through an action plan. The Marketing and Advertising functions assist all entities by promoting programming in a variety of ways to reach the largest audiences. Listeners: Audience Development functions work closely with the language services to provide the content the audience deems credible and reliable while producing programming that attracts the targeted audience. They lead, provide the research and/or participate in each of the language service's annual Program Review and the BBG's Language Service Review to inform decisions in carrying out the broadcasting mission.

Evidence: ?? U.S. International Broadcasting Act of 1994 ?? Foreign Affairs Reform and Restructuring Act of 1998 ?? Program Review ?? Language Service Review

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The purpose of the Broadcasting Board of Governors (BBG) Audience Development program is to provide audience development support to the BBG's broadcasting entities in their efforts to promote and sustain freedom and democracy by broadcasting accurate and objective news and information about the United States and the world to audiences overseas. Therefore, the long- term goal for the Audience Development Program is to contribute to the Agency's success in reaching the target audiences. Meeting this outcome goal will be tracked by the growth in BBG audience size, as measured by the worldwide, unduplicated number of weekly audience members. The long-term goal is to increase the worldwide reach to 160 million by 2010 from 140 million in 2005.

Evidence: ?? BBG Strategic Plan ?? BBG Audience Development FY 2008 Performance Plan

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The long-term target for increasing the audience reach is sufficiently ambitious since they entail successful growth in audience size compared with actual resources available to the program. Compared to previous time periods, the target set for 2010 is sufficiently ambitious.

Evidence: In FY04 and FY05, the audience reach was 130 million and 140 million. Estimates for FY06 (150,000 million) and FY07 (156,000) show slower but continued positive growth. The surge in audience size in 2004 and 2005 is attributed to budget initiatives that enabled new programming in new markets, which may not continue at the same pace due to constrained resources and/or more competitive markets. The percent increase for each year (compared to the previous year) is as follows: FY05 (8%); FY06 (7%); FY07 (4%); FY10, compared to FY04 (23%). Thus the estimate for meeting the 2010 target is sufficiently ambitious for the BBG.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: There are three components of measuring annual progress in achieving the long-term goals of the Audience Development Program: 1) all of the broadcasting entities share the long-term goal to build audience size in their target regions. In this way, when the entire agency undergoes the annual planning exercise with the goal of meeting targets related to building audience size, which includes support from research, performance review plus marketing and advertising, this program's progress toward achieving its long terms goals is reinforced. 2) The Program has separate work-plans with detailed and measurable annual action steps over several years for each of the three functions (Research; Performance Review; and Marketing and Advertising). The action steps are organized by performance objective and also have annual performance goals and targets, both numerical (discussed below) and anecdotal. 3) Each performance objective also has key measures for each function (seven total, including the efficiency measure, described in question 3.4) that support the long-term goals. These objectives and measures will be monitored each year as they contribute to both the long-term goal of audience reach and the overall efficiency of the program.

Evidence: The measures for the performance review function are the program quality ratings by broadcasting organization, the percentage of scheduled program reviews completed each year, and the level of compliance with recommendations that result from program reviews. The program quality rating directly relates to the ability to attract large audiences. Both the program review and the recommendations are directly designed to assist broadcasters in reaching the largest audiences, whether through increased technical capacity and signal strength, improved quality, more targeted programming, etc. The measures for marketing and advertising include percent of program reviews in which marketing and advertising participate, percent verified compliance from the broadcast stations that receive cash compensation, and the response time within 48 hours to affiliate concerns. Since marketing and advertising also contribute to the audience awareness and ability of the broadcasting programs to be seen or heard by target audiences, their participation in the annual review will support the broadcasters' ability to reach target audiences. The compliance from the broadcast stations ensures that the BBG's requirements regarding transmission of broadcasting programs are met, which contributes to audiences receiving the broadcast signal. This function's response time to affiliate concerns measures how this program maintains good relations with affiliates and therefore maintains a viable distribution method for broadcasting programs. The BBG does not currently have a measure or methodology for comparing the cost or performance of the marketing and advertising function with the private sector (see questions 1.3 and 4.4). This PART recommends that the BBG research and develop a measure that would benchmark this function to the private sector. This suggested improvement does not warrant a "no" answer for this question. Each program tracks progress toward the efficiency goal of program cost per audience member. As explained in question 3.4, this measure tracks growth in program cost relative to the growth in audience size. As this program aims to be more successful in growing the overall audience size, the program also aims to increase its efficiency by growing audience size at a higher rate than program cost.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The annual targets for BBG's annual measures are sufficiently ambitious given the increasingly complex media environment and the task of providing the BBG broadcasting entities market research to tailor their programming and broadcasting to reach the largest audience in the preferred media and in order for the BBG to carry out its mission of broadcasting accurate and objective news and information about the United States and the world to audiences overseas.

Evidence: ?? Program Quality for VOA - The baseline for this measure is 98% for the Voice of America broadcasting programs in FY05. The targets for FY06 (100%), FY07 (100%), and FY08 (100%) are sufficiently ambitious because they entail every program meeting the goal to be "good" or better each year. ?? Scheduled Program Reviews Completed - The baseline for this measure is 100%, and the goal is to maintain 100% each year. The targets are sufficiently ambitious because the actual number could vary as a result of logistical issues, for which this program has the primary responsibility. ?? Level of Compliance with Program Review Recommendations - The baseline for this measure is 81% for the Voice of America and Office of Cuba broadcasting programs in FY05. The targets for FY06 (83%), FY07 (85%), and FY08 (87%) are sufficiently ambitious because the program is developing new methods to ensure compliance with recommendations, and many of the recommended actions require budgetary or personnel resources that have different priorities within the overall agency and may not be implemented due to constraints in these areas from year to year. For this measure, the program is developing a baseline for FY06 and out year targets for other broadcasting organizations in addition to the VOA. This information will not be ready by the time of PART publication in 2006, and should be entered as part of the Spring update in 2007. ?? Percent of Program Reviews Attended by Marketing and Advertising - The baseline for this measure is 75% in FY05. The targets for FY06 (78%), FY07 (80%), and FY08 (83%) are sufficiently ambitious because the program does not have direct interventions for several language programs, and therefore does not need to attend all language reviews. ?? Verified Compliance from Broadcast Stations that Receive Cash Compensation - The baseline for this measure is 83% in FY05. The targets for FY06 (86%), FY07 (88%), and FY08 (90%) are sufficiently ambitious because the program is developing new methods to ensure compliance with recommendations. ?? Response Time within 48 Hours to Affiliate Concerns - The baseline for this measure is 80% in FY05. The targets for FY06 (82%), FY07 (83%), and FY08 (85%) are sufficiently ambitious because the the affiliates are spread out throughout the world, and communications may be difficult. The program is also developing new methods to follow up with affiliate concerns. ?? Program Cost per Audience Member -- The baseline for this measure is $0.17 in FY05. The targets for FY06 ($0.15), FY07 ($0.16), and FY08 (TBD) are sufficiently ambitious because the program is aiming to grow the audience (the denominator) while keeping program costs low. This will ensure that the program is becoming more efficient, while achieving better results. Each function of this program tracks their costs relative to audience size.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All offices of the Audience Development Program contributed to establishing the Strategic Plan and in the development of the annual performance plan, which include the annual and long-term goals of the program. Senior managers within the program commit to and work towards the goals through the BBG performance based appraisal system, which is specifically tied to program objectives and goals. Each year, managers review and include the most current objectives and goals in their annual appraisals. Contracts for the program are managed by contract officers who monitor compliance with performance standard clauses that impact and support the goals of the program, including schedule, cost and technical requirements. The audience research contractor in particular has the responsibility to provide research showing how BBG programming is reaching target audiences or not. In this way, the research performed by this contractor contributes directly to the attainment of the program's long-term goal of increasing audience reach. Since the contractor provides an objective review of audience size, it would be a conflict of interest to require the contractor to commit to the goals of the program. The Audience Development participation and contribution, by providing research data and analysis, in all language service Program Reviews and the annual Language Service Reviews also enforce commitment to the annual and long-term goals of the agency, thereby supporting the broadcasting mission of the BBG.

Evidence: ?? BBG Strategic Plan ?? Language Service Review ?? Program Review ?? Performance Based Employee Standards/Appraisals

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Internal: The Office of Marketing and Program Placement (OMPP) is included in the annual Language Service Reviews and Program Reviews conducted by the IBB's Office of Program Review, and RFA's and RFE/RL's program review offices. The annual reviews include an initial assessment of how this program is meeting its objectives and regularly scheduled follow-up meetings to check the progress on recommendations generated by the review. The reviews assess the effectiveness of marketing and advertising campaigns managed by OMPP. External: The International Audience Research Project contract is awarded and managed in full compliance with Federal Acquisitions Regulations, which requires that contracts be reviewed on a periodic basis. In the case of InterMedia, the contract is reviewed on an annual basis and the project schedule is established with the coordination of all entities. Government: The State Department's Office of the Inspector General conducted a review of the International Broadcasting Bureau's Office of Performance Review in May 2006. The review provided an evaluation of the effectiveness of this program and made recommendations to support program improvements. (See question 4.5 for further details.)

Evidence: ?? Language Service Review ?? Program Review ?? Federal Acquisitions Regulations 52.211-11, 52.211-12, 11.503 ?? OIG: Report of Inspection, The International Broadcasting Bureau's Office of Performance Review (ISP-IB-06-37)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: BBG has excelled at clarifying its mission, establishing implementation strategies and performance objectives that support the mission, as well as establishing performance indicators. Actions taken in 2006 and 2007 have enhanced the linkages between these strategic tools and increasingly shown how they are supported and implemented by BBG's budget each year. In its Budget requests the BBG explicitly linked all costs to six annual performance objectives that support BBG ' s mission, and will do so again in its FY 2008 budget request. In addition, each objective will link annual performance goals to the requested resources with performance measures will be monitored against the targets set for FY 2008. In FY 2008, the BBG will show how the performance objectives for the various broadcasting entities and support elements support the overall BBG strategic goal and annual performance goals. In FY 2008, the will again include the agency's annual performance plan and performance measures in budget requests. The IARP cost is based on matrix of research projects (qualitative and quantitative) for each of the BBG language services. This matrix is reviewed each year by the entity research directors to ensure that the projects planned for that fiscal year will meet the needs of the broadcast services and the program review process. The research directors meet regularly to coordinate the planning for the research projects. The Performance Review function uses the research project matrix to establish schedules for reviews with stable costs. Also, each research director has a small amount of funds available each year to conduct unique research projects that will meet the needs of the individual broadcast entity. Similarly, the Marketing and Advertising functions are tied to base resources plus targeted BBG high priority or new programming requirements.

Evidence: ?? FY 2006 Congressional Justification ?? Language Service Review ?? Program Review ?? FY 2006, FY 2007 and FY 2008 Budget Requests to OMB

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The BBG has a performance plan that reflects the functions of the Audience Development program. The performance plan is linked to the BBG Strategic Plan, and it ties individual program goals to the strategic goals of the agency. The BBG is developing a new strategic plan to update the current strategy that covers 2002-2007. The plan will reflect the experiences gained over the past several years and be a forward looking roadmap to achieve the goals of U.S. public diplomacy in reaching significant audiences in support of U.S. strategic interests On an ongoing basis, the research function ensures that all audience research is aligned with the strategic broadcasting priorities of the BBG and that the findings are useful to both the broadcasters and in evaluating the effectiveness of the BBG. As BBG entities respond to increasingly complex media environments worldwide, the research function expanded its scope beyond traditional broadcast media to support BBG's transformation into a multi-media broadcast service. Research also informs the program review and language service processes to evaluate individual language services' adherence to BBG priorities. The performance review function provides management with annual action plans that are linked to strategic goals, set goals and deadlines for improvements in news and program content, marketing activities, delivery system performance and overall management support. The marketing and advertising function is meeting the BBG's challenge of "Marrying the Mission to the Market" by developing and implementing well researched and effective annual and ongoing marketing and advertising strategies to meet key objectives.

Evidence: ?? BBG Strategic Plan ?? BBG Audience Development FY 2008 Performance Plan ?? FY 2006, FY 2007 and FY 2008 Budget Requests to OMB

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The BBG analyzes the performance data provided by its research contract throughout the year. As explained in question 1.5, audience research for each language program is provided to the BBG according to the contract schedule coordinated among the entities, which provides very credible and valuable performance information for this program, broadcasters, engineers, and the agency as a whole. Marketing and advertising performance is assessed through the annual Language Service Review and Program Review processes. Research provided on audience size, as explained above, also provides performance information about marketing and advertising initiatives. Annual Program Reviews include an initial assessment of each program's ability to increase audience awareness of BBG programming, with regularly scheduled follow-up meetings to check the progress on recommendations generated by the review. Recommendations may include management or performance issues and improvements. The Board uses this information to adjust resources and set strategic priorities.

Evidence: ?? Language Service Review ?? Program Review

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Senior managers, including the Senior Executive Service (SES) and other senior staff (GS 13-15), are under the BBG performance based appraisal system that requires annual "critical elements" and ratings based on the mission and performance objectives of the BBG. Specific goals with measurable targets are set for each element within the appraisal. The performance based appraisal system also includes individual accountability for cost and schedule results as appropriate. Through the ongoing appraisal reviews and final annual rating, all mangers and senior staff are held accountable for performance results. The annual Language Service Review holds managers and partners directly accountable for results. The impact of marketing and advertising services in achieving individual language service goals and objectives are reviewed and evaluated throughout the process. In addition to the individual employee appraisal system described above, the CFO's mid-year reviews of budget and performance results hold managers accountable for their annual resource level against expected expenditures, schedules and accomplishments. The International Audience Research Project (IARP) assures contractor compliance with cost, schedule and performance requirements dictated in the contract. Contracts include "negative incentives" in accordance with the Federal Acquisitions Regulations to discourage changes from the time or performance indicated in the scope of contract.

Evidence: ?? BBG Performance- based Employee Appraisals ?? Language Service Review

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: All requisitions must include verification that funds are available, and the funds are obligated when a contract or agreement is signed or a purchase order is done. The mid-year, 3rd quarter and year-end review provides an opportunity for managers to verify that unliquidated obligations are valid and that once an obligation has been fully expended, remaining balances are deobligated. Financial transactions are coded to both an organization and to a function or project. Given that BBG has a high percentage of fixed costs, funds must be obligated in a timely manner in order to continue operations. The BBG requires an annual financial plan from the IBB and the grantees, including Radio Free Asia and Radio Free Europe/Radio Liberty. The financial plans must detail anticipated costs for specific line items by each month of the fiscal year. BBG conducts a mid-year, 3rd quarter, and year-end review of each entity's financial plans and budgets to monitor funding and expenditures. BBG also requires a monthly status report of expenditures, disbursements and obligations, as well of the submission of the SF-169 Financial Status Report, and SF-172 Report of Federal Cash Transactions from the grantees. The reviews and the allotment control procedures enable the Agency to track obligations to ensure that funds are used for the intended purpose by the appropriate Agency office.

Evidence: ?? SF-169 Financial Status Report ?? SF-172 Report of Federal Cash Transactions ?? Monthly status reports on the IBB, RFA and RFE/RL on expenditures, disbursements, and obligations ?? Quarterly review reports

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The BBG uses the measure of BBG Audience Development Program Cost per Audience Member to evaluate the efficiency of the program. The efficiency indicator (program cost per audience member) is used to track improvements in efficiency over time. The program has separate performance plans for each of the three functions (Research; Performance Review; and Marketing and Advertising) with distinct performance objectives and measures. The program is expected to maintain or improve the achievement of all performance indicators at a relatively constant resource level. The program also achieves efficiencies by using incentives in its contracts, for example using competitive sourcing to assure cost effectiveness. These projects are managed by the BBG Audience Development program through the International Audience Research Project (IARP) to assure compliance with the cost, schedule and performance dictated in the contract. Contracts include "negative incentives" in accordance with the Federal Acquisitions Regulations to discourage changes from the time or performance indicated in the scope of contract.

Evidence: ?? BBG Audience Development FY 2008 Performance Plan ?? Language Service Review process ?? Program Review process ?? Federal Acquisitions Regulations 52.211-11, 52.211-12, 11.503

Does the program collaborate and coordinate effectively with related programs?

Explanation: The audience research offices coordinate with the broadcasting entities in setting schedules that provide the data in time for their annual Performance Reviews, and provide specific research for special projects. Program reviews require a collaborative process involving programs from across the agency to examine the effectiveness of one language service, and the Office of Performance Review manages the entire process, including scheduling, tracking the action plan and providing much of the analysis. The Office of Marketing and Program Placement works with all entities and the annual Language Service Review process to set priorities for marketing and advertising campaigns to raise audience awareness of BBG broadcasting.

Evidence: ?? U.S. International Broadcasting Act of 1994 ?? Foreign Affairs Reform and Restructuring Act of 1998

Does the program use strong financial management practices?

Explanation: The BBG is cross-serviced by the State Department to provide financial services using the Broadcasting Financial Management System (BFMS) system. Internal controls are in place to provide for the separation of duties, minimize erroneous payments, ensure recording of receipts, and the accurate reporting to external entities. The BBG Administrative Officers obligate financial transactions accurately and reconcile their allotments on a monthly basis. In the CFO Office, the Financial Operations, records and makes disbursements, reconciles the Fund Balances with Treasury, records and makes deposits, and prepares the monthly, quarterly and annual statement reports for the agency. FY2004 was the first year the BBG prepared the Performance and Accountability Report (PAR) and conducted a financial statement audit. The PAR was submitted on time and the BBG received an unqualified opinion on its Balance Sheet. Again in 2005, the BBG submitted an expanded PAR and received an unqualified opinion.

Evidence: ?? BFMS System ?? An independent audit conducted in October 2004 and October 2005 by the OIG offered an unqualified opinion of the BBG financial statements ?? The State Department has also received clean audit opinions

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The BBG consolidated its research function into a centralized contract in the 1990s, which has enabled the BBG to coordinate and oversee a priority schedule of research projects and to monitor the individual research projects to ensure that they adhere to high professional standards and utilize the latest and most appropriate research methodologies. The Performance Review function has benefited from the consolidated research contract to improve timeliness of the research for scheduled program reviews and ensure consistent methodologies are used. The marketing and advertising functions have made a recent change in strategic alignment and are now managed within the framework of the BBG strategic plan and targeted priorities. Between September 2000 and February 2004 the GAO conducted three evaluations of the BBG. While none of the reports focused specifically on BBG audience development, the findings, recommendations and resulting actions by the BBG impacted the entire agency, including this program. Follow-up testimony provided in 2004 shows that the Board has taken meaningful steps to address its management deficiencies. No other current evaluations regarding the BBG have been completed.

Evidence: ?? Program Reviews ?? BBG Performance Based Appraisal System ?? GAO Report: "Strategic Planning and Performance Management System Could be Improved" (GAO/NSIAD-00-222) ?? GAO Report: "New Strategic Approach Focuses on Reaching Large Audiences but Lacks Measurable Program Objectives" (GAO-03-772) ?? GAO Testimony: "Challenges Facing the Broadcasting Board of Governors" (GAO-04-711T)

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: As explained in question 2.2, in FY04 and FY05, the audience reach was 130 million and 140 million. Estimates for FY06 (150 million) and the targets for FY07 (156 million) and FY08 (TBD) demonstrate adequate progress in achieving the long-term goal of 160 million by 2010.

Evidence: ?? BBG Audience Development FY 2008 Performance Plan

Does the program (including program partners) achieve its annual performance goals?

Explanation: Four out of the seven annual measures are new to this program and do not have data or show results beyond the baseline 2005 data. There is no reason to believe that the program will not meet its 2006 goals, but these data are necessary to show that the program meets its annual goals and to warrant a "yes" for this question. Three measures do have data from before 2005, which provides clear evidence that the program is meeting these goals. For this reason, a Large Extent is warranted.

Evidence: Annual measures (as shown in the Audience Development FY 2008 Performance Plan) ?? VOA Program Quality - The baseline for this measures is 98% in 2005, and data available in 2003 (98%), 2004 (100%), and 2005 (98%) indicate that the program is on track to reach 100% in 2006, 2007, and 2008. ?? Scheduled Program Reviews Completed - The baseline for this measures is 100% in 2005, and data available in 2003 (100%), 2004 (100%), and 2005 (100%) indicate that the program is on track to reach 100% in 2006, 2007, and 2008. ?? Level of Compliance with Program Review Recommendations - The baseline for this measure is 81% for the Voice of America and Office of Cuba broadcasting programs in FY05. The program plans to meet the goal for FY06 (83%), which will put it on track to meet its FY08 goal. ?? Percent of Program Reviews Attended by Marketing and Advertising - The baseline for this measure is 75% in FY05. The program plans to meet the goal for FY06 (78%), which will put it on track to meet its FY08 goal. ?? Verified Compliance from Broadcast Stations that Receive Cash Compensation - The baseline for this measure is 83% in FY05. The program plans to meet the goal for FY06 (86%), which will put it on track to meet its FY08 goal. ?? Response Time within 48 Hours to Affiliate Concerns - The baseline for this measure is 80% in FY05. The program plans to meet the goal for FY06 (82%), which will put it on track to meet its FY08 goal. ?? Program Cost per Audience Member -- The baseline for this measure is $0.07 in FY05. The program plans to meet the goal for FY06 ($0.06) (see question 4.3).

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: There sufficient demonstrated evidence Audience Development achieves improved efficiencies or cost effectiveness in achieving program goals each year. The BBG uses the measure of BBG Audience Development Cost per Audience Member to evaluate the efficiency of the program. In FY03, FY04 and FY05, the actual program cost per audience member was $0.25, $0.21 and $0.17 respectively and the goal for FY06 is $0.15, which shows increased efficiency each year. These improvements have been gained through integration of central research functions across all entities, aligning resources with strategic priorities as determined by participation in the program review process, and providing audience development support in reaching a larger audience.

Evidence: ?? BBG Programming Support FY 2008 Performance Plan ?? Language Service Review ?? Program Review ?? IARP consolidated research contract

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no government programs that share similar goals and objectives with which the performance of this program can be compared. With regard to the private sector, the International Audience Research Project is a comprehensive, worldwide research management contract that is broken into smaller sub-contracts where necessary. All research adheres to market research industry standards, which indicates that the program compares favorably with the private sector. The performance review function is not duplicated in the private sector, as explained in question 1.3. The marketing and advertising function is a commercial function, and since the BBG has no basis to compare performance of this function to the private sector (as explained in question 1.3), this warrants a "large extent" answer for this question. This PART recommends that the BBG research and develop a measure that would benchmark this function to the private sector and/or consider performing a streamlined or standard competition for this function or sub-functions within this function in order to determine the optimum level of efficiency and whether their costs are competitive with the private sector.

Evidence: ?? U.S. International Broadcasting Act of 1994 ?? Foreign Affairs Reform and Restructuring Act of 1998

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In FY06 the Office of Inspector General conducted a review of the Office of Performance Review. The evaluation indicated that the program is effective and achieving results. The report notes that Performance Review provides regular quality controls for the agency through its annual reviews, conducts spot checks of certain broadcasts in support of BBG's mission to provide quality programming, and recommends program improvements. The OIG takes note of the efforts and successes of Research as they provide new avenues for increasing listeners and viewers in an ever increasingly competitive and complex multi-media environment. Annual Program Reviews and Language Service Reviews indicate that Marketing and Advertising functions are effective in supporting the broadcasting entities and is achieving results. Last, in FY2005 and FY 2006, the Congress has noted in the report to accompany annual appropriations that the annual Language Service Review process, supported by the Research and Performance Review functions, states "The Committee continues to support the Board's efforts to objectively and systematically review and evaluate the performance, results, and priority of every U.S. Government-sponsored international broadcasting language service and to propose corresponding reallocations of funds. The Committee endorses this process as a means to improve broadcast quality and meet emerging program priorities within limited resources."

Evidence: ?? Program Review schedules, results and action plans ?? Language Service Review analysis and results ?? InterMedia research including surveys, monitoring panels, and listener panels ?? OIG: Report of Inspection, The International Broadcasting Bureau's Office of Performance Review (ISP-IB-06-37) ?? The Conference report to accompany H.R. 4754 (108-576) containing the BBG's appropriation for Fiscal Year 2005