Detailed Information on the
Youth Anti-Drug Media Campaign Assessment

Contract awarded for new evaluation.

Consolidation of two youth web sites into a single youth web site.

Improved message testing by using online technologies based on best practices of the advertising industry, thus reducing travel costs, improving timeliness, and improving overall efficiency.

Evaluating how much funding the Campaign needs to accomplish its goal of reducing youth drug use.

Targeting the Campaign for maximum impact on youth and their parents.

Reviewing evaluation results and considering alternative evaluation strategies to demonstrate the impact of the Campaign.

The Campaign will work with other federal agencies to conduct analyses to identify the most efficient and effective means to improve the implementation of the FY08 anti-meth campaign

The Campaign will develop strategies and methods to effectively extend the reach of the Campaign messages and use of materials and resources among national and local organizations, for little or no cost to the FY08 prescription drug abuse campaign.

The Campaign, over a 12-month period, will perform website user satisfaction evaluations and make improvements on the program's four primary Websites.

Measure: Percent of youth ages 12-18 who believe there is great risk of harm from regular marijuana use.

Explanation:The data source for this measure is the Partnership Attitude Tracking Study (PATS), whose data are not available until Spring, 2007.

Measure: Percentage of teens 14-16 who think they will definitely not smoke marijuana at any time during the next year (New measure, added February 2008)

Explanation:To counter youth perception that marijuana use is a normal and socially-accepted behavior, the Campaign crafts messages dispelling these myths as well as debunking misinformation that drug use is harmless. This measure on teen intentions to use marijuana within the next year, tracks the Campaigns ability to communicate this anti-drug message platform effectively as well as any change in youth normative beliefs about marijuana use.

Measure: Percentage of teens, 14-16 who think they would definitely not smoke marijuana if offered by a close friend (New measure, added February 2008)

Explanation:To counter youth perception that marijuana use is a normal and socially-accepted behavior, the Campaign crafts messages dispelling these myths as well as debunking misinformation that drug use is harmless. This measure on teen intentions to use marijuana if offered by a friend, tracks the Campaigns ability to communicate this anti-drug message platform.

Is the program purpose clear?

Explanation: The "Drug Free Media Campaign Act of 1998" directs ONDCP to conduct a national media campaign to reduce and prevent drug abuse among young people in the United States.

Evidence: Authorizing statute "Drug Free Media Campaign Act of 1998." (21 USC 1801 et. seq.)

Does the program address a specific interest, problem or need?

Explanation: Current rate of drug use among youth (12-17) is unacceptably high (11.6%) and has increased since the early 1990s. Stopping drug use before it starts is one of three priorities in the President's National Drug Control Strategy. Research has established the efficacy of using public service announcements (PSAs) in public health campaigns, e.g., smoking cessation and seat belt use.

Evidence: National Household Survey on Drug Use and Health (SAMHSA)

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The National Youth Antidrug Media Campaign (NYADMC) accounts for the majority of current antidrug advertising in the traditional mass media and other less traditional venues. The Campaign's FY 2003 budget includes $121M for purchase of media time and space, and that amount must be matched by the media with contributions of advertising time/space or related in-kind services. Prior to the inception of the Media Campaign in 1998, the value of antidrug (PSAs) funded through private sources had declined from approximately $360M in 1990 and 1996 to approximately $260M.

Evidence: Media Campaign Operating Plan, Partnership for Drug Free America (PDFA)

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: The Media Campaign is the principal source for consistent reinforcing messages across a variety of media and for messages aimed at specific target audiences that may be difficult to persuade with standard messages. In addition, since the Campaign purchases time and space, it is able to place ads in broadcast prime time and in desirable locations. Ads provided by broadcasters and others as public service announcements are often shown during periods when there are few viewers/listeners.

Evidence: See number three.

Is the program optimally designed to address the interest, problem or need?

Explanation: The Campaign acquires time/space by contracting with established advertising company. Content of ads is generally provided pro bono through arrangement with not-for-profit organization. ONDCP lacks staff with the specialized skills to perform these tasks.

Evidence: No contrary indications.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: In response to the 2002 OMB PART review, ONDCP revised the Campaign's logic model and significantly changed its long-term and annual performance measures. These new measures were included in ONDCP's FY 2004 and FY 2005 GPRA plans.

Evidence: ONDCP Strategic Plan, FY 2004 Performance Plan, Annual Performance Reports (1999-2001), and Campaign Communications Strategy Statement.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: In response to the 2002 OMB PART review, ONDCP revised the Campaign's logic model and significantly changed its long-term and annual performance measures. These new measures were included in ONDCP's FY 2004 and FY 2005 GPRA plans.

Evidence: Annual Performance Plans and Reports

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: The Media Campaign's two major contracts are "cost plus fixed-fee," rather than performance-based, although the Campaign is currently exploring the possibility of issuing the upcoming re-competition of the non-advertising contract as a performance-based contract. Currently, contractors are held accountable only for meeting process goals and other goals that are not directly related to the outcome measures established by ONDCP.

Evidence: Discussions with ONDCP legal staff.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: ONDCP Campaign staff have established working relationships with related programs (e.g., Drug Free Communities, HHS treatment and research programs, and law enforcement).

Evidence: Media Campaign Operating Plan, discussions with staff from other agencies, and review of web sites.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: A comprehensive long-term independent evaluation is being conducted under a NIDA contract; reports are issued semiannually. The NIDA contract has expired and discussions are underway about the design of a replacement evaluation.

Evidence: Evaluation of the National Youth Anti-Drug Media Campaign (NIDA).

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: ONDCP is working to align its budget with its programs and performance measures.

Evidence: Communications Strategy Statement; ONDCP Strategic Plan and Annual Performance Plans and Reports

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: In May 2002, following the receipt of the semi-annual report detailing the disappointing results of the Campaign, ONDCP: increased testing of ads prior to airing them; targeted ads to a different age group; began involving ONDCP staff more directly in the message development process; and directed the Media Campaign staff to report directly to the ONDCP Director. In FY 2003, ONDCP redesigned its long-term and annual performance measures and goals.

Evidence: Revised Media Campaign Operating Plan

Are acquisition program plans adjusted in response to performance data and changing conditions?

Explanation: After reviewing the results of the most recent NIDA-managed evaluation, which showed that the Campaign was not yet having the desired effect on youth anti-drug attitudes and behaviors, several changes were made to the Campaign, including shifting the emphasis to a different age group, involving ONDCP staff more directly in the production process, focusing more on marijuana, and developing and airing more ads directed towards youth, rather than parents.

Evidence: Media Campaign documents, ONDCP press release

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule and performance goals?

Explanation: No documentation of a comprehensive trade-off analysis is available.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ONDCP has used the semi-annual reports from the NIDA contract to begin significant changes to Campaign operations. In addition, contract staff revise ad copy and adjust media buys in response to feedback about the effectiveness of specific ads.

Evidence: Annual Performance Plans and Reports; Media Campaign Operating Plan

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: These is no apparent linkage of program performance to agency managers or to program contractors.

Evidence: Discussions with ONDCP staff.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Obligations are timely. In 2002, $180M was appropriated and $170M (94%) was obligated within 12 months. In addition, agency reviews led to questioning contractor vouchers for approximately $7.6M. Contractor later settled by reimbursing Government $700,000 and dropping $1.1M in unsubstantiated charges.

Evidence: SF - 132s, SF -133s, Treasury reports. and GAO.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Media Campaign utilizes competitive sourcing to select contractors to complete the advertising and non-advertising aspects of the Campaign. Both of the campaign's major contractors were chosen through a full and open contract solicitation process and awarded contracts based on "best value" (a combination of past performance, cost, and strength of proposal).

Evidence: GPRA documents

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: Neither the Media Campaign Operating Plan nor ONDCP Budget submissions identify all direct and indirect costs for the program. However, these personnel and overhead costs are so small (approximately $1M) that they have no significant or measurable effect on the overall program budget.

Evidence: FY 2002 Media Campaign Operating Plan and ONDCP budget requests

Does the program use strong financial management practices?

Explanation: Questionable labor cost billings by principal contractor for the Campaign were challenged by the ONDCP contract manager and not approved. Principal problem was due to HHS Program Support Center issuing contract without requiring contractor to have in place adequate accounting system for government contracts.

Evidence: GAO review of Media Campaign contract management; PricewaterhouseCoopers management review of ONDCP.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: In May 2002, following the receipt of disappointing results in the semi-annual report, ONDCP: increased testing of ads prior to airing them; targeted ads to a different age group; began involving ONDCP staff more directly in the message development process; and directed the Media Campaign staff to report directly to the ONDCP Director.

Evidence: Discussions with Media Campaign staff and others.

Does the program define the required quality, capability, and performance objectives for deliverables?

Explanation: Since the spring of 2002, 100 percent of Media Campaign TV ads have been tested prior to airing. Advertising developed under this new standard first aired in October of 2002. ONDCP has worked closely with PDFA and its advertising contractor to develop specific processes for the development and testing of all advertising.

Evidence: Discussions with Media Campaign staff and others.

Has the program established appropriate, credible, cost and schedule goals?

Explanation: Cost and schedule estimates for the contract used to purchase the advertising space and time ($121M) are now being reviewed and validated by an independent entity outside the program. A procedure is in place for an outside review of the costs associated with new ad development.

Evidence: Discussions with Media Campaign staff and others.

Has the program conducted a recent, credible, cost-benefit analysis that shows a net benefit?

Explanation: No cost benefits analysis has been performed.

Evidence: Discussions with Media Campaign staff and others.

Does the program have a comprehensive strategy for risk management that appropriately shares risk between the government and contractor?

Explanation: Campaign contracts: (1) do not have a comprehensive risk management plan that identifies technical, cost, and schedule risks and describes how these risks will be isolated, minimized, monitored, and controlled, and (2) are not selected using contracts and pricing mechanisms that provide appropriate incentives for contractors to meet cost, schedule and performance goals.

Evidence: Discussions with Media Campaign staff and others.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: ONDCP has established goals for the Media Campaign. However, progress toward the long-term goal will be reviewed against the final the NIDA-managed evaluation that provides longitudinal data assessing the effect of exposure to the Media Campaign against youth attitudes and behavior. Long term measure established only this year; no long-term progress to demonstrate, only annual.

Evidence: ONDCP Strategic Plan, FY 2004 Performance Plan, Annual Performance Reports (1999-2001), and Campaign Communications Strategy Statement.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual goals have been established but progress towards these measures will not be available until the final NIDA evaluation report is made available. Unless similar questions are asked in any subsequent evaluations, new annual measures will be needed.

Evidence: ONDCP Strategic Plan, FY 2004 Performance Plan, Annual Performance Reports (1999-2001).

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The Media Campaign does not have efficiency measures and targets, such as per unit cost of outputs, timing targets, or other efficiency and productivity indicators.

Evidence: Media Campaign Operating Plan

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: Although there is no closely comparable Federal program (in terms of target audience, behavior change sought, etc.) there have been a number of other media efforts designed to change the public's behavior in health-related matters. An analysis of 48 other such health behavior-change efforts found an average short-term effect that "roughly translates" into 9% more people performing the desirable behavior after exposure to the media effort than before.

Evidence: "A Meta-analysis of U.S. Health Campaign Effects on Behavior: Emphasize Enforcement, Exposure, and New Information, and Beware the Secular Trend," L.B. Snyder and M.A. Hamilton,

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Outcome data from the evaluation suggest little or no direct positive effect on youth behavior and attitudes attributable to the Campaign to date. Perhaps some positive effect on parental attitudes/behavior but that has not yet translated into an effect on youth.

Evidence: Evaluation of the National Youth Anti-Drug Media Campaign (NIDA).

Were program goals achieved within budgeted costs and established schedules?

Explanation: See numbers 1 and 2.

Evidence: See numbers 1 and 2.