Detailed Information on the
Census Bureau: Survey Sample Redesign Assessment

More clearly incorporate programmatic changes into strategic planning documents, including redesigning samples on a regular basis using the ACS. (This is a cyclical program that continues through 2016)

Continue more external evaluations as the program shifts from redesigning based on decennial data to redesigning on a more frequent basis using the ACS and a continually updated Master Address File. (This is a cyclical program, which runs through 2016)

Measure: Program milestones for continuous redesign activities including developing strategies for the coordination and unduplication of samples between household surveys, how to use a continuously updated MAF, and how to use ACS data when designing and selecting samples.

Explanation:FY 2005 Target: (1) Develop a strategy for coordinating and unduplicating samples between household surveys FY 2006 Target: (2) Consult with various external groups for external evaluations FY 2007 Target: (3) Complete MOUs with survey sponsoring agencies outlining surveys needs for 2010 Sample Redesign FY 2008 Target: (4) Issue initial results of Master Address File new construction coverage evaluation FY 2009 Target Issue final evaluation results and recommendation on acceptability of Master Address File as the sampling frame. FY 2009 Target: Issue final evaluation results and recommendation on acceptability of Master Address File as the sampling frame. FY 2010 Target: Develop a frame improvement strategy for demographic surveys. FY 2011 Target: Develop primary sampling unit definitions for demographic surveys. FY 2012 Target: Develop sample dissemination needs for demographic surveys.

Measure: Unit cost of Sample Housing Unit. (New measure, added February 2008)

Explanation:This efficiency measure reports the cost per sample housing unit. Because this is a cyclical program (10 year cycle) the performance data reflects the last year in the cycle.

Is the program purpose clear?

Explanation: The program selects new, updated statistical samples of the population for each decade to maintain the accuracy, relevancy and quality of the results of major federal surveys of families and households.

Evidence: Demographic Surveys Sample Redesign Goal Statement, FY 2004 budget submission to Congress

Does the program address a specific and existing problem, interest, or need?

Explanation: Statistical samples of the population need to be developed after each decennial to be representative of the location and composition of the American public. This prevents deterioration in the accuracy of the federally sponsored demographic surveys. The major household surveys requiring this sample redesign include the Current Population Survey, the Consumer Expenditure Surveys, the American Housing Surveys, the National Crime Victimization Survey, the National Health Interview Survey, and the Survey of Income and Program Participation.

Evidence: Demographic Surveys Sample Redesign Goal Statement, FY 2004 budget submission to Congress

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The survey sample redesign program is the only program that develops samples for major federally sponsored demographic surveys. In developing these samples, the Bureau also ensures data confidentiality.

Evidence: Sample Redesign Memoranda of Understanding with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS) Title 13 US Code

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The 2000 survey sample redesign is adequate for completing the sample survey redesign using the 2000 Census. The program is working to shift from redesigning based on decennial data to redesigning on a more frequent basis using the American Community Survey and a continously updated Master Address File.

Evidence: FY2004-2008 Census Bureau Strategic Plan Research Agenda for the 2000 Redesign "The Future of Sample Redesign" presentation given at the Interagency Committee Meeting on Survey Sample Redesign, Nov 12, 2002

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Memoranda of Understanding (MOU) with sponsoring agencies contain reliability requirements and performance goals and milestones that seek to ensure that resources are effectively used.

Evidence: Sample Redesign Memoranda of Understanding with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The sample survey redesign program has one performance goal, to produce accurate, timely, and relevant statistics by developing new samples that reflect the current characteristics and geographic location of the population. Performance measures are the sample design requirements and goals including accuracy requirements and release schedules contained in Memoranda of Understanding with sponsor agencies.

Evidence: The DOC FY 2004 Annual Performance Plan and Census Bureau FY 2004-08 Strategic Plan Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS)

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: In general, the long term sample redesign targets are ambitious and meet the needs of the sponsor agencies. Further, the release schedule is ambitious because the 2000 redesign program accelerates the release of two samples by a year as compared to the release schedule of the 1990 redesign program.

Evidence: FY 2004-08 Strategic Plan Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS)

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The survey sample redesign has established annual milestones including developing and releasing sample frames that contribute to desired long-term outcomes. Milestone schedules are contained in MOUs with sponsoring agencies.

Evidence: FY 2002 Congressional Budget Submission; Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS)

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: Annual targets for 2000 redesign activities are ambitious and meet the needs of sponsor agencies. Further, the annual targets for the release schedules are ambitious compared to the 1990 redesign activities.

Evidence: Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS)

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The survey sponsors are consulted on a regular basis on statistical methodology for redesign of samples. Contracts for advisory and assistance services engage outside expertise in meeting annual and/or long term goals.

Evidence: Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS) Minutes of Sample Redesign Steering Committee Meetings with sponsoring agencies Contractual agreements documentation.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There were external evaluations conducted for the 1980 cycle of sample redesign due to extensive design changes. However, with fewer changes in 1990 and 2000 redesigns, there have been fewer external evaluations beyond input and comments provided by survey sponsors. Sponsors regularly monitor and assess redesign activities through the Interagency Sample Redesign Steering Committee. Source and accuracy statements are also released with survey results. Census should consider more external evaluations as the program shifts from redesigning based on decennial data to redesigning on a more frequent basis using the American Community Survey and a continously updated Master Address File.

Evidence: Minutes of Sample Redesign Steering Committee Meetings with sponsoring agencies Panel on Current Population Survey Redesign Research

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Census Bureau budget requests tie resource requests to the accomplishment of performance goals for new initiatives and incorporate unit costs for base and new activities. However, improvements can be made in tying base activities to annual and long term performance goals.

Evidence: FY 2004 Budget Request

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The sample redesign program does not have a well-developed strategy in place for redesigning on a regular basis using the American Community Survey and a continuously updated Master Address File. The program is currently developing a strategy and should work to develop targets in budget and strategic planning documents.

Evidence: Census Bureau's Strategic Plan for 2004 - 2008

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program maintains a program management information system that manages the redesign work in order to complete it on time. This system helps the program conduct ongoing analysis and review of program management practices. The program also regularly updates survey sponsors on progress on sample redesign activities.

Evidence: Program Management Information System Documentation memoranda on survey management Minutes of sample redesign steering committee meetings

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: Annual performance plans for managers include goals and meaures that are linked to Census Bureau's strategic goals. Manager performance plans contain specific performance standards for the sample redesign program including project deliverables for the seven major sample redesigns. All contracts are carefully monitored and contractors are held accountable for cost, schedule, and performance.

Evidence: Employee performance plans

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Monthly and quarterly obligation reports indicate the Census Bureau funds programs in a timely and appropriate manner

Evidence: Approximately 98% of budgetary resources in the Periodics and Programs account was obligated at the end of FY 2003.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In developing the 2000 redesign, the program improved efficiencies by creating an automated listing in the sample procedures, improving the stratification and sample selection and unduplication procedures, and incorporating a state based design for the SIPP into the current redesign at no additional cost to be able to provide state based estimates. The survey sample redesign program also competitively sources its information contracts and utilizes large in-place Bureau contracts to achieve efficiencies of scale.

Evidence: Research Agenda for the 2000 Redesign R&D 2007 contract and the Information Technology Services contract.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program collaborates and coordinates with other statistical agencies in developing and implementing sample redesign activities. Sample redesign requirements are developed and incorporated into MOUs with sponsor agencies. The program also regularly updates survey sponsors on progress on sample redesign activities.

Evidence: Interagency Sample Redesign Steering Committee Memoranda of Understanding with sponsoring agencies

Does the program use strong financial management practices?

Explanation: The program's financial management is reflected in Census Bureau's clean audit opinions.

Evidence: Clean audit opinions since 1999 reflected in the Bureau's Financial Reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: In general, the survey sample redesign program has adequate program management. The Census Bureau has developed areas of competency that managers must meet in their jobs and several programs have been developed to meet training needs in competency areas.

Evidence: Project Management Master Certificate Program

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: The survey sample redesign program is on schedule to complete the samples for the seven major household surveys by 2005

Evidence: Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS)

Does the program (including program partners) achieve its annual performance goals?

Explanation: The survey sample redesign program reports on redesign activities to the Interagency Survey Sample Redesign Committee meetings. Further, MOUs contain annual milestones. In general, the survey sample redesign program has met its annual performance goals.

Evidence: Sample Redesign MOUs with sponsoring agencies (e.g. BLS, HUD, NCHS, BJS), Progress reports to Interagency Survey Sample Redesign Committee

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The survey sample redesign program has improved efficiencies because the 2000 redesign funding is comparable to the funding received for the 1990 redesign and the following operational improvements have been added: automated listing and sample selection of the primary sampling units, incorporation of a state based design into the SIPP, completing the samples for two surveys at an accelerated schedule.

Evidence: Research Agenda for the 2000 Redesign Progress reports to Interagency Survey Sample Redesign Committee

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: The program has the largest scope and most advanced model for redesigning samples.

Evidence: The program is regularly consulted by statistical programs and agencies world-wide on its sample redesign process.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Evaluations completed by Sample Redesign Steering Committee Meetings indicate the redesigns are on schedule and meeting the needs of the sponsors.

Evidence: Minutes of Sample Redesign Steering Committee Meetings with sponsoring agencies