Detailed Information on the
Bureau of Justice Statistics Assessment

Reviewing data collection efforts for the National Criminal Victimization Survey to identify potential cost efficiencies that will still allow for statistically valid estimates.

Including performance information in budget submissions.

Planning a comprehensive review of the Bureau to demonstrate the impact of its programs.

Measure: Average number of user sessions per month on BJS and BJS-sponsored websites, including datasets accessed and downloaded via the Internet.

Explanation:The measure reflects the level of requests and utilization of BJS data. Dissemination-related measures provide data on the availability, accessibility, and distribution of products and services. One of the most fundamental long-term goals of BJS is to move from traditional paper-based dissemination to web-based distribution and utilization. As these targets increase, the targets for traditional dissemination and utilization will decrease.

Measure: Number of requests to seek correction of BJS data in accordance with the BJS Data Quality Guidelines

Explanation:Because the core function of BJS is the production and dissemination of the highest possible quality of justice statistics, there is no limitation on who may seek the correction of BJS statistics. Any individual, unit of government at the Federal, State, local, and tribal levels, or private organization has the right to seek the correction of published BJS data if the complaining entity has a good faith and nonfrivolous basis to believe that BJS failed to comply with the BJS Data Quality Guidelines. Guidelines became effective in October 2002.

Measure: Citations of BJS data in social science journals, law reviews and journals, and publications of secondary analysis using BJS data.

Explanation:Relevance measure to monitor the degree to which data and products are useful and responsive to users' needs. Indicators include the type and frequency of use of data. These measures are useful to determining whether we are meeting recognized governmental and societal information needs and addresses the linkage between statistical outputs and programmatic outcomes. (2005 data not available until March-April 2006.)

Measure: Congressional record and testimony citing BJS data

Explanation:Relevance measure to monitor the degree to which data and products are useful and responsive to users' needs. Indicators include the type and frequency of use of data. These measures are useful to determining whether we are meeting recognized governmental and societal information needs and addresses the linkage between statistical outputs and programmatic outcomes.

Measure: Number of products that BJS makes available online

Explanation:Includes electronic versions of questionnaires, BJS reports, downloadable spreadsheets, datasets, and codebooks. This is a program performance related measure designed to examine customer access to data, products, and services.

Measure: Agency-level response rate

Explanation:BJS established a threshold value of 95%, representing an acceptable and statistically sound level of response as established by the social science statistical community. This is an accuracy measure calculated as an average response rate of all data collection response rates in which an agency (i.e., correctional facility, court) responded. Data collection response rates are calculated as the percentage of agencies that voluntarily participate in the data collection.

Measure: Citizen-level response rate

Explanation:Average response rate of all survey response rates in which an individual (i.e., resident, inmate) responded. BJS uses a threshold value of 90%, representing an acceptable and statistically sound level of response as established by the social science statistical community. This is an accuracy measure calculated as an average response rate of all data collection response rates in which an individual (i.e., resident, inmate) responded. Individual survey response rates are calculated as the percentage of households/facilities in which respondents voluntarily participated in the survey.

Measure: Federal and State court opinions citing BJS data

Explanation:Relevance measure to monitor the degree to which data and products are useful and responsive to users' needs. Indicators include the type and frequency of use of data. These measures are useful to determining whether we are meeting recognized governmental and societal information needs and addresses the linkage between statistical outputs and programmatic outcomes.

Measure: Number of reports issued within one month of the expected release date.

Explanation:These reports represent the key indicators of crime and justice. Targets a threshold for annually recurring data releases of ??1 month of the median time for expected releases as measured against the planned date of publication. This is an accuracy measure used to assess the usability of information products.

Measure: Index of operational efficiency

Explanation:Program performance measure (cost and productivity) combined to monitor performance across operational goals and tasks. Sets 10 as the average for the baseline period (total index scores below 10 are years in which efficiency was lower than the baseline average and total index scores above 10 indicate above-average efficiency).

Measure: The number of scheduled data collection series and special analyses to be conducted

Explanation:Quality/accuracy measure to gauge success in providing useful comprehensive data covering each stage of the justice system and broadening the nation's knowledge base on crime and justice.

Is the program purpose clear?

Explanation: As the statistical arm of the U.S. Department of Justice, the Bureau of Justice Statistics (BJS) is responsible for the collection, analysis, publication, and dissemination of statistical information on crime, criminal offenders, victims of crime, and the operations of the justice system at all levels of government. The program purpose for BJS is explicitly set forth in the Justice Systems Improvement Act of 1979 as an amendment to the Omnibus Crime Control and Safe Streets Act of 1968 (42 U.S.C. 3731 3735). By making BJS a key component of the Justice Systems Improvement Act, the Congress clearly recognized the utility for accurate statistics to improve State and local justice systems. The Act authorizes BJS to provide justice statistics to the President, the Congress, the judiciary, State and local governments and the general public. Section 302(c) specifies in 23 subparts the statistical data BJS is authorized to collect and the types of related activities it is authorized to perform. The Act requires that BJS give primary emphasis to the problems of State and local justice systems. BJS' mission and program goals are clearly and consistently stated in numerous Administration and Departmental documents, as well as on the BJS and Fedstats.gov websites. It is the mission of BJS to collect, process, analyze, and disseminate accurate and timely information on crime and the administration of justice, and to assist States and localities to improve criminal justice record keeping. BJS strategic goals, as set forth in the BJS Strategic Plan, 2005-2008, include: Goal 1 To produce national statistics on crime and the administration of justice that facilitate measurement over time and across geographic areas. Goal 2 To improve record keeping by State and local governments and to improve the capacity of States and localities to produce statistics on crime and the administration of justice. Goal 3 To ensure public access to justice statistics and data.

Evidence: BJS authorizing legislation, 42 U.S.C. 3731-3735 assembler.law.cornell.edu/uscode/42/3731.html BJS mission and description of statistical programs available at: BJS website www.ojp.usdoj.gov/bjs/aboutbjs.htm FEDSTATS website www.fedstats.gov BJS Strategic Plan, 2005-08 www.ojp.usdoj.gov/bjs/pub/pdf/bjssp04.pdf Statistical Programs of the U.S. Government, FY 2005 www.whitehouse.gov/OMB/inforeg/05statprog.pdf Annual budget requests and related GPRA documents, including Analytical Perspectives, Budget of the United States Government, Fiscal Year 2006 www.whitehouse.gov/omb/budget/fy2006/pdf/spec.pdf

Does the program address a specific and existing problem, interest, or need?

Explanation: The BJS Criminal Justice Statistics Program addresses the need for objective national statistics on crime and justice for policy and decision making by Federal, State, and local officials, the judiciary, criminal justice practitioners, private organizations, academia, the media, and the general public. BJS data are in high demand as clearly demonstrated by the significant use of its website, which receives up to 20,000 users per day, with a recorded high of 68,000 users in one day. BJS data citations are monitored to gauge the need and relevance of its information. Many BJS reports are accompanied by press releases or are put directly on the newswire, achieving prominent coverage by the Nation's electronic and print media. Recent news reports cite BJS data on a myriad of issues including female homicide, property crime, forensic laboratory backlogs, sex offenders, prison population, alternatives to incarceration, drug offenders, teen victims, sexual assault against children, sentencing of convicted felons, law enforcement administrative data, carjacking, American Indian victimization, crime victim assistance funds, and alcohol and crime. Federal, State, and local officials and practitioners use BJS data for a wide variety of purposes such as policymaking, program development, resource and budgetary justification, grant proposal development, management analysis and program evaluation, and administrative process improvements. BJS provides technical expertise to other Executive branch agencies and frequently conducts special analyses and data collections on their behalf. Recent documented examples that demonstrate use of agency technical expertise and data can be found in Evidence.

Evidence: In 2004, BJS data were cited in the Congressional Record (23 times); Congressional testimony (12 times); Federal and state court decisions (20 case opinions); law reviews and law journals (296 articles); and social science journals (694 articles). Selected recent Administration documents citing BJS data: Economic Report of the President, February 2005 a257.g.akamaitech.net/7/257/2422/17feb20051700/www.gpoaccess.gov/eop/2005/2005_erp.pdf U.S. Efforts to Combat Trafficking in Persons, U.S. Department of Justice, May 2004 www.usdoj.gov/ag/050104agreporttocongresstvprav10.pdf Advancing Justice Through DNA Technology, White House DNA Initiative Policy Book, March 2003. www.whitehouse.gov/infocus/justice/dna_initiative_policy_book.pdf Sampling of news reports citing BJS data: news.google.com/news?hl=en&ned=&q=%22Bureau+of+Justice+Statistics%22 Congressionally-mandated reporting: Prison Rape Elimination Act of 2003 frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=108_cong_public_laws&docid=f:publ079.108.pdf Deaths in Custody Reporting Act of 2000 frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=106_cong_public_laws&docid=f:publ297.106.pdf Crime Victims with Disabilities Awareness Act frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=105_cong_public_laws&docid=f:publ301.105.pdf Violent Crime Control and Law Enforcement Act of 1994 (Police Use of Force) frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=103_cong_bills&docid=f:h3355enr.txt.pdf

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: BJS, as one of the ten principal Federal statistical agencies, is the only agency producing official national statistics covering every component of the criminal justice system. No other data sources produce national-level data that (1) describe characteristics and consequences of an estimated 24.2 million annual violent and property crime victimizations; (2) analyze the operations of some 50,000 agencies, offices, courts, and institutions comprising the justice system; (3) examine judicial decision-making and how criminal and civil cases are managed and adjudicated as they traverse the justice system; (4) count populations and describe the nearly 6.9 million adults who during an average day are subject to the care, custody, or control of Federal, State, or local criminal justice authorities; and (5) provide workload, activities, and outcomes associated with Federal criminal cases, using data collected from both the Executive and Judicial branches. BJS uniquely serves the policy and legislative needs of the U.S. Department of Justice, the Administration, Congress, the criminal justice community, and the general public. It is also the only Federal agency providing leadership and technical guidance to State-level Statistical Analysis Centers, agencies established by each State to coordinate and conduct statistical activities on crime and justice.

Evidence: "Statistical Programs of the U.S. Government, FY 2005 www.whitehouse.gov/OMB/inforeg/05statprog.pdf Annual budget requests and related GPRA documents, including Analytical Perspectives, Budget of the United States Government, Fiscal Year 2006 ttp://www.whitehouse.gov/omb/budget/fy2006/pdf/spec.pdf White House Social Science Briefing Room www.whitehouse.gov/fsbr/crime.html

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: BJS develops, maintains, and disseminates its statistics in accordance with established statistical and data quality standards. BJS adheres to the Principles and Practices for a Federal Statistical Agency as established by the National Academy of Sciences. BJS analysts use only accepted research methods and established statistical practice throughout the survey development and implementation process. BJS has established a data collection infrastructure based on skilled statisticians and experienced data collection agents. BJS staff regularly coordinates with statisticians and other experts at the Census Bureau and other Federal statistical agencies to ensure that programs are utilizing acceptable methodologies. BJS maintains a rigorous quality control process, as documented in its Data Quality Guidelines.

Evidence: BJS's procedures and statistical standards are well documented for internal and external review. BJS' data collections and methodologies are reviewed and approved by the OMB Office of Information and Regulatory Affairs through the OMB clearance process as required by the Paperwork Reduction Act. The U.S. Census Bureau, including the Statistical Methods Division, Demographic Survey Division, and Governments Division, provides technical and methodological appraisal of BJS data collections carried out by the Census Bureau. Other external review includes the U.S. Department of Justice Advisory Board of Research and Statistics; the OMB-chaired Interagency Council on Statistical Policy; the Federal Committee on Statistical Methodology; the National Academy of Sciences' Committee on National Statistics; the American Statistical Association; and experts in the justice and statistical community. BJS Data Quality Guidelines www.ojp.usdoj.gov/bjs/dataquality.htm Principles and Practices for Federal Statistical Agencies, National Academy of Sciences books.nap.edu/catalog/11252.html OMB Statistical Standards and Guidelines www.whitehouse.gov/OMB/inforeg/statpolicy.html National Archive of Criminal Justice Data, repository of all BJS public-use datasets www.icpsr.umich.edu/NACJD/index.html U.S. Dept. of Justice Memorandum establishing responsibilities of the Advisory Board Reports and memoranda from U.S. Census Bureau examining various statistical and methodological issues

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: BJS uses a variety of mechanisms to ensure that it identifies and responds to customer needs, and measures satisfaction with BJS products and services. Customer feedback surveys: BJS periodically conducts surveys among those using both electronic and print materials. By surveying our registered and targeted users, BJS can obtain information on what our customers do, whether they use statistics to perform their functions, and what information resources they use. Recent findings revealed that about half the customer base is composed of persons employed in the criminal justice system. Customers we are directly reaching use statistics and use BJS reports (92% of registered users use statistics to perform their jobs or activities) and most recipients keep or share the documents. Almost 95% of the registered users said that they looked for information on the Internet, the most frequent behavior listed. BJS is using these findings to improve distribution strategies, target user communities, and focus resources on web-based capabilities. For instance, BJS is launching a joint website with the FBI to provide a single portal for crime and justice data. Website usability testing: BJS utilizes a vigorous program of usability testing to learn about the customer' 'ability to navigate the website and whether they are finding the data they are seeking in a timely and efficient way. Based on findings, changes to the content and/or presentation of web-based BJS materials or information are made. Website user log analysis: In order to optimize access to the most sought-after information, BJS maintains data on user sessions to identify topics and findings most frequently accessed. For example, BJS uses this information to determine whether topical reports are still relevant and require updating and website content decisions including creating web pages dedicated to single topic areas, such as homicide and reentry of offenders. Technical assessments of dissemination modes: BJS examines its dissemination techniques (i.e., statistical compendia analysis, archive of public use data) to determine how users make choices among information sources and what tools and techniques would best support our users. For instance, a recent study supported by BJS showed that simply transferring publications to the web fails because users approach content on the web differently than content on paper. In response to this BJS is enhancing its interactive capabilities on its website, as well as redesigning its annual statistical compendia to a web-based format. As a result of an assessment of its public use data archive, machine-readable data are now available on multiple media (cd rom, downloadable via the web) and a better linkage has been provided between BJS reports and source datasets (users can link directly to the dataset from the publication web page). Analysis of ASKBJS questions/responses: BJS examines the content of all questions and replies to identify emerging issues and opportunities for data needed by the public that are not currently being collected. Commonly sought after information has been created in special topic web pages, such as drugs and crime. Budget enhancements frequently request funds to establish data collections to fill information gaps as identified through repeated inquiries (i.e., computer crime, identity theft, civil judgments). In addition, BJS tracks the use of BJS data as cited in Federal and State court opinions, Congressional Record and testimony, law reviews and social science journals, and the media. BJS also provides technical expertise to other Executive branch agencies, frequently conducting special analyses and expanding data collections to provide information essential to sound policy making (e.g., new questions on the NCVS related to identity theft and stalking, computer crime survey of businesses, HIV and tuberculosis in prisons and jails, racial profiling).

Evidence: BJS customer feedback surveys www.ojp.usdoj.gov/bjs/pub/html/bjscfs.htm www.ojp.usdoj.gov/bjs/abstract/msrbjspp.htm Technical assessment of the Sourcebook www.ojp.usdoj.gov/bjs/pub/pdf/ps21c_es.pdf

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Through its GPRA reporting, strategic planning, and management improvement processes, BJS has established performance measures designed to assess its statistical programs and operations. The long-term measures focus on product quality, encompassing relevance, accuracy, and timeliness, as well as program performance, factors such as cost, dissemination, and mission achievement. Measure #1 gauges BJS's success in providing useful comprehensive data covering each stage of the justice system and broadening the nation's knowledge base on crime and justice. These series generally recur at specific intervals in order to facilitate trend analysis. Measure #2 provides an index of operational efficiency designed to assess whether the results achieved justified the cost and resource burden. (This is both a long-term and annual measure.) Measure #3 provides information on the utilization of BJS data. This measure allows BJS to assess the adequacy of the availability and accessibility of its products and services, as well as the degree to which its data are useful and responsive to users' needs. (The "Yes" answer for this question is predicated on an understanding that these measures will be reflected in DOJ's 2007 budget and performance plan.)

Evidence: Long-Term Performance Measures: Outcome Measure - The number of scheduled data collection series and special analyses to be conducted. Efficiency Measure - Index of operational efficiency (setting 10 as the average for the baseline period - total index scores below 10 are years in which efficiency was lower than the baseline average and total index scores above 10 indicate above-average efficiency). Outcome Measure - Average number of user sessions per month on BJS and BJS-sponsored websites, including datasets accessed and downloaded via the Internet. BJS performance measurement concepts are further explained in the BJS Strategic Plan, 2005-08 www.ojp.usdoj.gov/bjs/pub/pdf/bjssp04.pdf

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: BJS' has reasonably ambitious targets and timeframes for its long-term measures. Targets are set using an evaluative and deliberative process that considers past year performance, resource availability, as well as the somewhat unpredictable customer demand for BJS data. A. Measures and Targets for Data Collection BJS targets for the number of collections to be conducted are based on estimates of the number of recurring series and special analyses funded within BJS budget requests. Data collection targets increase or decrease because of the cyclical nature of surveys. In addition, targets are adjusted to reflect actual appropriations for a given fiscal year. Over the last ten years, BJS's statistical program has significantly expanded to cover every stage of the criminal justice system, as well as addressed specific areas of concern to policy makers and practitioners, such as American Indian victimization, victimization of the elderly and disabled, identity theft, crime lab workload and resources, racial profiling, and hate crimes. B. Efficiency targets for BJS operations BJS' long-term efficiency indicator represents productivity relative to resources expended. BJS' targets focus on sustained efficiency improvements over the baseline period of 1996-2003. C. Customer use of BJS data One of BJS's most fundamental goals is to improve product accessibility. The targets demonstrate the continuing growth and improvement in the data accessibility, which leads to increased demand from the public. As demand is not always predictable, nor controllable by BJS, actual demand has fluctuated at times to levels higher than targeted. Yet, BJS targets are not set based on the actual performance for a single year. Instead, target setting considers recent performance for a number of recent years.

Evidence: See performance measures section of the PART, including reported targets and actuals. BJS Strategic Plan, 2005-08 www.ojp.usdoj.gov/bjs/pub/pdf/bjssp04.pdf

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: BJS, like the other Federal statistical agencies, has focused on measuring performance in the area of product quality, especially the aspects most amenable to quantitative measurement including (but not limited to) relevance, accuracy, and timeliness. These are outcome-oriented measures and are key to the usability of information products. Additional performance indicators assess performance by assessing whether customers receive the information they need via the most appropriate mechanisms. "Citizen-level response rate" and "Agency-level response rate" are accuracy measures that quantify the correctness, validity, and reliability of data. "Number of reports issued within one month of the expected release date" is a timeliness measure used to assess the timing of information releases. The schedule for the release of data is directly impacted by the timeliness of the front-end stages (i.e., data collection, analyses, and report writing). The data published in the reports monitored for this measure represent key indicators of crime and justice, including crime victimization rate, prison and jail counts, probation and parole counts, capital punishment data, federal case processing and sentencing, and background checks for firearm purchases. "Number and types of products that BJS makes available online" is program performance related measure designed to examine customer access to data, products, and services. A fundamental BJS goal is to improve product accessibility, moving from traditional paper-based dissemination to web-based accessibility. BJS has been expanding the number of products available on the BJS website to address growing demand for electronically formatted questionnaires, tables, charts, and spreadsheets. Datasets with supporting documentation also are available via online direct electronic access through the NACJD website. Indicators showing citations of BJS data are relevance measures used to gauge the degree to which data and products are useful and responsive to users' needs. These measures are useful to determining whether BJS is meeting governmental and societal information needs. These indicators track the citation of BJS data in court opinions, the Congressional record, and social science and legal research. "Number of requests to seek correction of BJS data" is both a product quality and program performance measure designed to assess the quality of BJS data and the credibility of its statistical program through customer satisfaction. The measure monitors the number of complaints filed in accordance with the BJS Data Quality Guidelines. Any individual, unit of government at the Federal, State, local, and tribal levels, or private organization has the right to seek the correction of published BJS data if the complaining entity has a good faith and nonfrivolous basis to believe that BJS failed to comply with the BJS Data Quality Guidelines. The "index of operational efficiency" is designed to assess whether the results achieved justified the cost and resource burden. (This is both a long-term and annual measure.)

Evidence: Performance Measures: Output - Citizen-level response rate Output - Agency-level response rate Output - Number of reports issued within one month of the expected release date. Outcome - Number and types of products that BJS makes available online Outcome - Federal and State court opinions citing BJS data Outcome - Congressional record and testimony citing BJS data Outcome - Citations of BJS data in social science journals, law reviews and journals, and publications of secondary analysis using BJS data. Outcome - Number of requests to seek correction of BJS data in accordance with the BJS Data Quality Guidelines. Efficiency - Index of operational efficiency (setting 10 as the average for the baseline period - total index scores below 10 are years in which efficiency was lower than the baseline average and total index scores above 10 indicate above-average efficiency). BJS Strategic Plan, 2005-08 www.ojp.usdoj.gov/bjs/pub/pdf/bjssp04.pdf Analytical Perspectives, Budget of the U.S. Government, FY 2006 www.whitehouse.gov/omb/budget/fy2006/pdf/spec.pdf

Does the program have baselines and ambitious targets for its annual measures?

Explanation: BJS has established reasonably ambitious targets for each of its annual performance measures. The targets are the outgrowth of an evaluative and deliberative process that places a premium on good program performance. Some fluctuation, especially for customer demand indicators, is expected. Because of this fluctuation, as well as the fact that targets are based on recent performance for more than a single year, some future targets are lower than recent reported actuals. At the same time, BJS' targets focus on continued or sustained improvement over established baselines.

Evidence: See performance measures section of the PART, including reported targets and actuals. The Department will include BJS' annual measures in the Department's 2006 GPRA performance plan/budget.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: BJS partners are integral to the production of national statistics on crime and the administration of justice, serving as either data collection agents or data suppliers. As a data collection agent, each partner functions as an extension of BJS under interagency and cooperative agreements that specify the protocols to be used for meeting data collection objectives. BJS staff identifies respondents, specifies methodologies to be used, and establishes benchmarks to be achieved in meeting BJS statistical responsibilities. Data collection partners are assessed on their conformity with BJS expectations and terminated if they do not meet the requirements set forth in the cooperative agreement. Data collection agents are expected to meet established timelines for the data collection. BJS tracks the time from the initiation of the collection to close of the reference period. For many of its statistical series, BJS recently put into place an automated tracking system. Managed by the data collection agent and accessible to BJS project monitors, the system allows "real time" monitoring of a project's progress. Problems such as nonresponse issues can be spotted and resolved at once. BJS primarily acquires data collection partners through a competitive process that insures that the awardee will be fully compliant with BJS objectives and will be held fully accountable for any steps that do not further these objectives. Data supply partners consist of the 50,000 agencies, offices, and institutions that compose the criminal justice system nationwide. BJS has a strong commitment from these components of the justice system to participate in national statistics series. Each series poses a substantial burden on these respondents to participate by agreeing to provide a variety of data requested by BJS or its agents. BJS enjoys very high response rates from these partners indicative of the value received in relation to the cost of participation. Collections from law enforcement agencies, prosecutors, local jails, and State and Federal prisons have routinely experienced 100% cooperation and participation. Accepting this burden is the strongest evidence of the commitment of agency officials to BJS objectives.

Evidence: See, for example, information on the BJS State Justice Statistics Program: www.ojp.usdoj.gov/bjs/pub/html/sjsp05sol.htm Also see: BJS Strategic Plan, 2005-08 www.ojp.usdoj.gov/bjs/pub/pdf/bjssp04.pdf

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The American Statistical Association (ASA) and National Academy of Sciences (NAS) provide high-quality, independent, and comprehensive reviews of BJS on a regularly recurring basis. In addition, numerous academic researchers and the Census Bureau, review the individual data series. Scope of Evaluations---BJS relies primarily on ASA and NAS for recurring comprehensive reviews of statistical programs. Individual series are reviewed in NAS publications and in refereed journal publications. Committees of the ASA and NAS meet twice annually. The ASA meetings focus solely on assessing BJS operations and statistical data. The NAS focuses more generally on Federal statistics, with a more intense review of BJS every other year. Most recently, BJS was subject to its NAS biennial review on May 6, 2005. Quality of Evaluations---ASA and NAS represent the highest quality reviews available in the field of statistics and social science. Independence of Evaluations---In each case, the respective ASA and NAS panels are independent of the Federal government, though the panels are partly funded by BJS. Panel composition for both ASA and NAS is largely drawn from the academic community.

Evidence: See for example: "Measurement Problems in Criminal Justice Research" National Academy of Sciences, 2003 www.nap.edu/catalog/10581.html In 2003, a major review of the National Crime Victimization Survey, which accounts for nearly two-thirds of BJS appropriated statistics funding, appeared in the NAS publication "Measurement Problems in Criminal Justice Research." The NAS review noted some problems in reconciling certain counts of incidents with those of other surveys but concluded that methodological differences could be the principal reasons for disparity. Also see, for example, interagency memoranda submitted to OMB indicating Census review of National Crime Victimization Survey.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: While budget submissions are generally well justified, there is no explicit linkage given in the OMB or the congressional budget submissions to achievement of performance targets. In fact, performance targets for BJS have not been included in recent OMB or congressional budget submissions. Making these linkages explicit in budget submissions is needed for budget decision-making, both within the Executive Branch and the Congress. BJS does attempt to connect budget and performance in its own internal budget submissions within the Department.

Evidence: 2006 Dept. of Justice Congressional Budget Submission. 2006 Dept. of Justice OMB Budget Submission.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: BJS does not face many fundamental strategic planning deficiencies. In recent years, appropriations generally have not kept pace with budget requests, requiring continual reprioritizing and adjustment to maintain critical data products. BJS has been successful in adjusting to these funding constraints. While significant performance information for BJS has been lacking from prior DOJ budget submissions, BJS is working with the Department to ensure that future budget requests include key BJS performance measures.

Evidence: BJS Strategic Plan FY 2005-2008 <www.ojp.usdoj.gov/bjs/pub/pdf/bjssp08.pdf> 2002-2005 appropriations and President's Budget. See performance measures.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: BJS collects a variety of performance data to inform program management, make resource decisions, and assess program performance as a part of the national series maintained by BJS. Performance information gathered from data collection agents focuses on the quality (accuracy and completeness) and timeliness of the data, as well as operating and cost measures. BJS and its collection agents continuously monitor the timeliness and quality of data submissions from data providers. Such information includes caseloads, completions, non-interviews, response rate, and costs. BJS examines these data to evaluate technical and methodological aspects of the program, to improve the quality and timeliness of its data collections, and to ensure that project activities are on track, meeting established timelines, and remain within spending limits. BJS dissemination and clearinghouse providers collect and report data on the products and services made available to users (i.e., electronic datasets accessible, information requests filled), as well as information used by BJS to assess their performance in delivering such services (i.e., length of time to respond to requests, customer satisfaction). Data gathered from these providers on customer use of products and services are used to assess the types and levels of demand for products and services and develop/change product lines, dissemination strategies, and distribution plans. BJS supports State-level Statistical Analysis Centers (SACs) to coordinate and conduct statistical activities and to serve as a liaison between BJS and respondent State and local agencies. Performance data from the SACs are used to monitor development of State and local databases, publication of studies and reports, and collaborative efforts and multi-state research projects with national implications. States and localities generally model their statistical collections to coincide with national collection programs such as the National Prisoners Statistics Program. BJS uses these performance data to focus resources on particular areas of research, technical assistance needs, and collaborative data collection efforts.

Evidence: See, for example: BJS Strategic Plan, 2005-08. BJS program solicitations specifying required performance data. Federal Justice Statistics Program (FJSP) solicitation, a task agreement with the Urban Institute to carry out the FSJP, and progress and financial reports from the Urban Institute to demonstrate performance data obtained. U.S. Census Bureau progress and financial report for the NCVS. Performance reports from Aspen, NCJRS. Progress and financial reports from the New York, Florida and Georgia Statistical Analysis Centers.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Individual performance plans for managers are linked to a series of organizational goals set forth in the BJS Strategic Plan, the OJP Management Plan, the DOJ Strategic Plan, and the President's Management Agenda. Performance ratings of managers are tied to the achievement of Departmental and program-specific goals and outcomes in the Department's Strategic Plan. Performance evaluations are based on accountability for organizational results, staff/workforce, and taxpayer value. Managers' performance standards incorporate BJS performance measures in the areas of operational and cost efficiencies, product quality, data availability and access, and customer satisfaction. Federal managers are required to commit to quality and professional standards of practice. Performance standards, quality assurances, and deliverables are delineated in funding agreements with data collection agents and other program partners. Data collection partners and other funding recipients are held accountable for effectively managing financial resources, completing tasks on time, providing reliable data, and delivering quality services and products. Data collection agents provide detailed information on performance including caseloads, completions, non-interviews, response rates, timeliness, and costs. Additionally, on June 30, 2004, the Attorney General transmitted decisions on the SES Performance-based Pay System to OPM and OMB with the Human Capital report, including a generic work plan for all Departmental SES members, with accompanying Performance "contract" that must explicitly relate to the Department's, the President's or the AG's defined goals. On December 10, 2004, DOJ obtained approval for the SES and GS/Prevailing Rate performance orders. DOJ has completed the application package for OPM/OMB certification of the DOJ SES Performance Management and Compensation Plan. DOJ components implemented five-level performance plans for all SES. New SES and manager plans include cascading tasks/assignments that are linked to the DOJ Strategic Plan and the PMA. By Dec 30, 2004, all DOJ components certified to the Attorney General that all SES and direct report performance work plans are in place. By the end of the December 2004, the Department had performance appraisals for most of the work force that: link to agency mission, goals and outcomes in DOJ's Strategic Plan; hold employees accountable for results appropriate to their level of responsibility; effectively differentiate between various levels of performance; and provide consequences based on performance.

Evidence: All managers are subject to the new DOJ Performance Management System. Attorney General's Memorandum dated August 10, 2004 - "Performance Management Plans for SES Members and Managers" www.usdoj.gov/jmd/ps/memagpma-ses1.htm Copies of performance standards for key Federal program managers. Key managers include: Director, BJS; Chief, Corrections Statistics Unit, Chief, Victimization Statistics Unit; Chief, Law Enforcement, Adjudication, and Federal Statistics; Chief, Research and Public Policy Issues; Chief, Criminal Statistical Improvement Program; Chief, Planning, Management, and Budget; Chief, Publication Development and Verification; and Chief, Publication and Electronic Dissemination. BJS Data Quality Guidelines www.ojp.usdoj.gov/bjs/dataquality.htm Examples are on file of performance standards required of data collection agents (Urban Institute, RTI, NORC) and other BJS partners (Aspen, NCJRS). BJS program solicitations and funding agreements clearly specify what the recipients must provide to measure program results, as well as to contribute to the achievements of BJS's overall statistical program. Funding recipients also must comply with OJP fiscal requirements set forth in the OJP Financial Guidelines (www.ojp.usdoj.gov/FinGuide/).

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All BJS program funds are obligated and committed within the fiscal year of the appropriation. The OJP Office of the Comptroller (OC), Contracts Office, and BJS staff monitor draw down and expenditure of awarded funds. Financial status reports from recipients are closely examined to ensure that funds are being spent as scheduled; are dedicated to costs allowable by program objectives, the terms of the agreement, and OJP fiscal requirements; and are in compliance with Federal cash management regulations and OMB Circular A-133, Audits of States, Local Governments, and Non-Profit Organizations, as appropriate. Office of the Comptroller onsite reviews determine whether: (1) grantees are properly accounting for the receipt and expenditure of Federal funds, and (2) expenditures are in compliance with Federal requirements and award special conditions. BJS program managers closely review financial reports and programmatic progress reports to ensure that funds are being spent for program purposes. BJS management rigorously assesses requests for no-cost extensions and changes to the budget.

Evidence: Spending: FY 2001: Appropriation, $28,991,000 Obligation, $28,875,753 FY 2002: Appropriation, $32,335,000 Obligation, $33,014,676 FY 2003: Appropriation, $32,125,000 Obligation, $31,563,786 FY 2004: Appropriation, $31,787,000 Obligation, $34,019,347

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Office of Justice Programs and BJS apply numerous mechanisms designed to cut costs and increase efficiency. For example, OJP is pursuing competitive sourcing activities that will deliver government services that are market-based and results oriented. OJP issued a formal public announcement on August 11, 2004 in FedBizOpps.gov. The 2nd draft solicitation was published in FedBizOpps.gov on 3/30/05. OJP is conducting an 18-month competitive sourcing study that should be completed by January 2006. OJP embraces wide-ranging business process improvements (BPI) as set forth in its Management Plan, April 2003. It has completed seven initiatives and is currently working on Customer Communication, Employee Communication, Enterprise Architecture, E-gov Compatibility, GMS Build Out, Performance Measurement, and a redesign of the GAN and the Time & Attendance processes. Representatives from all program and support offices comprise the BPI working groups. BJS exploits technology to improve operations and data collection. BJS has concretely improved the efficiency, accuracy, and completeness of statistical series by increasing use of web-based and electronic data collection. Among the collections from Federal, State, and local agencies amenable to web-based data submission, 4 out of 5 have been converted to direct, web-based transmission of data. For the remainder, BJS primarily relies on the submissions of electronic master files from which relevant data are extracted. BJS anticipates converting the largest of the personal interview sample surveys, the NCVS, to an automated format by January 2007. BJS has increasingly sought to use competitively acquired data collection agents. These agents complement the Census Bureau, the primary data collection agent for BJS programs, and require privacy and confidentiality protections and a national field interview staff. These additional data collection agents will now be assigned about half the data collection workload to be carried out during FY 2005. The BJS measures of operational efficiency combine key productivity indicators into an index that monitors performance across operational goals and tasks. BJS can determine whether results justify the cost and resource burden. BJS's efficiency goals include: (1) Reduce staffing required per collection unit; (2) Increase customer contact and data access per staff member; (3) Increase customer access to analytic facilities per staff member 4) Increase conversion from paper- based products to electronic products; and (5) Reduce cost of providing data services per user. The Index of Operational Efficiency, which gives equal weight to each of the 5 efficiency goals, indicates that since 1997 BJS has had annual increases in efficiency. Relative to the average efficiency for the period since 1996, BJS is estimated to be 30% more efficient in 2005 than it was the case in 1996. Over the period, the greatest improvement occurred in the reduced cost per user of BJS services and products - a 40% higher level of efficiency in 2004 than in 1996.

Evidence: Competitive Sourcing PWS solicitation reference number 2004PR784 (www.fedbizopps.gov) BJS Strategic Plan: 2003-2008: www.ojp.usdoj.gov/bjs/pub/pdf/bjssp08.pdf

Does the program collaborate and coordinate effectively with related programs?

Explanation: BJS coordinates with the other Federal statistical agencies through OMB's Interagency Council for Statistical Policy (ICSP), a council of Federal statistical agency heads chaired by the Chief Statistician of the U.S. BJS staff are active participants in the Federal Council on Statistical Methodology (FCSM) and frequently present and chair sessions at council meetings. BJS has been an active participant in the development of FEDSTATS - the public website and portal for Federal statistics. BJS maintains formal partnerships with other Executive branch agencies to collaborate on data projects or to provide technical expertise. BJS frequently collaborates with other agencies in conducting special data collections that benefit both agencies. BJS often performs special data collections or analyses in support of other criminal justice entities within the Department of Justice. BJS works in partnership with the FBI to build national infra- structures for criminal history records and incident-based reporting. BJS was the architect of the National Incident-Based Reporting System (NIBRS) for the FBI, resulting in improved national crime statistics from local law enforcement agencies. The BJS Director is a member of the DOJ Research and Statistics Advisory Board, comprised of agency heads from major DOJ components. Seeking to assure coordination of research, the Board helps identify information gaps. BJS seeks the expertise of the Department and its components on emerging issues for measurement. BJS maintains close working relationships with State/local government units. In addition, BJS works with selected national organizations to gather selected criminal justice statistics.

Evidence: For collaboration with OMB and other Federal statistical agencies see Statistical Programs of the U.S. Government, FY 2005 www.whitehouse.gov/OMB/inforeg/05statprog.pdf Recent work with other Federal agencies includes: a survey with the Department of Homeland Security to estimate the incidence and consequences of computer crime among businesses; a report with the Department of Education and the White House to reveal biennial estimates of school crime; a supplement to the victimization survey with the National Institute of Occupational Safety and Health to measure the incidence and prevalence of workplace violence; a study with the Department of Transportation to assess the viability and cost of obtaining race and ethnicity of the traveling public; and studies with the Centers for Disease Control and Prevention on injuries from crime and HIV and tuberculosis in prisons and jails. Recent work with DOJ components include: annual estimates of formula allocations of Federal funds (BJA, OJJDP); violent crime data from the Uniform Crime Reports (Weed and Seed); stalking victimization (OVAW); identity theft victimization (OVC); recidivism (FBI); sources of firearms used by criminals (ATF); operations, workload, and resources of State and local law enforcement agencies (COPS); manpower allocations for U.S. Attorneys' offices, Border Patrol offices, and U.S. Marshals Service field offices (DAG); forecasts of Federal parole populations (US Parole Commission); estimates of institutional rape and sexual assault (National Prison Rape Elimination Commission); and development of data on airport use to evaluate racial profiling activities in U.S. airports (DEA).

Does the program use strong financial management practices?

Explanation: Auditors were unable to express an opinion on the Department's 2004 financial statements because of an auditor's disclaimer on the Office of Justice Programs' 2004 financial statements. The disclaimer was the result of a problem with grant accruals, or the estimates of unspent funds. These estimates can be prepared in a variety of ways depending upon the assumptions made about to grantee spending behavior--- the auditors required OJP to devise a new procedure for producing such estimates. OJP is working with auditors to resolve the issue in time for the 2005 financial statements. Conversations with DOJ and OIG staff indicate, barring an unforeseen auditor issue, that OJP is on track to resolve this issue. As BJS' parent bureau, a disclaimer for the Office of Justice Programs under other circumstances might warrant questions about BJS' financial practices. However, the grant accrual issue is largely unrelated to BJS' line of business. Grant funds awarded by BJS for statistical activities are less than ?? of 1% (0.3% in FY 2004) of the total grant funds awarded by OJP. To control Federal funds recipients, BJS only utilizes cooperative agreements for the acquisition of data by data collection agents. The cooperative agreement, in contrast to the standard grant, allows much tighter scrutiny over the tasks and activities undertaken by the recipient. In addition, BJS typically adds special conditions to the cooperative agreements that enhance the level of control and scrutiny of the awardee by BJS staff. Program managers in BJS coordinate with the Office of the Comptroller on administrative and fiscal monitoring.

Evidence: DOJ 2004 Performance and Accountability Report, Independent Accountants' Report on Financial Statements, www.usdoj.gov/ag/annualreports/pr2004/P3/p009-10.pdf Conference call with DOJ financial management and OIG officials and staff, June 16, 2005.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: OJP and BJS have established specific procedures and policies to ensure good management controls. The goal is to reasonably ensure that (1) programs achieve their intended results; (2) resources are used consistent with agency mission; (3) programs and resources are protected from waste, fraud, and mismanagement; (4) laws and regulations are followed; and (5) reliable and timely information is obtained, maintained, reported and used for decision making and operational improvements. BJS managers continuously monitor and assess controls (identified as a specific task in annual performance appraisal) using various sources, including knowledge gained from the daily operation of agency programs and systems, IG and GAO reports, program evaluations, audits and monitoring reports of funding recipient programmatic and financial activities, financial reports generated by the Office of the Comptroller and Office of Budget and Management Services, and annual performance plans.

Evidence: Managers and employees identify deficiencies in management controls from the sources of information described above. Employees and managers report deficiencies to the next supervisory level, allowing the chain of command structure to determine the relative importance of and appropriate responses for each deficiency.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: With sustained incremental improvement, as seen in BJS' recent performance, BJS is on track to achieve its long-term performance goals. In recent years, BJS has made notable increases in efficiency, as demonstrated in its long-term efficiency index score. BJS also has exceeded long-term goals involving the quantity of data collection series and special analyses, as well as demand for its data as evidenced by internet usage and product downloading.

Evidence: See Performance Measures Section of the PART.

Does the program (including program partners) achieve its annual performance goals?

Explanation: BJS has established baselines and targets for its annual performance measures, and has achieved and exceeded its targets for most of its annual performance goals for 2004.

Evidence: See Performance Measure Section of the PART.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: BJS has shown efficiency gains throughout its operations. For example, BJS has reduced the number of staff needed relative to the number of statistical series conducted. In 1997, the peak year, BJS employed 1.89 staff per statistical series. In 2003--the figure was 1.29. In 2004, BJS utilized 1.06 staff per statistical program, an 18 percent improvement in staffing efficiency over 2003. Each year since 1996, BJS has improved the efficiency of the provision of data to the public. In 1996, there were 4,609 contacts per staff member attempting to obtain BJS data. In 2003, there were 58,587 contacts per staff member. In 2004, there were 67,876 contacts per staff member. This represents a 16 percent increase in public access to BJS data relative to staffing over the prior year. Since 1996, there has been a dramatic improvement in the ratio of electronic access to statistics versus printed reports. In 1996, 79 of every 100 users of BJS information received a printed report and the remainder (21 percent) received their information via an electronic download. In 2003, 83 percent used electronic access. In 2004, fewer than 8% of customers received a paper document and 92% utilized electronic access to BJS statistical reports and data. In 2000, BJS paid $207,261 for the printing of publications, compared to $70,962 in 2004. In 2000, BJS paid $460,000 for metered mail; in 2004, BJS mail costs were just over $61,000. BJS has been able to provide more services to more customers at a much-reduced cost per customer, a fundamental indicator of increased efficiency and cost effectiveness. Relative to appropriated funds, in 1996 the services provided to each customer (data collection, analysis, reporting, dissemination, archiving, etc.) using constant dollars translated into a cost of $23.76. In 2004, BJS appropriated funds represented a cost of $8.27 per customer--about a two-thirds reduction in the cost per user. BJS also has undertaken additional steps that will improve the efficiency of program administration. For example, BJS is now designing and using web-based data collection instruments and protocols to reduce the cost of manual keying of data and to avert the much higher error rate associated with manual data collection. Virtually all BJS censuses and surveys that obtain administrative data from State and local criminal justice agencies now provide a secure web option for responding. BJS and Census Bureau have been jointly working toward the elimination of paper forms from personal interview surveys, such as the NCVS, with direct, daily uploads of interview data from field staff and BJS has doubled the number of interviews conducted telephonically as opposed to in-person. Another cost-saving effort involves the selection of data collection agents. For many data collections, BJS has moved away from the Census Bureau to conducting competitive selection of data collection agents. Recent experience with the law enforcement collection program indicates that there are vendors who can offer quality and service equal to that of the Census Bureau but at a lower cost. For example, the 2000 Census of Law Enforcement Agencies, conducted by the US Census Bureau cost $925,538. In 2004, BJS competitively awarded the National Organization for Research to conduct the 2004 Census for $638,670.

Evidence: See efficiency index score in Performance Measures section of the PART. Other data submitted by BJS to OMB.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The extent to which a federal statistical agency's data products generate demand can be a useful indicator of performance relative to other federal statistical agencies. FEDSTATS.gov is a portal through which the statistics of over 100 U.S. agencies and programs can be accessed. The most recent FEDSTATS user referral data indicate that BJS is the 4th leading referral agency, well ahead of much larger Federal statistical agencies such as the National Center for Education Statistics, the Energy Information Administration, the National Agriculture Statistics Service, and the Bureau of Transportation Statistics. Non-response and refusal rates are commonly measured elements of statistical survey program operation across federal statistics agencies. BJS compares well with comparable programs for these measures. The National Crime Victimization Survey, the largest single data collection activity at BJS--accounting for nearly 70% of appropriated BJS funds, has maintained an aggregate refusal/non-response rate of 10%-13% over the last five years. According to the Census Bureau, aggregate refusal/non-response rates for similar-sized surveys over the last five years are: Consumer Expenditures Survey -39%-42%; Current Population Survey -10%-11%; National Health Interview Survey -16%-17%; and the American Housing Survey - 13%-17%.

Evidence: FEDSTATS web server log: Number of referrals to agency websites; U.S. Census Bureau Report: Non-interview Rates for Selected Major Demographic Household Surveys.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: BJS would benefit from a comprehensive review that could provide specific evidence of BJS' impact overall. Major reviews of BJS statistical activities conducted by the Census Bureau, the American Statistical Association, the National Academy of Sciences, and other external groups, have concluded that BJS adheres to standards of quality and practice that are consistent with the expectations for a national statistics agency. Such reviews are important for providing feedback and confirming success at an operational level. However, they do not provide any information on BJS' ultimate impact, which focuses on the production of national crime and justice statistics. Is BJS collecting the right kinds of statistics to meet the nation's needs? Are changes needed in the nation's system for collecting and producing crime and justice statistics? Would the nation be better served by an alternative organization (e.g., a consolidated statistical agency) for producing crime and justice statistics?--These are the kinds of questions a more comprehensive review could address.

Evidence: NAS Charter to Review BJS Operations and Statistical Data: www7.nationalacademies.org/cnstat/About_CNSTAT.html Firearms Inquiry Statistical Program Firearms and Violence: A Critical Review, Charles F. Wellford, John V. Pepper, and Carol V. Petrie, Editors, Committee on Law and Justice, National Research Council, The National Academies Press, 2004 books.nap.edu/openbook.php?record_id=10881&page=93#pagetop National Crime Victimization Survey Measurement Problems in Criminal Justice Research: Workshop Summary, John V. Pepper and Carol V. Petrie, Editors, Committee on Law and Justice, National Research Council, The National Academies Press, 2003 www.nap.edu/openbook.php?record_id=10581&page=11#pagetop Surveys of Prison and Jail Inmates Informing America's Policy on Illegal Drugs: What We Don't Know Keeps Hurting Us, Committee on Data and Research for Policy on Illegal Drugs, Charles F. Manski, John V. Pepper, and Carol V. Petrie, Editors, Committee on Law and Justice and Committee on National Statistics, National Research Council, The National Academies Press, 2001 books.nap.edu/openbook/0309072735/html/92.html National Judicial Reporting Program What's Changing in Prosecution?: Report of a Workshop, Philip Heymann and Carol Petrie, Editors, Committee on Law and Justice, National Research Council, The National Academies Press, 2001 lab.nap.edu/openbook.php?record_id=10114&page=23#pagetop Crime Victims with Disabilities Supplement Crime Victims with Developmental Disabilities: Report of a Workshop, Joan Petersilia, Joseph Foote, and Nancy Crowell, Editors, Committee on Law and Justice, National Research Council, The National Academies Press, 2001 books.nap.edu/openbook.php?record_id=10042&page=39#pagetop