Detailed Information on the
Health Care Patient Safety Assessment

Measure the number of Patient Safety Organizations (PSO) that become certified based on Patient Safety and Quality Improvement Act legislation.

Update performance targets with available data for FY 2007.

Develop a written summary of the current available data sources that could be used to measure the safety of health care in the U.S., and if none exist, prepare a high-level analysis of the feasibility of developing such a new data source (including a rough estimate of the cost and time required).

Analzye data from Patient Safety hospital survey and establish baseline for the percent of hospitals reporting on adverse events as standard practice.

Develop annual targets for long-term and annual measures for out-years.

2007 - Establish annual targets around the Patient Safety and Quality Improvement Act.

Monitor AHRQ's progress toward developing baselines for newly developed long-term and annual performance goals.

Measure: Increase the percentage of hospitals in the U.S. using computer-based patient safety event reporting systems.

Explanation:The use of computer-based patient safety event reporting -- as opposed to paper-based reporting systems, which are already pervasive -- is likely to lead to more standardized and timely information collection, thus increasing an organization's ability to capture and analyze data, identify potential interventions, and implement a successful solution to prevent those types of patient safety events in the future. Baseline data is from a RAND/Joint Commission survey presented to AHRQ in 2006, and subsequent years' data will come from a previously-developed survey which will be modified and used by RAND to survey a nationally-representative sample of hospitals to capture changes in the use of computer-based patient safety event reporting systems. This will be particularly helpful as the data will be captured before the launch of the regulations creating Patient Safety Organizations (PSOs) and the certification of the first PSOs -- creating an accurate baseline for the PSO effort.

Measure: Increase the number of patient safety events (e.g. medical errors) reported to the Network of Patient Safety Databases.

Explanation:A patient safety event is an incident that occurred during the delivery of a health care service that harmed, or could have harmed, a patient. The Network of Patient Safety Databases (NPSD) will contain voluntarily contributed non-identifiable information about patient safety events reported by Patient Safety Organizations (PSOs). The number of patient safety events reported to the NPSD is expected to increase over time, as more health care providers contract to report medical errors to PSOs for analysis, and more data is reported by PSOs to the NPSD. As more "near misses" and medical errors are reported to this database, researchers will have a richer data source to analyze to identify the causes of medical errors. As more causes of medical errors are identified, health care providers will have more information about how to modify organizational and clinical practices to avoid future medical errors - leading to safer patient care. Eventually, this number is expected to decline as medical errors are analyzed, trends in errors are isolated, best practices to combat errors are disseminated, and the frequency of medical errors starts to decrease.

Measure: Increase the number of U.S. healthcare organizations per year using AHRQ-supported tools to improve patient safety.

Explanation:AHRQ-supported tools are resources developed through grant or contract funding provided by AHRQ, in whole or in part. Examples of tools include: the TeamSTEPPS?? curriculum, which encourages health care providers to use a team-based approach; the Hospital Survey on Patient Safety Culture assessment tool, which hospitals can use to gauge their patient safety culture; and toolkits or implementation guides evolving from partnerships with other organizations, such as one aimed at preventing venous thromboembolisms in the hospital, and another called "Door to Doc" aimed at decreasing ER wait times by changing workflow process. The use of AHRQ-supported tools is expected to increase health care providers' knowledge and understanding of patient safety, promote a culture of patient safety, and improve the use of patient safety best practices by these providers. The program anticipates gathering this information systematically through an evaluation contract that is in the process of being awarded. In the meantime, it is using several existing projects to capture some of this information, noting that it is not based on a systematic, nationally-representative sample.

Measure: Reduce the cost per capita of treating hospital-acquired infections per year.

Explanation:As research findings, checklists, surveys, and implementation guides are produced and disseminated by the Patient Safety research portfolio and adopted into clinical and organizational practice, the number of hospital-acquired infections is expected to decrease, resulting in a lower cost per capita of treating accidental infections associated with the delivery of health care. This program's goal is to reduce infections due to medical care in non-immune compromised adult medical or surgical patients by 2% per year. AHRQ has funded several projects with the goals of eliminating hospital-acquired infections in general or related to a specific type of health care service, device or setting (e.g. AHRQ's Accelerating Change and Transformation in Organizations and Networks (ACTION) research projects, which partner researchers with health care providers interested in implementing findings). Data for this measure are calculated by AHRQ, which produces national annual estimates for patient safety indicators using data from AHRQ's Healthcare Cost and Utilization Project (HCUP) Nationwide Inpatient Sample (NIS). Through a voluntary partnership, HCUP collects hospital data from state data organizations that collect summary records for all hospitals in their state. HCUP processes the disparate state data into a uniform format for national analyses. Given the time lag for some partner states, the full year HCUP NIS can only be produced as fast as the slowest state reports its data. Currently, the NIS is released approximately 18 months following the close of the data year because some states submit their data over a year after the close of the data year (e.g., Texas 2006 data arrived in February 2008).

Is the program purpose clear?

Explanation: The Agency for Healthcare Research and Quality's (AHRQ) reauthorization directs AHRQ to "conduct and support research and build private-public partnerships to: 1) Identify the causes of preventable health care errors and patient injury in health care delivery; 2) Develop, demonstrate, and evaluate strategies for reducing errors and improving patient safety; and 3) Disseminate such effective strategies throughout the health care industry." In Appropriations Reports, Congress specifies the expected set-aside for AHRQ-funded patient safety (PS) activities. AHRQ has summarized its statutory authority by establishing as its mission to identify, understand and reduce the risk of harm associated with medical errors and health care system-related problems. To achieve this mission, AHRQ's PS research portfolio has four focuses: 1) Identify threats to PS, 2) Identify and evaluate effective PS practices, 3) educate practitioners, disseminate information and implement practices that will enhance PS, and 4) Monitor and evaluate threats to PS.

Evidence: 1) Healthcare Research and Quality Act (P.L. 106-129) - Title IX of the Public Health Service Act (www.ahrq.gov/hrqa99.pdf) AHRQ RFAs are available at grants.nih.gov/grants/rfa-files/.... 2) April 2001 - PS Research Dissemination & Education (/RFA-HS-01-008.html) 3) February 2001 - Improving PS Demos (/RFA-HS-01-003.html) 4) November 2000 - Developmental Centers for Evaluation & Research in PS (/RFA-HS-01-007.html) 5) February 2001 - Clinical Informatics to Promote PS (/RFA-HS-01-006.html) 6) April 2003 - Safe Practices Implementation Challenge Grants (/RFA-HS-03-005.html) 7) December 1999 - Systems Related Best Practices to Improve PS (/RFA-HS-00-007.html)

Does the program address a specific and existing problem, interest, or need?

Explanation: The occurrence of medical errors in hospital settings is not a new phenomenon. AHRQ, FDA, CDC, NCHS, CMS, and other Federal agencies funded PS activities prior to the AHRQ-funded study that lead to the November 1999 Institute of Medicine report, To Err is Human. This report concluded that between 44,000-98,000 Americans die each year due to medical errors, the majority of which were identified as systemic problems rather than poor performance by individual providers. The PS Initiative was established in FY 2001 and focuses on reducing the risk of injury and harm associated with medical errors and establishing and emerging IT that improve PS and quality of care. Since the IOM report other studies have estimated the number of errors to be higher and lower than those estimated by the IOM. Reporting is currently not mandatory, hospital charts are sometimes incomplete, and no entity has a system in place to collect uniform data on these errors. An actual number is unknown.

Evidence: 1) To Err is Human, Institute of Medicine 1999 The report noted that more individuals die each year from adverse events in the delivery of health care than from the combined number of deaths from automobile accidents (43,458) and workplace injuries (6,000).

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: Duplication exists; however, in its role on the Quality Interagency Coordination (QuIC) TF AHRQ coordinates and in some cases leads the research component of PS activities across government. The QuIC helps avoid duplication/cost inefficiencies and provides a forum for coordinating PS/quality care. AHRQ focuses on how/why medical errors occur; disseminates findings; and creates comprehensive, national solutions to mitigate/eliminate harm in all health care (HC) settings. HHS agencies fund complementary/overlapping activities. FDA focuses on manufacturers' mandatory reporting of adverse events involving medication errors, drug/therapeutic biological products and medical devices, and voluntary/confidential reporting of medication errors by HC practitioners and consumers. CDC maintains voluntary reporting of hospital-associated infections in acute care settings and adverse events associated with vaccination. NCHS collects data on avoidable hospitalizations and complications, and adverse events. CMS' national network of 53 Quality Improvement Organizations works with consumers, physicians, hospitals, and other caregivers to refine delivery systems to ensure patients receive proper care at the right time, particularly among underserved populations.

Evidence: 1) Patient Safety Reporting Systems and Research in HHS www.ahrq.gov/qual/taskforce/hhsrepor.htm 2) Quality Interagency Coordination Task Force www.ahrq.gov/qual/quicix.htm Note: Other Federal agencies and the private sector also fund complementary/overlapping activities. Also, DOD and VA are direct care providers that identify where/why errors occur in their respective settings. Private sector projects are consumer/practice/data system-focused rather than comprehensive.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The AHRQ PS portfolio is newly funded and research is conducted in stages. Now that best practices and lessons learned are becoming available AHRQ is moving toward taking lessons learned and implementing successful protocols to improve patient safety in their respective settings. AHRQ sees the need for such "hooks". In some, but not all, of its RFAs AHRQ "expects the funded organizations to have or develop a plan for sustaining the reporting system and all its component parts once the grant expires." In addition, it notifies the applicant that "AHRQ, at some point in the future, may begin requesting information essential to an assessment of the effectiveness of Agency research programs. Accordingly, grant recipients...may be contacted after the completion of awards for periodic updates on publications resulting from AHRQ grant awards, and other information helpful in evaluating the impact of sponsored research."

Evidence: AHRQ RFAs are available at http://grants.nih.gov/grants/rfa-files ... 1) February 2001 - Improving PS Demos (/RFA-HS-01-003.html) 2) February 2001 - Clinical Informatics to Promote PS (/RFA-HS-01-006.html) 3) April 2003 - Safe Practices Implementation Challenge Grants (/RFA-HS-03-005.html)

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: AHRQ's PS portfolio has research at its foundation. Through a variety of funding mechanisms (e.g. research demonstration and other grants, contracts, interagency agreements, and cooperative agreements) AHRQ makes awards to domestic, public and private non-profit organizations, including professional societies and associations, educational leadership organizations, provider organizations, health care delivery organizations, health plans, State and local governments, and eligible Federal agencies. These groups are most likely to be positioned to implement findings identified in AHRQ-funded research that could help improve patient safety. As a result, these entities' research efforts are targeted to the intended patient population or beneficiaries of safer patient care. In addition, applications that are complete and responsive to an RFA are evaluated for scientific and technical merit by an appropriate peer review group convened by AHRQ in accordance with the review criteria stated in the RFA.

Evidence:  

Does the program effectively articulate potential public benefits?

Explanation:  

Evidence:  

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation:  

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: OMB and AHRQ recently developed two long-term goals that link to the mission of the program.

Evidence: 1) FY 2005 GPRA Plan 2) See "Measures" tab for the long-term goals

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: When developing these long-term goals, specific attention was paid to highlighting baseline data and ensuring ambitious targets.

Evidence:  

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: OMB and AHRQ recently developed two annual output goals that demonstrate progress toward achieving the long-term goals for patient safety activities.

Evidence: 1) FY 2005 GPRA Plan 2) See "Measures" tab for the annual goals

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: When developing these annual goals, specific attention was paid to highlighting baseline data and ensuring ambitious targets.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The long-term and annual program goals themselves are not included in RFAs, contracts, cooperative agreements, or interagency agreements. However, AHRQ attempts to hold all parties accountable by specifying in RFAs a condensed and all encompassing goal, which is to "accelerate the implementation by local health care organizations of evidence based 'safe practices' that eliminate identified hazards and/or reduce risk of harm to patients". Project Officers measure progress toward this goal as they perform their annual site visits with each grantee. PS contract goals are negotiated with the contractor as part of their performance-based contract plans. Contractors are required to commit to milestones contributing to those performance goals and file reports by phone weekly, and written monthly and annual reports. If progress is judged as insufficient agreements may be terminated.

Evidence: 1) September 2002 - RAND Contract for Patient Safety Program Evaluation Center Contract 2) September 2001 - WESTAT Patient Safety Research Coordinating Center Contract

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: To independently evaluate the impact of the PS Initiative, AHRQ has a separate PS Program Evaluation Center through a multiyear contract with RAND, which began in September 2002. The objective of this contract is to establish a Center that shall 1) develop an implement an overall evaluation plan, 2) develop baseline PS evaluation measures, 3) utilize formative evaluation procedures, monitor progress, and make recommendations for improvement, 4) assess initiative impacts, outcomes, and adopt diffusion using both qualitative and quantitative assessment, and 5) document and prepare evaluation reports indicating results. The first major evaluation report is due from RAND at the end of September 2003, one year from the signing of the contract.

Evidence: September 2002 - RAND Contract for Patient Safety Program Evaluation Center Contract

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: AHRQ's OMB budget justification and Congressional justification display the AHRQ budget request. However, when AHRQ submits its budget request to the Department for review, the annual targets are adjusted according to the funding level requested and/or the final funding level passed back from the Department. Budget requests and funding level decisions are not made based on achieving the established long-term and annual performance goals. In addition, AHRQ does not have in place a model/mechanism that allows it to determine per unit cost of service to help in adjusting its budget or program targets accordingly.

Evidence: 1) OMB Budget Justification submitted each Fall 2) Congressional Justification submitted each February with the President's Budget

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: AHRQ has acknowledged the multiple difficulties of tracking budgetary expenditures along with tying these expenditures to actual program performance. AHRQ plans, using budgeted FY 2003 resources, to begin to deploy a reporting module (phase I) to the activity areas allowing them to view and track their own budgets. Phase II will allow the activity areas to interconnect appropriate areas of the AHRQ's planning system with the budget system through a set of common fields, and finally, the GPRA program goals. The ultimate goal of this project will be targeted integration of the existing AHRQ planning database with the budget database system, allowing AHRQ's leadership to easily identify, and flag for action those program areas that are not meeting their GPRA goals.

Evidence:  

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: AHRQ often reviews the intent of its program relative to the activities funded by other agencies. To this effect, AHRQ often fills the niche by partnering with other agencies to ensure that there is synergy across efforts. AHRQ is partnering with VA in developing the PS Improvement Corps. With DOD, AHRQ is helping to evaluate their training programs. Both efforts will help AHRQ to develop patient safety officers who will know how to work in cooperation with others in the field on this topic. In addition, AHRQ is working with FDA and CDC to bring together their databases such that there is communication across them.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: AHRQ uses for its own internal program management a ten year plan that has as its strategy to evaluate the context of medical errors and input evaluation data in a common report (FY 2001), evaluate the process for collecting and reporting common data (FYs 2002-2003), evaluate the products that exist to improve patient safety (FY 2004), and adopt those methods that have proven successful (FYs 2005-10).

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AHRQ is now requiring grantees to report quarterly on their progress and attend annual meetings where they submit progress reports describing their implementation activities, lessons learned, and preliminary findings. AHRQ has taken steps to withhold funding from grantees whose performance is unsatisfactory. Six months after the project was awarded the principal investigator/primary architect abruptly resigned, taking with him key personnel and university collaborators.

Evidence: 1) Work plan tasks and subtasks 2) Grantee progress reports 3) Grantee financial status reports

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: AHRQ's strategic plan guides the overall management of the agency. Each Office and Center has an individual strategic plan and annual operating plan. Cost, schedule and performance are part of the performance plans, including Division, Center, and Agency Directors. The annual operating plan identifies those things that contribute to AHRQ achieving its performance goals and internal management goals. These factors are incorporated into each employee's annual performance plan/review. At the end of each year, the Office and Center Directors review accomplishments in relation to the annual operating plans in preparation for drafting the next year's plans. The results of these reviews contribute significantly to Office and Center performance reports. Some managers' performance plans also take into consideration their staffs performance in managing program operation. In addition, contracts are performance-based.

Evidence: Program managers performance contract

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All appropriated funds are obligated in accordance with the annual operating plans, formulated for obligation and outlay on a quarterly basis.

Evidence: 1) Estimated obligations by quarter in apportionments for FYs 2001-2003 2) Actual obligations by quarter for FYs 2001-2003

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: AHRQ bids out its contracts to organizations with expertise in the area to ensure cost efficiencies and effective use of Federal resources. Contracts are cost plus fixed fee. In addition, AHRQ has managed a growing number of PS grants with minimal increases in staff to support this function; this too has lead to efficiencies. 84 grants were processed in FY 2002 at 5.5 man hours each up from 60 grants and 5.0 man hours each in FY 2001.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: AHRQ often reviews the intent of its program relative to the activities funded by other agencies. To this effect, AHRQ often fills the niche by partnering with other agencies so that there is synergy across efforts. AHRQ is partnering with VA in developing the PS Improvement Corps. With DOD, AHRQ is helping them to evaluate their training programs. Both efforts will help AHRQ to develop patient safety officers who will know how to work in cooperation with others in the field. In addition, AHRQ is working with FDA and CDC to bring together their databases to ensure communication across the databases.

Evidence:  

Does the program use strong financial management practices?

Explanation: The Department prepares audited financial statements for its largest components only, AHRQ's financial statements are not audited.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Department prepares audited financial statements for its largest components only; therefore AHRQ has not been audited in the past. However, seeing the need for outside assessment of its financial statements, AHRQ engaged Clifton Gunderson LLP for technical support consultation and analysis of certain financial management practices.

Evidence:  

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: AHRQ announces research grant opportunities through program announcements and requests for applications. Contract opportunities are announced through a similar process. Grant applications are reviewed for scientific and technical merit by a peer review group with appropriate expertise. Funding decisions are based on the quality of the proposed project, availability of funds, and program balance among research areas. Contracts are awarded using a similar process.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Every PS awardee provides progress reports to AHRQ Program Officers on a regular basis. This information includes: 1) a brief narrative on what was actually accomplished during the reporting period and a summation of the cost and level of effort expended for each task, 2) preliminary or interim results and conclusions, 3) problems or delays the awardee has experienced in the conduct of performance requirements including what specific action is proposed to alleviate the problems, 4) adjustments that are being implemented to study plans, and 5) planed activities during the next reporting period.

Evidence:  

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: AHRQ collects performance data, but has a unique method for making this data available to the public. AHRQ has published and physicians provide in doctors' offices 20 tips for consumers to help prevent medical errors and steps to safer health care. In addition, many organizations including the 16 demonstration projects, participating in reporting systems establish special PS committees made up of physicians, nurses, pharmacists, and other health care providers to examine medical error reports and identify actions to implement safe procedures and share strategies. The spread of information expands out from these committees. Also, some of the PS best practices identified in an AHRQ-funded report have been identified by JCAHO and incorporated into their guidance for practitioners. Other information regarding morbidity and mortality cases and medical errors.

Evidence: 1) www.ahrq.gov/consumer/20tips.htm 2) www.ahrq.gov/consumer/20tipkid.htm 3) www.ahrq.gov/consumer/5steps.htm 4) www.ahrq.gov/consumer/5tipseng/5tip.htm 5) www.webmm.ahrq.gov/

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: AHRQ's grant awards may be the result of investigator-initiated ideas or in response to program announcements, request for applications, or request for proposals, all of which are peer-reviewed. The peer review process takes into consideration previous experience, a definitive plan for the recruitment of diverse populations, and plans to ensure community involvement in the planning and design process. All research grants are awarded for a specified period of time, at the end of which they must re-compete for additional resources.

Evidence:  

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation:  

Evidence:  

Does the program adequately define appropriate termination points and other decision points?

Explanation:  

Evidence:  

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: Prior to this year, AHRQ has spent much of its time building the foundation. Progress on the long-term goal is expected to become quantifiable as of FY 2005-06.

Evidence: See "Measures" tab for the long-term goals.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The AHRQ PS Initiative began in FY 2001. Since this time, AHRQ has worked consistently toward achieving its annual output goals of granting awards, establishing the knowledge base, identifying best practices, initiating demonstration projects, and developing a reporting mechanism and data structure through the National Patient Safety network. All of this is the foundation for building up to the long-term national vision of improving patient safety.

Evidence: See "Measures" tab for the annual goals.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: AHRQ has managed a growing number of PS grants with minimal increases in staff to support this function; this too has lead to efficiencies. 84 grants were processed in FY 2002 at x.x man hours each up from 60 grants and x.x man hours each in FY 2001.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: AHRQ is the only Federal agency with a comprehensive purpose of identifying threats to PS; identifying and evaluating effective practices; educating practitioners, disseminating information and implementing practices to enhance PS; and monitoring and evaluating threats to PS. AHRQ seeks to fund demonstration projects and research efforts that can be generalized to provide national level data on possible technologies and health setting protocols that may improve patient safety.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: AHRQ's PS portfolio is new and many initial awards are in their final stage of funding. An evaluation of effectiveness could not be completed until these awards are finalized. The first major evaluation report is due from RAND at the end of September 2003.

Evidence:  

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation:  

Evidence: