Detailed Information on the
Health Information Technology Research (AHRQ) Assessment

Develop an efficiency measure that demonstrates program managers' efforts to improve economy in the acquisition and utilization of resources.

Improve the program's website by developing and publishing one "how to" guide per quarter.

Gain feedback on how to improve the program's website by conducting focus groups of program stakeholders and summarizing the results.

Develop plan for posting individual and aggregated grantee performance on the program's website on an annual basis.

Produce content development plan for improving the program's website by addressing the needs of the program's stakeholders based on feedback from evaluations, focus groups, and other sources.

Post individual and annually aggregated grantee performance on the program's website on an annual basis.

Develop long-term performance measure that gauge the program's success in developing and disseminating research on how health IT can improve the quality of health care.

Develop annual performance measures that gauge the program's success in developing and disseminating research on how health IT can improve the quality of health care.

Develop an efficiency measure that demonstrates program managers' efforts to improve economy in the acquisition and utilization of resources.

Is the program purpose clear?

Explanation: The purpose of the Health Information Technology research program at the Agency for Healthcare Research and Quality (henceforward, "AHRQ HIT program") is to develop and disseminate evidence about how health information technology ("health IT") can be used to improve the quality of health care. This purpose has evolved since the program's activities were first authorized in 1999 and since the program was established in 2004 -- when activities focused on advancing the adoption of health information technology (health IT). As a reflection of this shift, the program's website identifies program research activities related to both advancing the adoption of health IT and researching how such tools can improve the quality of health care. The three main research topics that the AHRQ HIT program currently focuses on are identifying improvements in quality of health care resulting from use of: 1) clinical decision support tools, which provide reminders to health care providers to give guideline-based care; 2) the exchanging of health information electronically, to facilitate care management between multiple health care providers; and 3) electronic prescribing systems, to increase the accuracy of medication management.

Evidence: AHRQ's FY 2009 Justification of Estimates for Appropriations Committees [http://www.ahrq.gov/about/cj2009/cj2009.pdf] describes the AHRQ HIT program's purpose as: "to develop evidence and inform policy and practice on how Health IT can improve the quality of American healthcare." As part of the 1999 reauthorizing legislation for the Agency for Healthcare Research and Quality (AHRQ), in a section titled "Part B - Health Care Improvement Research," Congress directed AHRQ to conduct research to advance the development of health information exchange networks, the use of computerized clinical decision support tools, the use of electronic health records, and other related activities (see Sec. 914 of the Healthcare Research and Quality Act of 1999). The AHRQ HIT program's website states that its mission is "to support and stimulate investment in health IT" [http://healthit.ahrq.gov, click "About" in the left menu] but also lists as one of its goals: "Help clinicians develop higher-quality, safer health care" [http://healthit.ahrq.gov, click "About" in the left menu, then "AHRQ and Health IT"].

Does the program address a specific and existing problem, interest, or need?

Explanation: The AHRQ HIT program addresses the lack of evidence on how to successfully implement and use health IT to improve the quality of health care. The Institute of Medicine and other health care stakeholders have expressed the need for better evidence on the potential benefits of, and best practices for deploying, health IT tools and systems, as a necessary precursor for wider health IT adoption. The AHRQ HIT program addresses these needs by funding and disseminating research on how to successfully implement health IT tools and systems, and how to use them to improve the quality of health care.

Evidence: The need for better evidence on the potential benefits of health IT is articulated in a 2000 report by the Institute of Medicine, "Crossing the Quality Chasm," Chapter 7.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The AHRQ HIT program is not duplicative of any other Federal program, though it does overlap slightly with some private foundations' efforts. The AHRQ HIT program differs from HHS' Office of the National Coordinator for Health Information Technology (ONC), which provides leadership for the development and nationwide implementation of an interoperable health information technology infrastructure to improve the quality and efficiency of health care and the ability of consumers to manage their care and safety. Although the AHRQ HIT program and ONC both have an ultimate goal of improving the quality of health care, the AHRQ HIT program explicitly researches how health IT tools can improve the quality of health care, while ONC focuses on advancing the adoption of health IT, which it assumes will then result in improved quality of health care. ONC's activities aimed at increasing adoption include developing health IT interoperability standards, assuring appropriate privacy and security protections, developing the technical specifications for electronic health information exchange, and convening public and private stakeholders. ONC does not focus primarily on research, but when it does, ONC primarily funds demonstrations of electronic health information exchange, the findings of which help inform development of the National Health Information Network. The first such demonstration funded four consortia of IT vendors and local groups to implement and demonstrate interoperable electronic exchange of health information and report back with descriptions of functional requirements of a network between health care providers, security models, areas for needed standards, an overall architecture, and business models. Although the AHRQ HIT program does not overlap with ONC's activities, the program's activities do overlap slightly with private foundations which have funded some modest-sized research activities aimed at identifying how health IT can improve the quality of health care. For instance, the Robert Wood Johnson Foundation funded a study that found that Michigan hospitals that had computerized information systems in their intensive care units that were more automated and user-friendly reported fewer catheter-related infections in a patient's bloodstream. The overlap of the AHRQ HIT program with other organizations' activities was noted in a recent independent program evaluation, which stated that other organizations are producing tools that could be helpful to the AHRQ HIT website's user groups and that, for this reason, the AHRQ HIT program should work to ensure that it does not develop duplicative tools.

Evidence: The ONC research described above is detailed more fully in a report summarizing these four projects [http://www.hhs.gov/healthit/healthnetwork/resources/summary_report_on_nhin_Prototype_architectures.pdf]. ONC's mission is described in its "Statement of Organization, Functions, and Delegations of Authority" at [http://a257.g.akamaitech.net/7/257/2422/01jan20051800/edocket.access.gpo.gov/2005/pdf/05-16446.pdf], and its activities are described in the National Coordinator for Health Information Technology's FY 2009 Justification of Estimates for Appropriations Committees [http://www.hhs.gov/budget/09budget/oncfy09opa.pdf]. Some private foundations funding health IT research and/or demonstrations include the Commonwealth Fund [http://www.commonwealthfund.org/General/General_show.htm?doc_id=289934], the Markle Foundation [http://www.markle.org/markle_programs/healthcare/projects/index.php], the Robert Wood Johnson Foundation [http://www.rwjf.org/pr/topic.jsp?topicid=1158], and the California Health Care Foundation [http://www.chcf.org/topics/index.cfm?topic=CL108]. The Michigan study funded by the Robert Wood Johnson Foundation is described on the Foundation's website [http://www.rwjf.org/pr/product.jsp?id=19183&topicid=1158] and in an academic journal [http://www.jamia.org/cgi/content/abstract/14/3/288].

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is not evidence that another program design would more effectively or efficiently produce research findings about how health IT can improve the quality of health care. The AHRQ HIT program pursues this purpose by funding competitively-awarded research grants and demonstration contracts in response to solicitations for proposals, and then disseminating the findings of this research on their website. (The program also devotes a modest amount of funds to providing technical assistance to its grantees, free training webinars for the public, and convening meetings of health care stakeholders.) Other entities that produce research findings about how health IT can improve the quality of health care, such as private foundations, generally use a similar approach. However, one of these foundations has recently shifted away from awarding grants for research, and now operates most of its own programs directly. The foundation's website explains: "We have found that the most effective way for us to leverage our resources is to structure and operate our own projects in cooperation with our partners instead of working as a traditional grantmaking organization." A comparison of the effectiveness of the Markle Foundation's new approach and the AHRQ HIT program's design has not been conducted.

Evidence: The websites for private foundations that award health IT grants and contracts indicate that they generally use a similar application and dissemination approach as the AHRQ HIT program [http://www.commonwealthfund.org/resources/resources_show.htm?doc_id=317121; http://www.rwjf.org/applications/; http://www.chcf.org/grantinfo/]. The Markle Foundation's new funding approach is described at [http://www.markle.org/about_markle/faqs/index.php].

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The AHRQ HIT program targets its resources toward activities consistent with its purpose, which is to develop and disseminate research findings about how health IT can improve the quality of health care. The AHRQ HIT program disseminates research findings produced by its grantees and contractors through its public website, conferences, and webinars. The intended beneficiaries of the program's research findings and resources are health care stakeholders who are interested in using health IT to improve the quality of health care, including: health care providers; other government agencies; legislators and other policymakers; health care researchers; health IT vendors; and health IT consultants. A recent program evaluation discussed the AHRQ HIT program's mixed success in reaching these beneficiaries, noting that outside of the grantee world, relatively few people they interviewed knew about the website or its tools and those that did recommended that the website provide more 'how to' tools designed to assist projects at all stages. The evaluation also found that one key stakeholder group -- health care provider associations active in health IT issues -- was relatively unaware of the website, and once made aware of the website, found it difficult to navigate and its content not targeted enough toward specific audiences looking for practical advice on how to use health IT. Despite these difficulties, the program's website has been receiving a growing number of visitors, with an average of 15,000 visitors per month in 2008 (up from an average of 9,000 per month during 2006 and half of 2007). Although the AHRQ HIT program's ability to reach all of its intended beneficiaries through its website has been mixed, the program is attempting to improve its ability to reach these groups by: increasing awareness of the website among stakeholders outside of the grantee community, especially representatives of healthcare provider associations; improving the ease of searching and navigating the website; and better synthesizing the program's research findings into "how to" tools. The evaluation noted high rates of satisfaction with the AHRQ HIT program's webinars and teleconferences for the public, and national conferences for AHRQ grantees. To limit unintended subsidies, the AHRQ HIT program states that it does not fund research projects that have other sources of funding; however, the program may be unintentionally subsidizing some activities that would have occurred without the program, since projects funded by the AHRQ HIT program may have been able to secure funding from private foundations if the AHRQ HIT program did not exist.

Evidence: Information on how funds are allocated within the AHRQ HIT program is available in the Justification of Estimates for Appropriations Committees for the Agency for Healthcare Research and Quality for Fiscal Year 2009, available at [http://www.ahrq.gov/about/cj2009/cj2009.pdf]. The primary dissemination vehicle for the AHRQ HIT program's research findings and resources is the program's website, available at [http://healthit.ahrq.gov]. A discussion of who the intended beneficiaries of the AHRQ HIT program are and how well the program reaches them is included in the program evaluation of the AHRQ HIT program. Some private foundations funding health IT research and/or demonstrations include the Commonwealth Fund [http://www.commonwealthfund.org/General/General_show.htm?doc_id=289934], the Markle Foundation [http://www.markle.org/markle_programs/healthcare/projects/index.php], the Robert Wood Johnson Foundation [http://www.rwjf.org/pr/topic.jsp?topicid=1158], and the California Health Care Foundation [http://www.chcf.org/topics/index.cfm?topic=CL108].

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The AHRQ HIT program has proposed four long-term performance measures which are not recommended for use since they would not accurately reflect the purpose of this program -- which is to produce research showing how health IT can improve the quality of health care. Three of the proposed measures would gauge national adoption rates of three health IT tools or systems which are widely believed to hold promise in improving the quality of health care: 1) clinical decision support tools (which provide reminders to health care providers to give guideline-based care); 2) networks that allow the electronic exchange of health information (to improve care management between multiple providers); and 3) electronic prescribing systems (to improve the safety and accuracy of medication management). A fourth proposed performance measure would have measured the percentage of Chartered Value Exchanges (CVEs) that have an AHRQ HIT grantee or contractor as an active member. Further details on these four proposed long-term measures follows. The first measure would have focused on increasing the percentage of ambulatory (outpatient) medical practices that use electronic health records with built-in reminders to give guideline-based interventions and screening tests. This measure is not recommended, since it would not allow the AHRQ HIT program to determine whether increased adoption of clinical decision support tools is in fact accompanying any improvements in the quality of health care. The second proposed long-term measure would have gauged the number of states with market-based initiatives allowing health care providers to exchange health information electronically. This measure is not recommended, since it would not allow the program to determine whether an increase in the number of states that are home to initiatives exchanging health information electronically is in fact accompanying any improvements in the quality of health care. This measure is also problematic because it suggests that the underlying goal of the Federal government is for all states to have at least minimally-functional health information exchange activities underway. The Federal government favors allowing the market to determine which types of electronic health information exchange activities occur. Also, it is unclear of what value this measure's results would be, since the definition of a health information exchange activity (as used by the proposed data source for this measure) is broad. The third measure focuses on increasing the percentage of ambulatory (outpatient) physicians using electronic prescribing. This measure is not recommended, since it would not allow the program to determine whether increased adoption of electronic prescribing is in fact accompanying any improvements in the quality of health care. Another problem with this measure is it would not use a sufficiently precise data source, since the data's numerator and denominator are measured using an inconsistent methodology and unsourced estimates. A fourth proposed long-term measure, also not recommended for use, would gauge the percentage of Chartered Value Exchanges (CVEs) with one or more AHRQ HIT grantee or contractor as an active member. CVEs are local groups of health care stakeholders working to disseminate information rating the quality of care provided by local physicians, to help the public choose high-quality physicians. This measure is not recommended since it does not align with the AHRQ HIT program's activities -- the AHRQ HIT program does not conduct research on disseminating quality information about care provided by physicians, nor does it develop quality measures to be used by CVEs. Furthermore, there is no evidence that participation of an AHRQ HIT grantee increases the success rate of CVE efforts.

Evidence: On April 17, 2008, the AHRQ HIT program submitted a final draft of their proposed program performance measures via email. The AHRQ HIT program's purpose is articulated in AHRQ's FY 2009 Justification of Estimates for Appropriations Committees [http://www.ahrq.gov/about/cj2009/cj2009.pdf] as: "to develop evidence and inform policy and practice on how Health IT can improve the quality of American healthcare." The AHRQ HIT program has also contracted with Abt Associates to develop performance measures by the summer of 2009. Three of the four measures being developed by Abt Associates would gauge adoption rates of health IT tools, without attempting to measure the effects of such tools on improving the quality of health care, and are therefore not appropriate given the purpose of the AHRQ HIT program. These three measures are as follows: 1) the number of patients who can electronically access information on medication therapy; 2) the number of clinicians who can electronically access evidence-based prevention or treatment information; and 3) the number of clinician organizations that have adopted evidence-based decision support technologies. A fourth measure would gauge the reduction in medical errors due to adoption of computer prescribed order entry. Because this last performance measure would measure improvements in the quality of health care associated with adoption a health IT tool, it is an appropriate measure for this program; but since the other three measures being developed by Abt Associates would not measure improvements in the quality of health care resulting from health IT tools, they are not appropriate measures for the program. The performance measures being developed by Abt Associates are described in Competitive Task Order #07R000150, dated July 25, 2007.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The AHRQ HIT program does not have appropriate long-term performance measures.

Evidence: On April 17, 2008, the AHRQ HIT program submitted a final draft of their proposed program performance measures via email.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Although the AHRQ HIT program has proposed one long-term measure that may be acceptable as an annual performance measure, as well as two additional annual measures that would gauge the extent to which the program's research findings are used, there are currently no appropriate long-term goals to which these measures can be linked. An additional annual performance measure has been proposed, but it is not recommended for use. In addition to the long-term measure described in the answer to question 2.1 that could be an appropriate annual measure (which would measure the percentage of ambulatory practices that use electronic health records with built-in reminders to give guideline-based interventions and screening tests), the AHRQ HIT program proposed an additional two annual measures which would have measured the percentage of AHRQ HIT grantees and contractors whose findings are cited by other researchers and/or used to change clinical practice or policies of health care stakeholders within three years of grant completion. These additional two measures would have tracked the use of research findings in two of the program's three main research areas: 1) clinical decision support tools (which provide reminders to give guideline-based care); and 2) the exchanging of health information electronically (to improve care management by multiple providers). These measures would have indicated the degree to which AHRQ HIT grantees are producing relevant research that is moving the state of knowledge and practice forward in terms of improving the quality of health care through the use of health IT. A different annual measure related to the program's third main research activity (electronic prescribing systems, to prevent errors in medication delivery) would have measured the number of foundational electronic prescribing standards (identified in the Medicare Prescription Drug, Improvement, and Modernization Act of 2003) that have been adopted and published by the Department of Health and Human Services as requirements for health care providers participating in the Medicare program. The AHRQ HIT program was unable to demonstrate a link between publishing all nine of these foundational standards and improving the safety or accuracy of electronic prescribing. The link to improvements in quality may exist, but only through a somewhat tenuous link: publishing more standards is expected to make the data that is collected and transmitted through electronic prescribing systems more detailed and valuable to researchers, which has the potential to result in more specific, targeted research being conducted on the effect of electronic prescribing on improving the quality of care for specific populations.

Evidence: On April 17, 2008, the AHRQ HIT program submitted a final draft of their proposed program performance measures via email.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The AHRQ HIT program does not yet have appropriate annual performance measures. Among the three potentially-appropriate annual measures described in the answer to question 2.3, one has ambitious targets while two do not. For the proposed long-term measure that could be appropriate as an annual measure, the program's target would have been for 50% of ambulatory practices to use electronic health records that provide reminders to give guideline-based interventions and screening tests by 2014. This would have been an ambitious goal, since the rate of adoption of this type of health IT tool is currently increasing by only 2.4 percentage points per year (10.7% in 2005, to 13.1% in 2006), meaning that if adoption continued at this rate, only 21.6% of providers might be expected to adopt such tools by 2014. The AHRQ HIT program's goal of reaching 50% means the speed with which providers adopt such tools would have had to double to meet the program's target. Note that because only two years of baseline data are currently available for this measure, targets might have needed to be adjusted in a few years once trends in adoption rates were more apparent. The two other potentially-appropriate annual measures proposed by the AHRQ HIT program lacked ambitious targets. These would have measured the percentage of AHRQ HIT grantees in two of the program's three main areas of research whose work was used by others within three years of grant completion. These measures would have included modest 2010 targets of only 25% of grantees' findings being used within 3 years of completion of the first round of grants in these two research areas.

Evidence: On April 17, 2008, the AHRQ HIT program submitted a final draft of their proposed program performance measures via email.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: As reflected in the responses to questions 2.1 and 2.3, the AHRQ HIT program does not currently have appropriate long-term and annual performance measures. Although a recent AHRQ HIT funding announcement specified that the program would evaluate projects using measures related to the quality of health care (e.g. whether patients participating in funded projects were receiving appropriate care, could access reports on the quality of their ambulatory care providers, and were satisfied with their care), the program does not specify to which of the AHRQ HIT program's long-term goals (if any) these measures tie. The measures in this funding announcement are different from the measures the program uses in its annual budget requests, and different from the measures proposed as part of this PART process. Therefore, program partners appear to be committing to work toward performance goals that do not explicitly tie to the goals of this program.

Evidence: A sample Request for Applications (RFA-HS-08-002) shows that grantees are not required to commit to work toward any AHRQ HIT long-term and/or annual performance goals, however they are required to submit annual progress reports and financial statements.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The AHRQ HIT program commissioned independent third-party program evaluations in 2007 and 2008 to evaluate the effectiveness of the program and to provide detailed recommendations designed to support program improvements. These evaluations were conducted by consultants not associated with the Federal government and AHRQ. The 2007 evaluation assessed the degree to which the AHRQ HIT website contractor was adhering to its contract, while the 2008 evaluation provided a more full-bodied assessment of the program's website, the research products it creates and disseminates to the public, and the technical assistance and public training it provides. Although the 2008 evaluation did not explicitly examine the AHRQ HIT program's grant-issuing activities, it assessed the quality of the research products produced using research findings from the program's grants and contracts, which allows an indirect analysis of the appropriateness and quality of the research being funded by the AHRQ HIT program.

Evidence: A recent independent program evaluation assessed the number of users accessing the AHRQ HIT webpage and conducted qualitative interviews with health IT stakeholders to assess the relevancy of the program's research products and the effectiveness of its dissemination strategy. The evaluation included numerous recommendations about how the program could re-target and re-package its research products to better meet users' needs. A July 2007 program evaluation by Engineering Services Network/Sycamore Services, Inc. focused more narrowly on the AHRQ HIT website -- the public-facing aspect of this program -- and found that the contractor managing the AHRQ HIT program's website and delivery of technical assistance was in compliance with 71% of the requirements of its contract. This evaluation included 21 recommendations related to improving the contract with this contractor, the governance and oversight of the overall AHRQ HIT program, and future website/software acquisitions.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: AHRQ HIT budget requests do not mention how funding requests are linked to accomplishment of annual or long-term program performance goals. Also, resource needs are not presented in a complete and transparent manner in the program's budget: although the fiscal year 2009 budget request includes the total funding request for the AHRQ HIT program, along with funding levels for a few types of activities (such as new grants, continuing grants, and two specific small projects), the budget request does not identify the three main topics of research of this program: clinical decision support tools, electronic exchange of health information, and electronic prescribing -- nor does it provide funding levels associated with these three main topics of research.

Evidence: The AHRQ HIT program's budget request is articulated in AHRQ's FY 2009 Justification of Estimates for Appropriations Committees [http://www.ahrq.gov/about/cj2009/cj2009.pdf].

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The AHRQ HIT program has taken meaningful steps to correct its strategic planning deficiencies. To address the program's lack of performance measures associated with improving the quality of health care through health IT in ambulatory (outpatient) settings, the AHRQ HIT program contracted with Abt Associates in July of 2007 to identify new performance measures and data sources. New performance measures are expected to be developed by July of 2009. However, since three of the four measures being developed would only measure adoption rates of health IT tools, and not the impact of such tools on improving the quality of health care, they would not meaningfully correct this program's lack of appropriate performance measures that tie to the purpose of the program. The AHRQ HIT program is therefore encouraged to work with Abt Associates to update the Statement of Work to request the development of performance measures that relate to the program's stated purpose -- which is to develop and disseminate evidence about how to use health IT to improve the quality of health care, not to increase adoption of health IT tools. The program plans to correct other program deficiencies by reviewing and implementing, where appropriate, recommendations from a recent program evaluation, such as improving the search functionality of its website and improving the content of its website, such as by developing user-friendly summary briefs on various health IT topics.

Evidence: The AHRQ HIT program contracted with Abt Associates to develop program performance measures by July of 2009 through Competitive Task Order #07R000150, dated July 25, 2007. Issues with these measures are described in the "Evidence" section of question 2.1.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The AHRQ HIT program could not provide evidence that the program and its steering committee periodically assess and compare the program's efforts with those of a variety of public and private organizations to ensure the relevance and benefit to health IT stakeholders of the AHRQ HIT program's activities.

Evidence: None provided.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The AHRQ HIT program is guided in its budget decisions by priorities articulated by two advisory committees: 1) the AHRQ National Advisory Council, which advises the entire AHRQ agency on priority activities to pursue, and 2) the AHRQ HIT program's own steering committee. For instance, after a suggestion was made by the AHRQ National Advisory Council at an April 2005 council meeting, the AHRQ HIT program announced clinical decision support demonstration contracts in 2007. Similarly, a November 2006 meeting of the AHRQ HIT program's steering committee recommended that the AHRQ HIT program pursue research on health IT specifically in ambulatory (outpatient) care settings; based on this suggestion, the AHRQ HIT program announced a new Ambulatory Patient Safety program in 2007.

Evidence: The minutes from the AHRQ National Advisory Council meeting of April 15, 2005 include comments from council members suggesting that AHRQ pursue research related to clinical decision support tools; the minutes from the AHRQ HIT program's steering committee meeting of November 29, 2006 reflect the committee's interest in the AHRQ HIT program pursuing research on the use of health IT specifically in ambulatory (outpatient) care settings; the FY 2009 Justification of Estimates for Appropriations Committees for the Agency for Healthcare Research and Quality states that the Ambulatory Patient Safety program and clinical decision support demonstration contracts were first announced in 2007, and requests $29 million in FY 2009 for the Ambulatory Patient Safety program, $3 million of which is included for clinical decision support demonstration contracts [http://www.ahrq.gov/about/cj2009/cj2009.pdf].

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The AHRQ HIT program reviews the progress of both its grantees and the program overall, and uses this performance information to manage both grantees and the larger program. An overwhelming majority of grantees voluntarily submit quarterly progress reports, which the AHRQ HIT program uses to determine which grantees may require technical assistance to successfully complete projects on time and on budget. Annual progress reports are required of all grantees, and are used by the program to determine whether to continue (on an annual basis) grant funding. Progress reports allow the AHRQ HIT program to improve its performance by allowing it to identify projects with poor performance and to either intervene to improve the success of these projects or to discontinue funding.

Evidence: A sample quarterly progress report describes progress made toward completing planned deliverables, the rate of spending compared to the grant's original spending plan, issues and changes, preliminary research findings, outputs produced, and partnerships established. HHS policy allowing the discontinuation of a grant based on poor performance is stated in HHS Awarding Agency Grants Administration Manual, Chapter 3.07.107. A sample Task Order to a contractor shows that monthly progress reports are required, while a sample contract quarterly report shows that that particular contract required quarterly reporting. A sample Notice of Award to a grantee shows that quarterly progress reports are "strongly encouraged," while annual progress reports are required. The AHRQ HIT program advised in an email dated April 28, 2008 that all AHRQ HIT grantees are required to submit annual progress reports, and quarterly progress reports are required of grantees in the implementation-oriented Transforming Healthcare Quality Through Information Technology program. The more research-oriented Ambulatory Safety and Quality program only "strongly encourages" quarterly progress reports, since the program thought such reports could be an undue reporting burden on grantees not involved in implementation, since there are fewer deliverables and milestones on which to report -- but in practice, an overwhelming majority of grantees in this program submit quarterly progress reports nevertheless. As an example of how the AHRQ HIT program has used grantee progress reports to manage, when AHRQ received a quarterly report from a grantee that described their plans to spend an unnecessarily large percentage of their budget on evaluation efforts, AHRQ intervened and assigned technical assistance (a workshop on evaluation best practices and telephone support with AHRQ HIT staff and the program's technical assistance provider) to modify the grantee's evaluation plan and allow the grantee to complete their project on budget. As an example of a grant that was discontinued due to unsatisfactory progress, the AHRQ HIT program provided documentation showing that funding was discontinued at the end of the first year of a three-year grant, thereby saving the AHRQ HIT program $1 million.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Although program partners are held accountable for cost, schedule and performance results, Federal managers are only held accountable for some schedule results. Specifically, annual performance appraisals for Federal managers of the AHRQ HIT program only include consideration of whether employee deadlines for submitting funding recommendations and invoice approvals were met, but the appraisals do not hold Federal managers responsible for cost or performance results of AHRQ HIT grantees or contractors. Meanwhile, program partners of the AHRQ HIT program, such as grantees and contractors, are held accountable for cost, schedule, and performance results: when performance targets such as planned deliverables or milestones are not met by grantees or contractors, grants or contracts are discontinued or technical assistance is provided by the AHRQ HIT program to improve future performance. To ensure funds are spent for their intended purpose, AHRQ HIT project staff monitor programs through review of progress reports and site visits. If it is discovered that funds have been spent outside their purpose, AHRQ remediates the problem by temporarily withholding cash payments, employing a reimbursement payment method, suspending or terminating an award, withholding a non-competing award, disallowing costs (meaning AHRQ HIT refuses to reimburse these costs), or employing government-wide suspension and debarment.

Evidence: A sample employee performance appraisal shows that Federal managers' performance is appraised partly based on whether employee deadlines for submitting funding recommendations and invoice approvals are met. As an example of a grant that was discontinued due to unsatisfactory progress, the AHRQ HIT program provided documentation showing that funding was discontinued at the end of the first of the three years of a grant, thereby saving the AHRQ HIT program $1 million. Descriptions of HHS remediation policies are described in HHS Awarding Agency Grants Administration Manual, Chapter 3.07.107, "Termination and Enforcement."

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The AHRQ HIT program obligates funds (including research grants, contracts, inter-agency agreements, and requisitions) in a timely manner, accurately reports use of funds, and ensures they are spent for their intended purpose. To ensure timely obligation of funds, AHRQ monitors awards by maintaining a commitment register for funds it monitors and verifies, and ensures accounting data is accurate and funds are available for obligation. AHRQ does not audit all grantee finances, but HHS conducts A-133 audits annually for grantees that receive more than $500,000 in Federal funds per year. The AHRQ HIT program tends to obligate the majority of its funds late in the fiscal year (with 94% of funds obligated in the fourth fiscal quarter of 2007). AHRQ's Senior Leadership Team is currently striving to more evenly distribute obligations and has recently required that FY 2008 spending plans reflect a schedule for obligations that more evenly obligates funds over fiscal quarters (with 25% specified as the ideal obligation distribution per quarter). In response to this requirement, the AHRQ HIT program's planned obligations for new contracts are more evenly distributed between quarters in FY 2008. AHRQ HIT grantees establish schedules for obligations, by specifying in grant applications which activities and funding level will occur in each 12-month budget period for multi-year projects. The AHRQ HIT program promptly reports grants and contracts awards to applicable Federal databases. For grants, transactions are transmitted to the NIH's grants database (IMPAC II, which stands for "Information for Management, Planning, Analysis, and Coordination") within one day of a grant award. This NIH grants database then automatically transmits this grant award information to HHS' grants database (TAGGS, the "Tracking Accountability in Government Grants System") twice a month, after AHRQ has confirmed its accuracy. Prior to April 2008, grants data in this HHS database was then transmitted to a Federal database called FAADS ("Federal Assistance Award Data System"); however starting in April 2008, this data began being transmitted to www.USASpending.gov (a publicly-accessible, searchable website listing information on grants and contracts issued by the Federal government). The process for reporting contracts is similar: once a contract is executed, within one business day that contract action is reported into HHS' contracts database (DCIS, the "Department Contract Information System"), which simultaneously also updates a Federal contracts database (FPDS-NG, the "Federal Procurement Data System - Next Generation"). Each contract action that is reported is independently reviewed to ensure the accuracy of the information reported.

Evidence: Evidence on whether the AHRQ HIT program has ever lapsed funds was not available, but a review of AHRQ's FY06-FY09 Justifications of Estimates for Appropriations Committees [http://www.ahrq.gov/about/budgtix.htm#budget] shows that in the past four years, the AHRQ agency as a whole has lapsed no more than a tenth of a percentage of its funds per year -- meaning it returned these funds to the Treasury Department as a result of failing to commit the funds through contracts or grants within a year of being appropriated such funds. If these lapsed funds were entirely attributable to the AHRQ HIT program, the percentage of the AHRQ HIT program's funds that would have lapsed would still only be a quarter of a percentage point. AHRQ provided a sample audit of an AHRQ grantee dated July 30, 2007, and has an Audit Liaison Officer who coordinates any response required to resolve A-133 issues involving its grantees. The Budget Execution and Monitoring Cycle Memo for FY 2007 documents key internal controls and processes for AHRQ's budget execution and monitoring activities. The Grants Management Cycle Memo documents the entire grants management cycle at AHRQ (including the procedures to record a grant obligation (page 19) and monitoring activities (page 33)). A table displaying FY 2007 obligations by quarter shows that 94% of obligations were made in the last quarter of the fiscal year. An October 29, 2007 slide presentation by AHRQ HIT to AHRQ senior leadership shows that obligations for new contracts are more evenly spread over FY 2008's four fiscal quarters. The budget period process used by the AHRQ HIT program is described in the HHS Grants Policy Statement [www.hhs.gov/grantsnet/docs/HHSGPS_107.doc].

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The AHRQ HIT program does not have an efficiency measure in place to allow it to measure cost savings achieved each year.

Evidence: An efficiency measure proposed by the AHRQ HIT program on April 17, 2008, which is not recommended for use, is the "Per unique user cost of the AHRQ HIT website," which would measure the amount spent on a contract to maintain the website and prepare research products for publication on the website divided by the number of visitors to the website per year. This measure would be an incomplete measure of the true cost of the website, since it would not capture program managers' efforts to improve economy in the acquisition and utilization of resources, and since the number of visitors to a website is easily susceptible to manipulation. The AHRQ HIT program also proposed an efficiency measure that would track the percentage of grantees that use the program's electronic application and reporting system instead of submitting documents in hard copy format. This measure is not recommended, since it did not trace back to producing more value for each program dollar spent.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The AHRQ HIT program participates in many health IT stakeholder organizations and committees, which have led to greater collaboration and coordination of AHRQ HIT activities with other stakeholders' activities. For instance, as a result of AHRQ HIT staff participation in a workgroup of ONC's American Health Information Community, the AHRQ HIT program has agreed to convene a new multi-stakeholder steering committee to guide, critique, and comment on AHRQ HIT contracts related to clinical decision support tools.

Evidence: Minutes from the October 25, 2007 meeting of the American Health Information Community's Personalized Health Care Workgroup described the new steering committee that will be convened to review AHRQ HIT contracts related to clinical decision support tools [http://www.hhs.gov/healthit/ahic/materials/summary/phcsum_102507.html]. AHRQ HIT staff participate at all levels on a wide array of public and private health IT organizations, including leadership roles in the American Health Information Community, commissioner roles in the Certification Commission for Health Information Technology, (both of which are entities under the purview of the Office of the National Coordinator for Health IT), and board-level roles in the National Quality Forum (a non-profit membership organization), the National Committee on Vital and Health Statistics (an advisory board to the Secretary of HHS), Connecting for Health (a public-private collaborative with representatives from 100+ health care stakeholder organizations), and the Electronic Health Record Vendor Association. The AHRQ HIT program collaborates with State governments through its state and regional demonstration projects, and works with the National Governors Association's State Alliance for e-Health.

Does the program use strong financial management practices?

Explanation: The Department of Health and Human Services (HHS) is the parent agency of the AHRQ HIT program, and received an "unqualified" (clean) opinion in its latest Department audit. Although the AHRQ agency has issued only a "qualified" statement of assurance that its internal controls and financial systems currently meet the objectives of the Federal Managers Financial Integrity Act (FMFIA), this is solely because AHRQ's financial accounting service provider (HHS' Program Support Center (PSC)) currently has material weakness and nonconformance issues related to FMFIA. PSC is working to correct these deficiencies, and AHRQ has taken steps to counter PSC weaknesses relevant to AHRQ activities (which are described in the "Evidence" of this answer). On a related note, auditors have issued an "unqualified" (clean) statement of assurance that PSC's grant payment management services (which are used by AHRQ) meet the objectives of FMFIA and OMB Circular A-123. Examples of responsible financial management practices employed by AHRQ include the following practices: 1) participation in PSC's A-123 review, and a robust A-123 Appendix A Assessment of Internal Controls over Financial Reporting at AHRQ; 2) yearly Improper Payment Information Act Risk Assessments; 3) participation in HHS' CFO audit; and 4) quarterly reporting of disbursements to PSC's Payment Management System. In addition, at the grantee and contractor level, AHRQ conducts: 1) annual progress reporting by grantees and contractors; 2) annual financial status reporting of expenditures by grantees and monthly financial reporting of expenditures by contractors to AHRQ; 3) site visits; and 4) reviews and takes appropriate action for annual A-133 audits of grantees that receive $500,000 or more.

Evidence: A memo titled "FY2007 FMFIA/ICOFR Management Assurance Statements" from the Director of AHRQ to HHS Senior Assessment Team states that AHRQ provides a "qualified" statement of assurance that its internal controls and financial systems meet the objectives of FMFIA and OMB Circular A-123, and describes three material weakness and nonconformance issues. The material weaknesses identified in the FY 2006 CFO audit of PSC follow. 1) A material dollar amount of accounting entries on the Summary of Unadjusted Differences was identified. This occurred because PSC compiles its financial statements through a multi-step process using a combination of manual and automated procedures, which increases the risk of entry errors. Until uniform policies and procedures for the preparation of financial statements at PSC are in place, AHRQ will continue to review quarterly financial statements and sign off on data that is included in PSC's Automated Financial System (see Quarter 1, FY 2008 Data Attestation, which shows that AHRQ reviews quarterly financial statements and signs off on the data that is included in PSC's Automated Financial System). 2) PSC had not yet completed a full annual reporting cycle using its new financial management system, UFMS. AHRQ has continued to enter all accounting information in its Budget Database System, and do a major reconciliation once key reports are available. 3) The documentation for the financial reporting closing process has not yet been finalized, since the current year-end annual financial statements have not yet been validated. In terms of PSC's grant payment management services, these were given an "unqualified" or clean statement of assurance that they met the objectives of FMFIA and OMB Circular A-123, as documented in a signed memo from October 2006 from AHRQ's financial accounting service provider, the HHS Program Support Center (PSC). Examples of responsible financial management practices employed by AHRQ (and further discussion of evidence provided) include the following: 1) participation in PSC's A-123 review, and a robust A-123 Appendix A Assessment of Internal Controls over Financial Reporting at AHRQ (the results of AHRQ's most recent A-123 assessment of internal controls are detailed in Appendix V: Control Deficiencies Log, and Appendix VI: Corrective Action Plan; this review found no material weaknesses or significant deficiencies); 2) yearly Improper Payment Information Act Risk Assessments (AHRQ's 2007 Improper Payments Information Act Risk Assessment assesses the agency's susceptibility to improper payments; the program within which the AHRQ HIT program resides was deemed "low risk"); 3) participation in HHS' CFO audit (HHS' FY 2007 Agency Financial Report [http://www.hhs.gov/afr/] shows that AHRQ participates in this audit, which was listed as unqualified or "clean" in FY 2007); and 4) quarterly reporting of disbursements to PSC's Payment Management System (which is used to conduct quarterly review of grantee financial data, and provides disbursement, grant monitoring, reporting, and cash management services to both awarding agencies and grant recipients [http://www.dpm.psc.gov]). In addition, at the grantee and contractor level, AHRQ conducts: 1) annual progress reporting by grantees and contractors; 2) annual financial status reporting of expenditures by grantees and monthly financial reporting of expenditures by contractors to AHRQ; 3) site visits; and 4) reviews and takes appropriate action for annual A-133 audits of grantees that receive $500,000 or more (an example of an annual audit of an AHRQ grantee that receives $500,000 or more in Federal funds was provided). AHRQ's key financial internal controls are documented in the following FY 2007 transaction cycle memos dated September 30, 2007: the Budget Execution and Monitoring Cycle Memo, Grants Management Cycle Memo, Funds Management Cycle Memo, and the Revenue Cycle Process Documentation.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The AHRQ HIT program has established systems and procedures for identifying and correcting program management deficiencies. The Program has instituted periodic surveys of stakeholders to identify and prioritize program focus areas and to assess how well the program is meeting its stakeholders' needs. Based on feedback from 238 participants of a March 2008 webinar, the program identified a discrepancy between activities sought by its stakeholders and the program's planned activities, and subsequently re-prioritized the topics for its national webinars to better address these needs. Also, surveys from AHRQ HIT program stakeholders after the AHRQ Annual Conference revealed that stakeholders wanted less academic research findings and more practical findings and tools to apply in health care settings. As a result, AHRQ has directed its NRC contractor to focus more on delivering practical resources (e.g. emerging lessons learned and tools) instead of more academic summaries of individual studies. The program has also made program management improvements in response to deficiencies identified by program management. For instance, an identified lack of the ability to provide summary statistics on the number of grantees focusing on a particular research area was recently fixed by migrating the program to an electronic grant reporting system, which now gives staff the ability to run database queries.

Evidence: In response to an electronic request for suggestions of future webinar topics, 238 webinar participants suggested topics. Suggestions were widely arrayed among many topics, but the two most popular topics were electronic health records as they related to pay-for-performance activities currently underway through the Centers for Medicare and Medicaid Services (CMS), and the privacy and security of electronic health information exchange. A screen shot provided by the AHRQ HIT program from their new electronic database shows what fields of information are collected and searchable.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: AHRQ HIT grants are awarded through a competitive process. AHRQ HIT funding opportunities are published publicly on two centralized websites: the NIH Guide for Grants and Contracts, and the centralized website listing government grant opportunities at http://www.grants.gov. The program provides a reasonable amount of outreach to encourage participation of new grantees; specifically, the AHRQ HIT program posts funding announcements on their website and mentions them in AHRQ's e-newsletter, which reaches over 60,000 subscribers, and also transmits announcements to health IT community partners for inclusion in articles and other newsletters. When funding opportunities have targeted non-traditional partners, such as rural and small hospitals, the AHRQ HIT program has made special efforts to reach out to them, and has had these efforts rewarded by some of the highest grant application response rates in AHRQ's history. All applications are peer-reviewed and ranked based upon published funding criteria which include scientific merit as well as other measures, and are approved for funding in order of merit, ensuring not to award duplicative grants.

Evidence: The "Extramural Grants Policies (Awarding Agency Grants Administration Manual)" details standardized information to be included in all published funding opportunities, requirements to be met when requiring applicants to meet qualifying criteria, and peer review requirements. The "HHS Grants Policy Statement," issued by HHS' Assistant Secretary for Resources and Technology on January 1, 2007 articulates for the extramural research community the commitment of HHS operating divisions (which includes AHRQ) to the policies outlined in the previous piece of evidence. AHRQ regulations (in the Code of Federal Regulations, at Title 42, Volume 1, Part 67, or CFR 67.12 - 67.17) require peer review of grant applications, stipulate funding criteria that may be used other than scientific merit, and generally define the structure of the AHRQ grant process [http://www.access.gpo.gov/nara/cfr/waisidx_99/42cfr67_99.html]. Many grants awarded by AHRQ are multi-year projects, funded 12 months at a time. Such non-competitive renewal awards are issued based on demonstration of satisfactory progress toward meeting the performance goals of the project as stated in the original application, continuing agency need for the project, availability of funds, and recipient compliance with the terms and conditions of previous award(s). An example of criteria used by expert scientific panels when reviewing AHRQ HIT grant applications are listed in the Funding Opportunity Announcement: "RFA-HS-08-002 - Ambulatory Safety and Quality Program: Improving Management of Individuals with Complex Healthcare Needs through Health IT (R18)" [http://grants.nih.gov/grants/guide/rfa-files/RFA-HS-08-002.html].

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The AHRQ HIT program collects and analyzes quarterly and annual grantee progress reports. These reports ensure that grantees are using funds appropriately. Grantees report on whether they met milestones and deliverables and provide a brief description of accomplishments during the reporting period. Grant deliverables and spending activities are reviewed frequently, typically on a monthly basis throughout the term of the contract. Invoices and reports that are submitted to AHRQ are reviewed to ensure that all expenditures are allowable and allocated to the correct cost categories and potential problems are identified and resolved.

Evidence: "AHRQ Research Reporting System - User's Guide" specifies how grantees and contractors report information to the AHRQ HIT program. A sample quarterly progress report describes progress made toward completing planned deliverables, the rate of spending compared to the grant's original spending plan, issues and changes, preliminary research findings, outputs produced, and partnerships established. AHRQ conducts audits of its grants on an annual basis and also before no-cost time extensions are granted in order to complete projects.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Although the AHRQ HIT program collects grantee progress reports, the program does not make this information available to the public at either the grantee level or aggregated at the program level on an annual basis. The public is therefore unable to determine whether grantees are meeting performance targets, such as whether planned deliverables and milestones are being met by grantees, prior to the availability of grantees' final reports. Instead, information that is collected regularly -- such as interim reports summarizing research activities and findings -- is used to develop research products on the program's website, such as short briefs describing "Emerging Lessons Learned" from multiple grantees (which do not identify grantees or discuss the degree to which grantees are meeting performance goals). The program states that final reports outlining research findings for individual grantees are available through a search engine that stores all information products produced by AHRQ over the past 35 years, however AHRQ HIT reports are not displayed and organized in a user-friendly way, and searches for known grantees' reports sometimes return zero results. Also, grantees do not report information on whether their research findings are actually used by other health care providers or health services researchers, which would suggest whether their findings are having an impact on improving the quality of health care (the stated purpose of the AHRQ HIT program).

Evidence: The "AHRQ Research Reporting System - User's Guide" describes how information is reported by grantees and contractors [https://arrs.ahrq.gov/ARRS]; the "Health IT Tools" webpage of the AHRQ HIT website provides links to various research products reflecting findings from previous AHRQ HIT contracts and grants [http://healthit.ahrq.gov/portal/server.pt?open=512&objID=919&parentname=CommunityPage&parentid=5&mode=2&in_hi_userid=3882&cached=true]; an example of a grantee's final report is available at [http://healthit.ahrq.gov/portal/server.pt/gateway/PTARGS_0_3882_227310_0_0_18/SureScripts%20-%20Final%20Report.pdf]; the AHRQ searchable database that stores grantees' final reports is available at [http://ahrq.ntis.gov/default.aspx].

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation:

Evidence: The AHRQ HIT program is a competitive grant program.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The AHRQ HIT program's proposed long-term performance measures focus on increasing national adoption rates for various types of health IT tools, which are not appropriate measures of this program's success, since this program focuses on developing and disseminating research on how health IT can improve the quality of health care, not on increasing the adoption of health IT. The AHRQ HIT program has not yet developed long-term performance measures that measure the extent to which health IT can improve health care.

Evidence: The program's performance measure regarding electronic prescribing is mentioned in AHRQ's FYs 2008 and 2009 Justification of Estimates for Appropriations Committees [http://www.ahrq.gov/about/cj2008/cj2008.pdf; http://www.ahrq.gov/about/cj2009/cj2009.pdf]. Other long-term performance measures were submitted on April 17, 2008.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The AHRQ HIT program does not have appropriate annual performance measures that measure progress toward achieving long-term program goals.

Evidence: The AHRQ HIT program submitted proposed annual performance measures on April 17, 2008, which are not recommended for use in the absence of appropriate long-term measures to which these annual measures could be linked.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The AHRQ HIT program has been unable to demonstrate improved efficiencies or cost savings, primarily because it does not have an efficiency measure. In addition, although the AHRQ HIT program has taken steps to computerize and streamline program management processes, it has been unable to show any cost savings resulting from these changes.

Evidence: The AHRQ HIT program proposed efficiency measures that are described in the "Evidence" for question 3.4 and are not recommended for use. The improvements to AHRQ HIT program management processes that the AHRQ HIT program has pursued include: 1) migrating from a paper-based grant application process to an online system for accepting grantee applications and progress reports (which allows staff to run database queries to produce summary statistics about its grants; this is described in the "AHRQ Research Reporting System - Online Help Manual"); and 2) streamlining the process by which it approves new research products for its website, with the goal of reducing the cycle time for this process (shown in a workflow chart provided by the program).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: A comparison of the performance of the AHRQ HIT program to other health IT research efforts has not been conducted. Additionally, without a clearly defined set of long-term or annual performance measures, it is difficult to compare the AHRQ HIT program's performance to that of other programs with similar purposes and goals.

Evidence: None provided.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: An independent evaluation suggests that the AHRQ HIT program's website is somewhat effective. In a summary assessing the strengths and weaknesses of the AHRQ HIT program, the evaluation noted that the program's conferences and teleconferences have been well received, and several practical tools have been produced. It also noted that the program's website averaged 9,000 visits per month (3,500 of whom stayed on the website for at least 2 minutes) in 2006 and half of 2007, and that 60% of the 621 website users who responded to a survey reported high levels of satisfaction. However, this evaluation indicated that nearly two-thirds of the users who visit the program's website leave the site in under 2 minutes, and nearly half leave within 30 seconds -- suggesting many potential users of the program's website are not finding what they need on this website. Some other weaknesses highlighted by the evaluation that prevent this program from being considered to be effective to a "large extent" include the fact that: technical assistance was either not used, or was considered not entirely "on the mark" due to site-specific grantee idiosyncrasies that made the available information limited in its usefulness; the private AHRQ HIT website for grantees was generally not well used; and provider associations involved in health IT issues were unaware of the website, and when they were shown it, found a need for the website to be easier to navigate and search for content, and for the content to be more targeted to specific audiences and to include more direct advice on adopting and using health IT. In addition, the program cannot be considered to be effective to a "large extent" since it does not measure the extent to which it is achieving results -- such as by measuring the percentage of grantees and contractors whose research findings are used either by health care providers or health services researchers, or by measuring the extent to which health care providers who use health IT have been able to improve the quality of care they provide.

Evidence: The AHRQ HIT program has reported that in 2008, its website averaged 15,000 visits per month, though the percentage of visitors who stayed on the website for at least 2 minutes was not available.