ExpectMore.gov


Detailed Information on the
Access to Recovery Assessment

Program Code 10009019
Program Title Access to Recovery
Department Name Dept of Health & Human Service
Agency/Bureau Name Substance Abuse and Mental Health Services Administration
Program Type(s) Competitive Grant Program
Assessment Year 2007
Assessment Rating Moderately Effective
Assessment Section Scores
Section Score
Program Purpose & Design 100%
Strategic Planning 88%
Program Management 70%
Program Results/Accountability 67%
Program Funding Level
(in millions)
FY2007 $98
FY2008 $96
FY2009 $100

Ongoing Program Improvement Plans

Year Began Improvement Plan Status Comments
2008

Finalize design/data collection tools for comprehensive, cross-site evaluation of ATR

Action taken, but not completed Data collection tools in the process of being finalized. OMB clearance process will begin by October 2008.
2007

Providing guidelines and targeted technical assistance to grantees to further define the most appropriate recovery support services.

Action taken, but not completed
2007

Establishing formal linkages with the DOJ , HUD and other relevant agencies as appropriate.

Action taken, but not completed Discussions have begun about process for formalizing linkages.
2007

Finalizing contract for Independent cross-site evaluation of Access to Recovery Program by Fall of 2007.

Action taken, but not completed
2007

Developing a refined efficiency measure by Spring 2009.

Action taken, but not completed Measure to be completed by March, 2009. SAMHSA will provide ONDCP and OMB draft revisions in November, 2008 and February, 2009.
2007

Convening a Technical Consultation Group by December 2007 and reporting on the findings for incorporating both cost and quality drivers into the efficiency measure.

Action taken, but not completed
2008

Incorporate findings from Technical Consultation group on cost and begin to integrate actual data with recommendations to start finalizing proposed efficiency measure

Action taken, but not completed
2008

Begin discussions with grantees on evaluation data collection activities

Action taken, but not completed Discussions to be completed by 3/09

Completed Program Improvement Plans

Year Began Improvement Plan Status Comments

Program Performance Measures

Term Type  
Long-term Outcome

Measure: Percentage of individuals receiving services who had no past month substance use.


Explanation:The target for number of clients served was substantially exceeded. Grantees performed exceptionally well once infrastructure and program processes were full in place. The targets for future years reflect the new cohort of grantees, which will be in their first year of service delivery in 2008 and thus are expected to serve fewer clients. The second cohort of grantees (to begin reporting performance data in FY 2008) will have a significant focus on methamphetamine users. These clients may require additional resources beyond those of other clients, which may result in a decrease in numbers served. Targets have been set in collaboration with OMB.

Year Target Actual
2010 82% TBR 12/10
2009 81% TBR 12/09
2008 80% TBR 12/08
2007 81% 84.7%
2006 79% 81.4%
2005 Baseline 78%
Long-term Output

Measure: Number of clients gaining access to treatment.


Explanation:The target for number of clients served was substantially exceeded. Grantees performed exceptionally well once infrastructure and program processes were full in place. The targets for future years reflect the new cohort of grantees, which will be in their first year of service delivery in 2008 and thus are expected to serve fewer clients. The second cohort of grantees (to begin reporting performance data in FY 2008) will have a significant focus on methamphetamine users. These clients may require additional resources beyond those of other clients, which may result in a decrease in numbers served.

Year Target Actual
2010 65,000 TBR 12/10
2009 65,000 TBR 12/09
2008 30,000 TBR 12/08
2007 50,000 79,150
2006 50,000 96,959
2005 Baseline 23,138
Annual Outcome

Measure: Percentage of individuals receiving services who had improved family and living conditions.


Explanation:Data from the first cohort of ATR grantees demonstrarte that 35.5 percent of Meth clients had improved family and living conditions versus 51 percent of all clients. The second cohort of ATR grantees, which begin in 2007 is expected to have almost triple the number of menthamphetamine users.

Year Target Actual
2010 52% TBR 12/10
2009 52% TBR 12/09
2008 52% TBR 12/08
2007 52% 59.9%
2006 63% 51.0%
2005 Baseline 62%
Annual Outcome

Measure: Percentage of individuals receiving services who had no involvement with the criminal justice system.


Explanation:Data from the first cohort of ATR grantees shows that 95.2 percent of meth clients had no involvement with criminal justice versus 96.8 percent of all clients.

Year Target Actual
2010 97% TBR 12/10
2009 96% TBR 12/09
2008 96% TBR 12/08
2007 97% 97.6%
2006 95% 96.8%
2005 Baseline 95%
Annual Outcome

Measure: Percentage of adults receiving services who had improved social support.


Explanation:There were no significant differences between the methamphetamine users and other drug users for this measure. The target for the first performance year of the new cohort has been set at the current performance level, wuith a gradual increase in the least year of the program. The target for improved social support was missed, although the actual performance of 75% reflects a significant achievement. CSAT is reviewing program information and consulting with grantees to determine the reason for the decline, and will continue to work with grantees in cohort 2 to improve data on this particular measure. Since the 2007 results appears to be an anomaly compared to the previous two years' results of 89% and 90%; targets are being maintained at an ambitious level until further information is obtained.

Year Target Actual
2010 91% TBR 12/10
2009 90% TBR 12/09
2008 90% TBR 12/08
2007 90% 75.1%
2006 90% 90%
2005 Baseline 89%
Annual Outcome

Measure: Percentage of individuals receiving services who are currently employed or engaged in productive activities.


Explanation:Data from the first cohort of ATR grantees shows that 40 percent of meth clients were currently employed or engaged in productive activities versus 50 percent of all clients. Thus, the targets are ambitious because they are higher than current perfromance despite the fact that the second cohort is expected to triple the number of methamphetamine clients. The 2007 target for employment was significantly exceeded, reflecting very active effort by grantees to ensure that clients improved their overall life quality. The target was set based on actual performance for the previous two years and was equal to the actual performance in 2006. The second cohort of grantees includes a significant emphasis on methamphetamine users, who are expected to present additional challenges for securing employment beyond those of other clients. Therefore the 2007 level of performance is not expected to continue in future years. Targets for 2008 and 2009 are still higher than the 2007 target and thus represent an ambitious level.

Year Target Actual
2010 53% TBR 12/10
2009 53% TBR 12/09
2008 53% TBR 12/08
2007 50% 61.7%
2006 57% 50%
2005 Baseline 56%
Annual Efficiency

Measure: Average client cost per person served through the ATR Program


Explanation:

Year Target Actual
2007 Baseline $1605
2008 $1605 12/08
2009 $1588 12/09
2010 $1572 12/10

Questions/Answers (Detailed Assessment)

Section 1 - Program Purpose & Design
Number Question Answer Score
1.1

Is the program purpose clear?

Explanation: Access to Recovery (ATR) provides grants to States, Tribes, and Tribal organizations to carryout voucher programs that deliver substance abuse clinical treatment and recovery support services. ATR was created in FY 2004 as a consumer driven mechanism to: 1) expand capacity, 2) promote choice, and 3) enhance accountability within the substance abuse treatment system. The program expands substance abuse treatment capacity by increasing the number and types of providers who deliver clinical treatment and/or recovery support services such as medical detoxification, residential services, peer support, case management, housing, job training and placement, daily living skills, childcare, and transportation. In addition, ATR promotes choice by facilitating the pursuit of recovery via many different and personal pathways and allows people in need of treatment to choose from a wide array of clinical treatment and recovery support services providers, including faith and community-based providers. Finally, ATR enhances accountability by measuring outcomes and monitoring data to deter fraud and abuse.

Evidence: The Authority for ATR is provided under section 501(d)(5) and 509 of the Public Health Service Act (42 U.S.C. sections 290aa(d)(5) and 290bb-2). ATR uses vouchers coupled with State flexibility to promote choice and create positive change in the substance abuse treatment and recovery service delivery system. The Program's purpose is clearly articulated in the 2004 and 2007 grant announcements http://www.samhsa.gov/Grants/2007/TI_07_005.pdf and the FY 2007 Congressional Justification http://www.samhsa.gov/Budget/FY2008/SAMHSA08CongrJust.pdf.

YES 20%
1.2

Does the program address a specific and existing problem, interest, or need?

Explanation: In 2005, the National Household Survey on Drug Use and Health (NSDUH) found that 20.9 million persons needed treatment for an illicit drug or alcohol use problem, but did not receive treatment at a specialty facility. ATR is helping to fill this need by expanding access to treatment and recovery support services through a unique voucher based system. Among other benefits, vouchers draw in a greater breadth and diversity of service providers, including some that would not collaborate with the government to deliver services under traditional grant contract arrangements. For example, faith-based providers that view religious activities or messages as a vital part of their recovery program would be forced to abandon these elements if funded via direct Federal funds. As a result, some high-quality faith-based organizations choose not to participate in government-funded programs. However, when vouchers are employed and clients are offered equivalent non-religious options, programs with religious elements can be included among those offered to clients. This allows faith-based organizations to retain the characteristics of their programs, enables clients to choose a program best suited to his or her unique needs, and broadens the pool of service providers from which government can draw services. ATR reduces the barriers that faith-based organizations confront in providing clinical treatment and recovery support services via direct Federal funds.

Evidence: The National Household Survey on Drug Use and Health (2005) found that 23.2 million persons aged 12 and older reported that they needed treatment for an alcohol or drug problem. Of these, 20.9 million persons did not receive treatment in the past year. (http://www.oas.samhsa.gov/NSDUH/2k5NSDUH/2k5results.htm#7.3) http://www.oas.samhsa.gov/2k5state/ageTabs.htm#Tab21. The Charitable Choice regulations, http://a257.g.akamaitech.net/7/257/2422/09nov20051500/edocket.access.gpo.gov/cfr_2005/octqtr/pdf/42cfr54a.1.pdf), which govern direct Federal funding, limit the participation of faith-based organizations. ATR allows a much larger community of faith-based organizations to receive federal funding because the direct funding barrier is eliminated. The Religious Freedom Restoration Act of 1993 states, "Governments should not substantially burden religious exercise without compelling justification. (http://frwebgate.access.gpo.gov/cgi-bin/useftp.cgi?IPaddress=162.140.64.45&filename=h1308enr.txt&directory=/disk3/wais/data/103_cong_bills)

YES 20%
1.3

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ATR is a voucher-based system of care, uniquely designed to provide clinical treatment and recovery support services. A voucher-based model is fundamentally different from the more traditional ways that treatment and recovery support services are currently provided (i.e., Block Grants and/or categorical grants). In particular, a voucher-based system allows faith-based and community-based organizations to participate, thus expanding capacity. Most importantly, the voucher-based mechanism enables client choice allowing the individual to select his/her provider. ATR is the only program within the Substance Abuse and Mental Health Services Administration and other State-based substance abuse activities that utilize client-based vouchers. Additionally, recovery support services play an integral role in this initiative, which are not traditionally provided through the Substance Abuse Prevention and Treatment Block Grant. Although other State, Federal (i.e., ACF, HUD, SAMHSA, and TANF), and local efforts provide some of these same recovery support services (i.e., housing, transportation, counseling, and peer support), ATR allows individuals to choose their personal pathway for recovery among non-traditional providers. To reduce the potential for overlap at the State, Federal, and local level, ATR grantees may not supplant existing funds.

Evidence: The ATR voucher-based system is unique from other substance abuse systems and is described in the FY 2004 and 2007 Requests for Applications (http://atr.samhsa.gov/downloads/ti04009_ATR.doc p.4, and http://www.samhsa.gov/Grants/2007/TI_07_005.pdf p.4). The unique aspects of the program are also described in the 2004 HHS Press Release http://www.hhs.gov/news/press/2004pres/20040803c.html.

YES 20%
1.4

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: ATR was created to establish a new mechanism for providing substance abuse treatment because of limitations within the current system to expand treatment and develop a recovery-oriented system of care that benefits from the contributions of community and faith-based providers. The key to ATR's design is the relationship established between the States/Tribal Organizations and clients receiving services to ensure clients have a genuine and independent choice among faith-based and secular providers. The voucher-based design permits access to a full range of providers and services, specifically unique to the ATR program. Approximately 64 percent of ATR clients have received recovery support services, a central piece of the ATR program. ATR is the only mechanism within the current treatment system that provides the flexibility to maximize participation of faith and community-based providers in providing recovery support services. The unique design of ATR requires substantial start-up time and infrastructure to administer the program (i.e, documentation of services delivered, and vouchers redeemed) successfully. The emphasis on developing a strong infrastructure is not a design flaw, but presents some programmatic challenges. Some grantees have struggled with the initial implementation and management of the vouchers and have allocated up to 20 percent of ATR grant funds to develop administrative systems.

Evidence: The FY 2007 Congressional Justification provides information on the effectiveness of ATR. http://www.samhsa.gov/Budget/FY2008/SAMHSA08CongrJust.pdf

YES 20%
1.5

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: ATR directly places the resources and the choice of treatment providers into the hands of the program's intended beneficiaries. ATR requires grantees to ensure that clients have a genuine and independent choice to substance abuse clinical treatment and recovery support services. Grantees assess all clients to determine the appropriate clinical treatment and recovery support services. Grantees determine whether resources are reaching the intended beneficiaries and manage client vouchers through data collection on intake assessments, clinical treatment and/or recovery support services rendered, and vouchers redeemed. To avoid unintended subsidies, SAMHSA requires that ATR funds supplement, not supplant, current funding for substance abuse clinical treatment and/or recovery support services.

Evidence: To ensure resources reach intended beneficiaries, ATR grantees are required to provide all eligible clients a clinical assessment prior to receipt of services. Requirements for defining eligible clients are specified in the RFA http://atr.samhsa.gov/downloads/ti04009_ATR.doc ATR grantees are also required to collect data to ensure intended beneficiaries receive identified resources. Grantees are required to track each voucher issued and report fiscal and outcome data related to the voucher while also tracking each client who receives a voucher. Voucher related data is collected on the Voucher Information data collection form (OMB No. 0930-0266). Grantees submit quarterly reports to SAMHSA that include information on these vouchers. Furthermore, grantees are required to take active steps to utilize electronic systems to monitor and prevent fraud and abuse including random audits and trend analyses of provider billings and service data.

YES 20%
Section 1 - Program Purpose & Design Score 100%
Section 2 - Strategic Planning
Number Question Answer Score
2.1

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The concept of recovery is central to the goals of the ATR Program. A primary principle of maintaining successful recovery is abstinence from alcohol and drug use. As part of SAMHSA's Programs of Regional and National Significance, ATR is using the following long-term measure: Individuals who have received drug treatment services that show no past month substance use. In addition, a central principle of ATR is expanding treatment capacity. To assess the achievement of this goal, SAMHSA is using the number of clients served as an additional long-term measure.

Evidence: Measure 1: Percentage of individuals who have received drug treatment services that show no past month substance use. ["No past month use" is no use of drugs or alcohol in the last 30 days]. Measure 2: Number of clients served through ATR. ["Clients" is defined as an unduplicated count of individuals who have received an ATR service voucher]. In addition to these two long-term measures, SAMHSA also tracks data on the number of participating faith and community-based providers, as well as program oversight efforts (i.e., site visits, random audits, and trend analysis). The 2004 and 2007 Requests for Applications include additional outcome data reported by grantees to measure the effectiveness of the program. http://atr.samhsa.gov/downloads/ti04009_ATR.doc and http://www.samhsa.gov/Grants/2007/TI_07_005.pdf .

YES 12%
2.2

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: SAMHSA has set ambitious targets for both of its long-term measures; the percent of users with no past month substance use; and the number of persons served. Although the targets for the number of clients served is slightly lower than the first cohort, the target of 160,000 persons served over three years is ambitious given the higher percentage of Methamphetamine clients that may be served within the second cohort. Clients with an addiction to Methamphetamine tend to utilize services that are more intensive and have a higher cost per person than the average ATR client.

Evidence: (See Measures in Performance Section). The second cohort of ATR awards includes a target of $25 million for services for methamphetamine users. Data from the first cohort of grantees shows that 77.1 percent of methamphetamine users had no past month substance use at discharge, compared to 81.4 percent of all clients, a 5.3 percent lower outcome. Therefore, a gradual increase in targets for past month substance use at discharge in the second cohort represents a modest target. Data through December 2006 show that ATR has exceeded its target of clients served of 87,500. ATR has served over 137,500 clients since its inception through December 31, 2006. This represents a coverage rate of 157%. In other words, 57% more clients have been served than were originally projected. SAMHSA intends to serve 160,000 clients in the second cohort.

YES 12%
2.3

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ATR has a limited number of annual measures that directly correlate with the long-term measure of abstinence from substance abuse. These measures serve a core function in assessing the effectiveness of substance abuse treatment and recovery support services. The measures are primarily outcome measures and correlate directly with the long-term measure of abstinence from substance use. One of the key objectives of ATR is also to enhance accountability, as grantees are required to manage performance, based on outcomes that demonstrate patient successes. In addition to reporting on annual measures, ATR grantees provide data on key elements relating to the goals and objectives of the program such as the extent to which recovery support services are offered and the participation of faith-based organizations. These data allow SAMHSA to directly measure whether or not grantees are meeting the program purpose. ATR also measures efficiency via the average cost per client served. SAMHSA is working to refine the efficiency measure to incorporate both cost and quality drivers specifically related to ATR.

Evidence: The annual measures are the percentage of individuals receiving services who: 1) had no past month substance use, 2) had improved family and living conditions, 3) had no/reduced involvement with the criminal justice system, 4) had improved social support, and 5) were currently employed or engaged in productive activities. The annual measures are reported in SAMHSA's FY 2008 Congressional Justification http://www.samhsa.gov/Budget/FY2008/index.aspx and are also contained in the 2004 and 2007 Requests for Applications (http://atr.samhsa.gov/downloads/ti04009_ATR.doc [Section 3.1], and http://www.samhsa.gov/Grants/2007/TI_07_005.pdf [page 7-10]. In addition, to these annual measures, SAMHSA also tracks data on the number of participating faith and community-based providers, as well as program oversight efforts (i.e., site visits, random audits, and trend analysis). The 2004 and 2007 Requests for Applications includes additional outcome data reported by grantees to measure the effectiveness of the program. http://atr.samhsa.gov/downloads/ti04009_ATR.doc and http://www.samhsa.gov/Grants/2007/TI_07_005.pdf .

YES 12%
2.4

Does the program have baselines and ambitious targets for its annual measures?

Explanation: ATR has baseline data and targets for all annual measures. The 2007 targets for the ATR measures are 81% for abstinence, 50% for employment, 52% for stable living, 97% for criminal justice involvement, and 90% for social support. SAMHSA annually reviews these targets and manages program changes as appropriate. Methamphetamine clients will account for a larger percentage of clients in the second ATR cohort, which begins in FY 2007. Therefore, the targets for FY 2008 - FY 2010 have decreased slightly to reflect the challenges of achieving outcomes for this population. Previous data from the first cohort of ATR grantees showed differences in outcome data between "average" methamphetamine client and the "average" ATR client. Targets for all ATR measures have been set at least equal to current (2006) performance levels and most of these targets are ambitious given the demonstrated challenge of achieving behavioral outcomes among methamphetamine users.

Evidence: The second cohort of ATR awards includes a target of $25 million for services for methamphetamine users. Data from the first cohort of ATR grantees show that Methamphetamine clients tend to have lower outcomes in comparison with the average ATR clients. Data from the first cohort of ATR grantees show that 77.1 percent of methamphetamine users had no past month substance use at discharge compared to 81.4 percent of all ATR clients. In addition, 35.5 percent of meth clients had improved family and living conditions versus 51 percent of all ATR clients and 95.2 percent of methamphetamine clients had no involvement with criminal justice versus 96.8 percent of all ATR clients. For the percentage of clients currently employed or engaged in productive activities, 40.9 percent of meth clients were currently employed or engaged in productive activities versus 50 percent of all clients. Methamphetamine clients had the same percentage of improved social support outcomes as the average ATR client.

YES 12%
2.5

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All partners involved in the ATR Program commit to and work toward both the annual and long-term goals of the Program. All ATR grantees submit data quarterly on each of the specified measures utilizing the SAMHSA's Services Account Ability Improvement System (SAIS), an online data entry and reporting system. SAIS also includes detailed reports on whether grantees are compliant with data reporting. For example, prior to the grant award, SAMHSA negotiates a target number of clients to be served by each grantee. SAMHSA utilizes the SAIS reports to monitor whether or not grantees are meeting those targets. Grantees have also emphasized the importance of outcome data reporting and incorporated this into their own practices. For example, many grantees have implemented a voucher reimbursement system that is directly tied to data reporting (i.e., vouchers are not paid until providers comply with data reporting requirements). Finally, SAMHSA has revised the FY 2007 RFA to include supplemental awards based on performance (depending on availability of funds).

Evidence: The 2004 and 2007 Requests for Applications require grantees to report on each of the annual measures (http://atr.samhsa.gov/downloads/ti04009_ATR.doc [Section 3.1] and http://www.samhsa.gov/Grants/2007/TI_07_005.pdf [pages 7-10 and 62-81]). The Services Accountability Improvement System (SAIS) serves as SAMHSA's central data repository and contains the requirements and instructions for submitting data as well as the data collection tool required for the ATR Program. Grantees submit performance data through a web-based system at https://www.samhsa-gpra.samhsa.gov.

YES 12%
2.6

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: SAMHSA will contract with an independent entity in FY 2007 to conduct a high quality, unbiased evaluation of ATR. This cross-site evaluation will include outcome, cost effectiveness components, and examine rates set for recovery support and other clinical services. The evaluation will also determine the impact ATR has had on the existing treatment system, including barriers and challenges of implementing a voucher management system. In addition to this independent evaluation, the Office of Inspector General (OIG) has proposed to evaluate the early implementation of ATR in their FY 2007 and FY 2008 work plan. The objective of the OIG evaluation would be to provide SAMHSA with an early look at how states have implemented the Access to Recovery Program including a review of program oversight efforts. In the interim, the majority of current ATR grantees are conducting need assessments/evaluations of their projects to make program improvements, evaluate effectiveness, and determine the impact ATR has had on client outcomes and substance abuse treatment systems.

Evidence: The Statement of Work for the cross-site ATR evaluation shows sufficient rigor and comprehensiveness and will be conducted by an independent entity. The OIG work plan appears at http://oig.hhs.gov/publications/docs/workplan/2007/Work%20Plan%202007.pdf Other grantee evaluations include: California Rural Indian Health Board (CRIHB). EMT Associates, Inc, an independent evaluator was contracted by CRIHB to perform ongoing client and provider assessments. Connecticut. Yale University is conducting an evaluation to determine which clinical and recovery support services, or combination of both, results in the best outcomes for the grantees' target population. Florida. Through the University of Miami's Miller School of Medicine, Florida is conducting a continuous process evaluation that examines the outcomes of the ATR services delivered, provider satisfaction, and participant satisfaction. Missouri. The University of Missouri-Columbia's Institute of Mental Health is conducting an outcome evaluation study on the Missouri ATR project. New Mexico. Through its administrative services organization (ASO), Value Options, New Mexico ATR has conducted comprehensive evaluations on an ongoing basis. Tennessee. The University of Memphis' Institute for Substance Abuse Treatment Evaluation continues to conduct a statewide outcomes evaluation study. California. The grantee plans to conduct an internal final report that will include outcome data from the GPRA measure, service invoice data, the internal Customer Satisfaction Survey, and a focus group report. Washington. The State is currently negotiating funding for an evaluation of the ATR program. The grantee has indicated that it is planning to examine standard cost offsets for those receiving ATR with some analysis of which type of RSS (if any) correlate with more positive outcomes. Idaho. Through its contractor, Business Psychology Associates (BPA), the State of Idaho instituted a continuous evaluation process that permits state-level staff to have access to crucial data about the program and its impact over time. Illinois. This grantee has implemented a continuous consumer satisfaction survey to capture participating clients' perceptions of and satisfaction with the services they receive. Louisiana. LA-ATR is in the planning stages of preparing a comprehensive, statewide process and outcomes evaluation. New Jersey. New Jersey has instituted three client-centered evaluation activities to ensure that feedback is continually garnered and used to improve program processes. Texas. Through the University of Texas Gulf Coast Addiction Technology Transfer Center (ATTC), Texas ATR is examining client outcomes to provide preliminary information regarding program effectiveness. ATR clients who received substance abuse treatment were compared to non-ATR criminal justice and non-criminal justice clients receiving state-funded treatment services.

YES 12%
2.7

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: It is not clear from the Justification how funding is directly correlated to ATR annual and long-term performance outcomes. Nor does SAMHSA provide sufficient evidence that the resources/performance mix is the most appropriate ratio. Finally, the FY 2008 Budget Justification has not attributed the full cost to individual measures.

Evidence: SAMHSA's FY 2008 Congressional Justification http://www.samhsa.gov/Budget/FY2008/index.aspx

NO 0%
2.8

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: SAMHSA constantly monitors ATR to ensure project improvements. For example, SAMHSA has held over 130 site visits and conferences to ensure the exchange of "lessons learned" among grantees. Bi-annual conferences discuss fiscal management practices, data reporting requirements (e.g. the Financial Status Report), and how to strengthen client and provider outreach. SAMHSA has also adopted a limited number of specific long-term performance goals to document progress. In addition to these long-term goals, SAMHSA also monitors quarterly updates of faith-based providers, recovery support services, and vouchers redeemed. SAMHSA has also significantly revised the FY 2007 RFA to strengthen program oversight efforts and data management. For example, the 2007 ATR grantees will have a required 6-month follow-up, and will set targets for these measures based on baseline data (expected 12/08). In addition, the revised FY 2007 RFA requires grantees to specify how they will create incentives for positive outcomes/improved efficiencies (e.g., adjusting provider reimbursement based on outcomes). SAMHSA will also assess upcoming ATR grantee performance using a number of accountability measures included in the 2007 RFA including possible supplemental awards based on performance.

Evidence: Grantees complete a standard monthly status report with information on vouchers, collaboration, challenges, and implementation issues, as well as financial information, which are incorporated into quarterly profiles for each grantee. The requirement for 6-month follow-up and the incentives are described in the 2007 RFA: page 11-12 (http://www.samhsa.gov/Grants/2007/TI_07_005.pdf ) SAMHSA's FY 2008 Congressional Justification contains the long-term and annual goals for Access to Recovery regarding capacity and helping clients achieve abstinence from alcohol and drug use. http://www.samhsa.gov/Budget/FY2008/index.aspx

YES 12%
Section 2 - Strategic Planning Score 88%
Section 3 - Program Management
Number Question Answer Score
3.1

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ATR requires grantees to collect quarterly data on outcome and output measures utilizing a uniform data collection tool. In addition to outcome and output data, SAMHSA collects and monitors data from grantees on fraud and abuse, client characteristics, the number of vouchers issued, and other administrative data. SAMHSA uses these data to set performance targets, adjust program priorities, allocate resources, and take appropriate management actions where appropriate. Grantees also submit quarterly reports on the progress of their programs in addition to the electronic performance data submission. This allows grantees the opportunity to provide formal information to their Project Officer on information not captured in the performance data. SAMHSA then combines data into the Program Profiles, which show the progress of each grantee on a quarterly basis.

Evidence: SAMHSA collects timely performance information from program partners through their real-time data entry and reporting system, the Services Accountability Improvement System (https://www.samhsa-gpra.samhsa.gov). Standard data collection tools can be found in the 2007 RFA (pages 63-88, http://www.samhsa.gov/Grants/2007/TI_07_005.pdf) Grantees also submit quarterly reports to SAMHSA and communicate frequently by email when problems arise. For example, if a grantee has difficulty-recruiting clients, the SAMHSA Project Officer will become aware of this through a system report. The Project Officer then works with the grantee to create a corrective action plan, which may include targeted technical assistance to the grantee. Through this data collection, SAMHSA noticed that faith-based organizations were not being adequately recruited and enrolled. To increase capacity of these providers, SAMHSA contracted with Faith Based Transition Coordinators and liaisons to assist the ATR grantees in the effort to recruit more faith-based providers. As a result, the number of faith-based organizations providing ATR services has increased from 940 to 1,051 over the last three quarters. .

YES 10%
3.2

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Program managers are held accountable for achieving key program results through annual reviews. If program staff have not been successful for that performance year, action plans are created as appropriate. Staff in turn manages program results through quarterly submissions to the Payment Management system (PMS), which indicate drawdown of Federal Funds. To enhance accountability, the new ATR RFA also indicates that annual continuation awards will be contingent on the availability of funds, grantee progress in meeting project goals and objectives, timely submission of required data and reports, and grant expenditures. The new RFA also includes supplemental awards for the strongest performers (contingent upon available funds). SAMHSA will review each grantee's Government Performance and Results Act (GPRA) data submissions and assess whether a grantee has: 1) met or exceeded its target for the number of clients served by 25 percent or more, 2) met or exceeded its target for 6-month follow-ups, and 3) provided services within approved cost-bands.

Evidence: Grantees are held accountable for performance through the setting of grantee level client targets, which are negotiated with each grantee and formalized in the Notice of Grant Awards document. Government employees are held accountable for program performance by the Performance Management Appraisal System and through their individual performance plans. Data and reporting requirements for grantees are documented in the 2004 RFA (page 21, http://atr.samhsa.gov/downloads/ti04009_ATR.doc). Evidence of these data submissions can be found in the quarterly program profiles compiled by SAMHSA staff. A new performance mechanism to reward the strongest performing grantees has been incorporated into the 2007 RFA (pages11-12, http://www.samhsa.gov/Grants/2007/TI_07_005.pdf)

YES 10%
3.3

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: ATR grantees have not disbursed funds in a timely manner. As of August 2, 2006 ATR grantees had an average unobligated balance from Year 1 to Year 2 of $5.0 million per grantee. As of Year 2 and Year 3, grantees still had an average unobligated balance of $1.2 million. Random and targeted audits have also detected some erroneous payments. For example, some grantees have found inappropriate provider billings, services inconsistent with the State's Recovery Support credentialing guidelines and that facilities did not meet local codes. Since vouchers were a new mechanism for States, the establishment of the infrastructure to handle vouchers has taken longer than SAMHSA had anticipated. Although grantees did not disburse funds in a timely manner, Federal funds were obligated in accordance with Federal Appropriations law.

Evidence: Grantees are required to provide Financial Status Reports (SF269) to SAMHSA on a regular basis. General information on this reporting requirement can be found http://www.samhsa.gov/grants/management.aspx and a copy of the reporting form is available at http://www.gsa.gov/Portal/gsa/ep/formslibrary.do?viewType=DETAIL&formId=91E81CF31C11935585256AA100426D84. HHS Core Accounting Report. The HHS GPS can be found at http://www.hhs.gov/grantsnet/docs/HHSGPS_107.doc

NO 0%
3.4

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ATR has several management procedures in place to measure efficiency. Major procedures implemented to achieve efficiencies in ATR include an approved efficiency measure, caps on administrative funds for the 2007 cohort, and incentives to enhance efficiency and performance. The efficiency measure, "Reduced average client cost per person served" is an incentive for grantees to manage program expenses and provide services more efficiently i.e., administrative costs). The 2004 ATR RFA capped administrative costs at 15 percent. The 2007 RFA expands upon this language, setting a target for pervious grantees with new awards at 10 percent of their award. In addition, the 2007 RFA adds a funding incentive for grantees to achieve additional efficiencies in their program. SAMHSA is also working to ensure that grantees are providing the most effective and cost efficient recovery support services. Grantees are currently given wide latitude in determining what constitutes a recovery support service and it is unclear whether grantees are appropriately reimbursing providers. SAMHSA is currently reviewing the most effective recovery support services.

Evidence: SAMHSA will report the new efficiency measure in the FY 2009 Budget. The 2004 ATR RFA informs all grantees about the cost bands that service providers will be expected to work within (page 52, http://atr.samhsa.gov/downloads/ti04009_ATR.doc) as well as requiring grantees to provide details in their application of how they will control costs, prevent fraud, waste and abuse, and maintain administrative costs (pages 13-14). The 2007 RFA for ATR adds an additional requirement that current grantees limit administrative costs of the program to 10-15% (pages 12-13, http://www.samhsa.gov/Grants/2007/TI_07_005.pdf)

YES 10%
3.5

Does the program collaborate and coordinate effectively with related programs?

Explanation: SAMHSA has set up linkages between ATR and the Department of Labor's (DOL) Prisoner Re-entry Initiative (PRI); however, the program has not established meaningful collaborations with other Federal, State, and local programs to realize the full benefits of the program. Even within the PRI program, it is unclear whether ATR clients have access to PRI services provided within the DOL. SAMHSA has not developed formal policies, or recruitment/referral processes with the Department of Justice, Department of Housing and Urban Development (HUD) to maximize resources. For example, HUD and other HHS programs may reimburse for some of the same housing and recovery support services, yet there are no formal linkages and/or referrals between these two Agencies.

Evidence: While grantees report referrals of PRI clients to ATR services and vice versa, there are no similar linkages between the Department of Justice and Department of Housing and Urban Development (HUD). To support the development of collaborative relationships between ATR and PRI, several grantees provided presentations at the most recent ATR grantee meeting (January 2007 meeting agenda) (Examples: Department of Labor, Connecticut, Wisconsin, Community Partners in Action). However, there is no evidence that ATR is managing resource allocations with related and similar activities across Federal, State, and local programs.

NO 0%
3.6

Does the program use strong financial management practices?

Explanation: There are wide variations on financial management controls across ATR grantees. Random and targeted audits have detected erroneous payments within ATR. Within the first year of the program, there were more than 30 erroneous practices, including some billing errors, overpayments, and billing for services that were not appropriate or necessary, that were rectified and reimbursed. For example, in some states, inappropriate provider billings were discovered and services were found to be inconsistent with the State's Recovery Support credentialing guidelines and facilities did not meet local codes. Not all of the 15 grantees have random audits or crosschecking payments system. In one State, five different site visits found that providers were ineligible because of weak financial management practices, eligibility criteria and/or other operational issues. Some grantees have also documented that the current voucher system has allowed providers to inappropriately backdate a voucher request in the system, and allowed unauthorized providers.

Evidence: Grantee and State Audits and Financial Management Practices from SAMHSA ATR Conference.

NO 0%
3.7

Has the program taken meaningful steps to address its management deficiencies?

Explanation: ATR has multiple systems in place to identify and correct deficiencies that may arise. Government project officers on the ATR grants in collaboration with grants management staff, oversee the ATR grants through an extensive set of required reports. SAMHSA also provides technical assistance to individual grantees to ensure successful projects and develop corrective action plans where appropriate. Financial data is used by SAMHSA to monitor costs and ensure funds are being used for appropriate and intended purposes. SAMHSA has also taken meaningful steps to strengthen the ATR grant announcement and has significantly revised the FY 2007 RFA.

Evidence: The grantee monthly reporting template requires grantees to provide ongoing information on administrative costs, voucher issuance and redemption, as well as total obligated dollars to date. SAMHSA staff monitors data submitted by grantee monthly reports to identify obstacles in obtaining program goals and/or other reporting issues. As needed, SAMHSA staff work with grantees that are not able to meet these requirements and develop Corrective Action Plans. Recent interventions with New Jersey are an example of how SAMHSA corrects grantee deficiencies in relation to program goals and the timely obligation and expenditure of grant funds. SAMHSA worked with New Jersey to submit a corrective action plan (with deadlines) and is currently monitoring progress to ensure changes were implemented.

YES 10%
3.CO1

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The ATR grants are awarded in a clear and competitive process that includes a qualified assessment of merit. SAMHSA seeks reviewers knowledgeable in the field to objectively score the application based on set review criteria. The grantee applications are reviewed and assigned a point value based on how well they met the criteria for responses to the following five sections outlined in the RFA: A) statement of need, B) project plans to meet original ATR goals, C) proposed approach, D) readiness to implement/expand the voucher system, and E) management, staffing and cost controls. These criteria must be met to receive full points on the five sections. For example, in Section A, applicants are required to describe the current substance abuse treatment system including current providers and availability of recovery support services, information on the nature and prevalence of substance abuse problems in the target area, and the need for methamphetamine-specific service provision as well as other key pieces of information. Applicants scores are compared with each other and the best applications are typically awarded the grant funds. Prior to the initial release of the Request for Application, SAMHSA also holds numerous technical assistance conferences and meetings around the country, and maintains a telephone help-line to answer any questions regarding the program.

Evidence: The competitive review process is an extensive process. The selection criteria for applicants are articulated in both the 2004 (pages 17-20, http://atr.samhsa.gov/downloads/ti04009_ATR.doc) and 2007 RFAs (pages 22-30, http://www.samhsa.gov/Grants/2007/TI_07_005.pdf).

YES 10%
3.CO2

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: SAMHSA has sufficient oversight of ATR grantees. Project management ensures the ATR grantees are adhering to established program goals. The Center for Substance Abuse Treatment (CSAT) regularly collects timely and credible performance information for the ATR grantees and uses it to manage the program and to improve performance. On a quarterly basis, ATR Grantees must report financial and outcome data to SAMHSA. These reports include fiscal items such as administrative costs as well as many other reporting categories, which indicate how the grantee is achieving the ATR programmatic goals (e.g. clients served, vouchers issued, providers recruited, faith-based participation, etc.). To ensure data quality, the SAIS system has automatic validation and verification checks. SAMHSA also provides targeted technical assistance, which has included developing appropriate incentives to encourage high quality provider performance, provider recruitment and outreach assistance, voucher management, and fiscal management. SAMHSA has provided 253 technical assistance events including 125 site visits to grantees and 128 conference calls with grantees.

Evidence: Grantees are required to comply with several reporting requirements in the 2004 RFA (page21, http://atr.samhsa.gov/downloads/ti04009_ATR.doc) including the provision of monthly reports. Grantees are monitored through their required data submission through the Services Accountability Improvement System (SAIS) standard instruments (available at https://www.samhsa-gpra.samhsa.gov). Evidence of these data submissions can be found in the quarterly program profiles compiled by SAMHSA.

YES 10%
3.CO3

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects grantee performance data on a quarterly basis and aggregate data are available to the public on SAMHSA's website. In addition to these reports, ATR grantees submit GPRA outcomes data to SAMHSA via the online Services Accountability Improvement System (SAIS) system. SAMHSA also requires monthly reports from ATR grantees on their programmatic activities. These reports include detailed information on vouchers as well as fiscal data such as administrative costs, vouchers issued, expired vouchers, paid claims, and services provided but not yet paid. All these data and information are shared with the public and grantees via several different channels.

Evidence: Information is posted on the SAMHSA ATR website at the following link: www.ATR.SAMHSA.gov. In addition, performance data for the program as a whole is reported in the annual SAMHSA budget justifications, available at http://www.samhsa.gov/Budget/FY2008/index.aspx . ATR aggregate data is posted on the SAMHSA public web site http://atr.samhsa.gov/AggregateDataProfiles.htm.

YES 10%
Section 3 - Program Management Score 70%
Section 4 - Program Results/Accountability
Number Question Answer Score
4.1

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The ATR Program has a long-term goal of increasing the percentage of adults receiving services who had no past month substance use to 82 percent by 2010. The 2005 baseline was 78 percent, and the 2006 actual was 81.4 percent. The second long term measure: the number of clients gaining access to treatment also represents ambitious targets, of which SAMHSA has already exceeded for FY 2006.

Evidence: SAMHSA's FY 2008 Congressional Justification: The Congressional Justification contains the reported progress on the percentage of clients reporting abstinence from alcohol and drug use. http://www.samhsa.gov/Budget/FY2008/index.aspx (PD 35)

LARGE EXTENT 17%
4.2

Does the program (including program partners) achieve its annual performance goals?

Explanation: ATR has been successful in meeting or exceeding five of its seven performance targets. Grantees have shown a commitment to working toward each of the outcomes set forth in the RFA. The output measure of clients served was exceeded showing a marked effort to increase capacity. The annual targets for abstinence (also long-term measure), and criminal justice involvement, were all exceeded with reported performance at 81.4 percent and 97 percent respectively. The target for social support was met at 90 percent reporting social support. The two targets which were not met were employment and stable living.

Evidence: SAMHSA'S FY 2008 Congressional Justification: Performance on each of the annual measures for ATR is reported in the Congressional Justification.

LARGE EXTENT 17%
4.3

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: SAMHSA has achieved several efficiencies in its management of the ATR program, which include a new cost efficiency measure for the ATR program, "Reduced average client cost per persons served." This measure will provide an incentive for grantees manage program expenses (i.e., administrative costs) and provide services more efficiently. A national substance abuse treatment cost study, the Alcohol and Drug Services Study (ADSS) quotes the average cost of services at approximately $1850 per admission. Data on ATR show that the Program is currently below this rate at $1605 per client. This represents an average cost savings of $250 per client. In addition, SAMHSA is actively monitoring the spending of award funds on administrative expenses. Beyond requiring a cap on administrative costs in the 2004 and 2007 RFAs, SAMHSA has been monitoring administrative spending for each grantee regularly. During Year 1 of the 2004 cohort, the average administrative cost among the 15 grantees was over 15 percent. By year two of the program, average administrative spending was less than14 percent of their award. In addition, SAMHSA has included incentives in the FY 2007 RFA to improve cost savings in this program over time.

Evidence: ATR 2004 RFA: The ADSS Cost Study: Costs of Substance Abuse Treatment in the Specialty Sector: A major component of the Alcohol and Drug Services Study (ADSS) was the assessment of substance abuse treatment costs. http://www.oas.samhsa.gov/ADSS/ADSSCostStudy.pdf (p. 21) The 2004 ATR RFA requires grantees to provide details in their application of how they will control costs, prevent fraud, waste and abuse and maintain administrative costs (pages 13-14). The 2007 RFA for ATR adds an additional requirements that current grantees limit administrative cots of the program to 10-15 percent. http://www.samhsa.gov/Grants/2007/TI_07_005.pdf

YES 25%
4.4

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: As discussed in Question 1.3, the ATR initiative is unique from other substance abuse treatment activities. No other program providing substance abuse treatment services offers services through a voucher-based mechanism. In addition, two other elements of the ATR Program make it significantly different from other treatment programs. These are: client choice and the more extensive participation of faith-based organizations. ATR is unique in its targeted effort to ensure client choice with the expansion of the faith-based provider network.

Evidence: As discussed in Question 1.3, the ATR initiative is unique from other substance abuse treatment activities. No other program providing substance abuse treatment services offers services through a voucher-based mechanism. In addition, two other elements of the ATR Program make it significantly different from other treatment programs. These include client choice and the more extensive participation of faith-based organizations. ATR is unique in its targeted effort to ensure client choice with the expansion of the faith-based provider network.

NA 0%
4.5

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Preliminary findings of two interim independent evaluations of individual grantees indicate that ATR is effective and achieving results. The grantees' evaluations are being conducted by independent entities such as local universities (e.g. Boise State University, Yale University, and University of Memphis) or consulting companies (e.g., EMT Associates and Business Psychology Associates). All evaluation activities have been designed for the specific purposes of making program improvements, and evaluating effectiveness. However, only two of the grantee evaluations (Connecticut and Texas) are far enough along to have interim results. In Connecticut, a high quality evaluation conducted by Yale University indicated that recovery support services (RSS) were more predictive than clinical services of decreases in alcohol and drug use, jail time and arrests, and of improvements in housing status and employment. More specifically, reductions in drug use were associated with alternative living centers, case management, and vocational services. The Texas ATR interim evaluation found that ATR clients achieved better outcomes in the areas of treatment completion, past month abstinence, and Alcohol Anonymous attendance at discharge relative to both non-ATR Department of State Mental Health Services criminal justice and non-criminal justice clients. Contrasts of ATR program completion and referral source groups indicated that retention in the ATR program, greater amounts of care coordination, and the provision of treatment only or treatment in combination with recovery support services were associated with positive outcomes.

Evidence: 1. ATR Profile 2. Connecticut Evaluation Information 3. Texas Evaluation information

SMALL EXTENT 8%
Section 4 - Program Results/Accountability Score 67%


Last updated: 09062008.2007SPR