Detailed Information on the
Mental Health Programs of Regional and National Significance Assessment

Developing efficiency measure

Developing client functioning long-term measure

Complete automated web-based performance system (TRAC) with common measures for technical assistance and infrastructure development.

Implementing automated web-based performance system including development and implementation of common performance measures

Measure: Number of a)evidence-based practices (EBP) implemented per State and b) percentage of service population coverage for each (reported as percentage of service population receiving any EBP, adults and children)

Explanation:The use of evidence-based practices measure the program's efforts to improve the efficiency and effectiveness of mental health services. Evidence based practices come from SAMHSA's National Registry of Effective Programs and Practices (NREPP), from SAMHSA produced toolkits (6 currently completed), and from other valid evidence that show that a practice is effective. Data will be collected by SAIS, once it becomes available. Baseline data are expected in September 2005.

Measure: Percentage of people in the United States with Serious Mental Illnesses in need of services from the public mental health system, who receive services from the public mental health system.

Explanation:This measure is intended to capture access to the public mental health system and includes people receiving services in state psychiatric hospitals as well as those receiving services through community mental health programs. The measure uses system-level data and does not reflect results from SAMHSA grantees. Denominator: This measure is derived from the National Co-morbidity Study Replication, a national survey on mental health prevalence that is expected to be conducted every ten years, and looks at the percentage of the population with a serious mental illness (http://archpsyc.ama-assn.org/cgi/content/full/62/6/593). Numerator: Number of persons receiving services through the state mental health system. Data definitions and guidance for the collection of this information is available at http://www.mhsip.org.

Measure: Increase the percentage of clients receiving services who report improved functioning.

Explanation:In December 2007, the TRAC system reporting capability was not complete which necessitated that the original baseline be calculated by hand. This calculation, however, was made using the wrong cut off score. The TRAC reporting capability is now complete and the correct cutoff score is programmed into reporting system which will prevent future errors in reporting.

Measure: Rate of consumers/family members reporting positively about outcomes. (Program Participants)

Explanation:Studies have found that patient's satisfaction with services was related to improvements as a result of treatment (Customer Satisfaction and Self-Reported Treatment Outcomes, Holcomb et al, 1998; Mental Health, United States, SAMHSA, 2000). This long-term measure reflects overall conditions in the States, and not individual PRNS grantees. State-wide standardized surveys on consumer outcomes are conducted by State Mental Health Authorities over the telephone, by mail, or in a peer-to-peer handout to consumers of State-run mental health services. This data is entered into the Uniform Reporting System (URS) which States submit each December in their Community Mental Health Services Block Grant (MHBG) Implementation Report. This measure is also used for the MHBG program. Currently, 44 States report on this measure and all States are expected to report by 2007.

Measure: Dollars spent per person served (National Child Traumatic Stress Initiative) (New measure, added February 2008)

Explanation:The efficiency measure is calculated by dividing the budget for the program by the target for numbers served.

Is the program purpose clear?

Explanation: The purpose of the Mental Health Programs of Regional and National Significance (PRNS) is to improve mental health services in the United States in order to help individuals achieve recovery and to transform the mental healthcare delivery system into one that can better support and sustain recovery for persons with or at-risk for mental illness. The program works to achieve its purpose through grants and contracts that support States and communities in carrying out an array of activities intended to improve mental health services. Program funds are divided into two categories: 1) Targeted Capacity Expansion, funding intended to improve the infrastructure of mental health delivery systems and to provide services to vulnerable populations, and 2) Best Practices, funding designed to identify and promote effective and efficient recovery-based service models.

Evidence: SAMHSA's mission statement and 2006 Congressional Justification discuss the purpose of the PRNS. The program is authorized in Section 520(A) of the Public Health Services Act, which directs SAMHSA to address priority mental health needs of regional and national significance through assistance for knowledge development, training, technical assistance, targeted capacity response, and system change grants. Sections 506, 581, and 582 also direct SAMHSA to address needs of homeless individuals with mental health problems, children dealing with violence, and persons suffering from trauma. The PRNS is the main vehicle for implementing the President's New Freedom Commission on Mental Health recommendations to improve the mental health delivery systems. (Achieving the Promise: Transforming Mental Health Care in America??Final Report of the President's New Commission on Mental Health, July 2003).

Does the program address a specific and existing problem, interest, or need?

Explanation: Mental disorders are prevalent and costly to society. The President's New Freedom Commission on Mental Health reported that "the mental health care system is fragmented and in disarray??lead[ing] to unnecessary and costly disability, homelessness, school failure and incarceration." The Commission developed six goals for a transformed mental health system: 1) Americans understand that mental health is essential to overall health; 2) Mental healthcare is consumer and family driven; 3) Disparities in mental health services are eliminated; 4) Early mental health screening, assessment, and referral to services are common practices; 5) Excellent mental health care is delivered and research is accelerated; and 6) Technology is used to access mental health care and information.

Evidence: Mental health system transformation is discussed in "Achieving the Promise: Transforming Mental Health Care in America" (July 2003). In June 2005, the National Institutes of Health (NIH) released results of the National Co-morbidity Survey Replication study which found that about 26% of adults are affected by mental disorders during a given year, and of those, 22% experience a disorder classified as serious. The study also found a median delay of about ten years between onset of disorders and treatment. Long delays in treatment can lead to more frequent and more severe episodes and make a disorder more resistant to treatment. Finally, the study found only a third of Americans seek help for a mental disorder receive treatment that meets professional standards. In 2001, an estimated $85 billion was spent on the treatment of mental health disorders in the United States, accounting for 6.2% of all health care spending (National Expenditures for Mental Health Services and Substance Abuse 1991-2001, SAMHSA, 2005).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The PRNS Program is the only federal program designed to improve mental health services delivery systems and to target priority mental health needs.

Evidence: Improving mental health services is a diverse and complex issue, and numerous organizations work individually and in conjunction with the PRNS to address aspects of the problem. Other federal programs address mental health, but have different areas of focus compared to the PRNS. NIH's National Institute of Mental Health which focuses on large-scale clinical trials to identify successful, evidence-based treatments, but does not promote field implementation. Medicaid, Medicare, and the Health Resources and Services Administration (HRSA) provide mental health services, but are not focused on nationwide improvement of mental health services. SAMHSA's PATH Homeless grants focuses on homeless outreach as opposed to providing mental health treatment and SAMHSA's Mental Health Block Grant is designed to help States improve planning. Many States fund similar projects (for example, suicide prevention), but these efforts are not focused nation-wide. Private organizations for example, Robert Wood Johnson Foundation funds mental health programs, but these efforts are more targeted than the PRNS Program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Current program design makes it difficult to assess the overall impact of the program. The PRNS Program engages in many very diverse activities and it is not evident that all the activities are the most effective or efficient methods of meeting the program's defined purpose and long-term performance goals. The PRNS is lacking in a clear design that links all program projects to its performance goals. The PRNS has 28 separate activities, of which many are so small that it is not clear how they will improve overall mental health service systems. In addition, 30% of PRNS funds support activities that do not directly contribute to the program's annual outcome goals. Program design can be improved by collecting and using performance information that: 1) demonstrates how all the program components help the program achieve its purpose; and 2) helps the program target funds to priority areas in order to have the greatest national impact.

Evidence: The impetus by which PRNS projects are developed comes from multiple sources: Congressional direction, the Administration, and program priorities. PRNS Programs are discussed in SAMHSA's 2006 Congressional Justification. Grant announcements, which discuss purposes of various activities, can be viewed at www.samhsa.gov/grants/index.aspx.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program accomplishes its goals primarily through competitive grants and performance-based contracts, which allows the program the flexibility to develop grant announcements that require applicants to address areas of greatest need.

Evidence: Applicants are required in the grant announcements to describe the need of the target population and proposed approach and evaluation plan. The competitive grant process allows the PRNS the flexibility to develop grant announcements that require applicants to address areas of greatest need. Grant announcements are designed so that appropriate entities are eligible. For example, although most announcements accept applications from State and local governments, community organizations, tribes, and nonprofits, the Mental Health Transformation State Incentive Grants are open only to Governors' offices (and Tribal equivalents) since they require development of a comprehensive statewide mental health plan.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has two long-term measures and is developing one more long-term measure that reflect on the program's goals to transform mental health services. A transformed mental health service system would demonstrate increased recovery and increased access. The program's measures are intended to demonstrate improved client outcomes resulting from treatment and improved access to the mental health system for those needing treatment. The program uses system-level data for these measures which does not reflect results from individual SAMHSA grantees.

Evidence: The program's three long-term measures are: (1) Rate of consumers/ family members reporting positively about outcomes (State mental health system); (2) Percentage of people in the United States with Serious Mental Illnesses in need of services from the public mental health system, who receive services from the public mental health system, and (3) Client functioning (developmental)

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has established targets for the majority of its long-term measures. The program has established ambitious targets and timeframes for the two long-term measure for which it has data. The program will develop long-term targets for its remaining measure by FY 2007.

Evidence: The program's goal is for 75% of adults and 68% of family members (for children and adolescents) to report positively on outcomes by 2008 (an increase from the respective baselines of 70% and 63% in 2002). This target is also used by SAMHSA's Mental Health Block Grant. The program's also has a long-term goal to increase access to the public mental health system to 50% from a baseline of 44%. The program will establish targets for its long-term measures to improve client functioning by FY 2007.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has three annual measures that reflect on outcomes of direct grantee activities of mental health services. Data to report on these measures will be primarily collected from the grantees that participate in the program's services activities??approximately 60% of the program's budgeted activities. (Of the remaining program activities, about 30% fund infrastructure building activities, 5% fund technical assistance activities, and 5% fund evaluation activities).

Evidence: The program's three annual measures are: (1) Rate of consumers/family members reporting positively about outcomes; (2) Number of evidence-based practices (EPB implemented and percentage of service population coverage for each); and (3) Cost measure (developmental)

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has targets but not baselines for its annual measures.

Evidence: The program has not begun collecting performance data on its annual measures, It expects to fully collect performance measures by FY 2007 and will be able to establish baselines. The program uses system-wide data collected by State mental health authorities to set current program targets.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Some, but not all, grantees and contractors measure and report on their performance as it relates to program goals.

Evidence: The program is in the process of developing a set of core outcome measures which eventually all grantees and contractors will be required to report.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: GAO recently completed an evaluation of SAMHSA's grant-making activities (Substance Abuse and Mental Heath Services Adminstration, Planning and Changes and Future Workforce Needs is Incomplete, June 2004) . The PRNS is planning or in the process of undergoing independent evaluations for about 9 of its 28 activities. The program has recently completed independent evaluations for 4 other activities. These evaluations are conducted by independent contractors chosen in a competitive process using a selection panel of individuals not associated with the program. Most of the evaluations used methodologies that relied on surveying program participants (e.g. traumatic stress, chronic homelessness), although some have used trials with controlled comparison groups (e.g. women with co-occurring disorders). In addition, the program working to improve its evaluations by developing a strategy to ensure high standards and quality. The strategy should be implemented in FY 2006.

Evidence: SAMHSA evaluations are planned for the following activities: the Garrett Lee Smith Suicide Memorial Grants, Co-occurring State Incentive Grants, and Safe Schools Health/Students Grants. Evaluations are in progress for the following activities: Jail Diversion, Traumatic Stress, Homelessness, Older Adult, and Suicide Hotline. Evaluations have been completed for the following activities: women co-occurring disorders, ACCESS, Consumer-Operated Services, and PRISM-E. In FY 2005, SAMHSA developed an Evaluation Workgroup that will (1) examine whether current evaluations of programs are of sufficient scope and quality to support the identification of program improvements and to evaluate effectiveness; and (2) develop recommendations to ensure that the future selection of program category-specific evaluation methods provide the most rigorous evidence of program effectiveness that is appropriate and feasible. The Workgroup expects to issue recommendations and guidelines by the end of FY 2005.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program does not provide a budget presentation that clearly ties the impact of funding decisions on expected performance or explains why the requested performance and resource mix is appropriate.

Evidence: SAMHSA's 2006 Congressional Justification lays out and discusses the Mental Health PRNS budget and performance information.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program is working on developing uniform performance measures, so the performance across PRNS activities can better be reflected. In the past year, the program has developed new long-term and annual performance measures for its services programs (approximately 60% of funding) that better reflect the program's purpose and activities of grantees. The program is working on developing performance measures for all of its infrastructure (approximately 30% of funding) and technical assistance (approximately 5% of funding) programs.

Evidence: The program has recently taken action to inventory all of its programs to see what data are being collected and what data would be most useful, The program is developing a comprehensive web-based performance system, Services Accountability Information System (SAIS), that involves the development and implementation of common performance measures that will be used by all PRNS activities.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects some performance data that relates to program goals for most, but not all, of its grants and contract projects. Data collection has been irregular across programs and the program has not used performance data from grantees to establish baselines. All projects submit reports at least annually, and program managers use performance data from these to provide technical assistance as needed and to redesign and adjust directions of programs. The program has not historically terminated grants as a result of performance information, but it does have the ability to place grantees on "high risk" status, whereby grantees are required to submit supporting information and to seek approval for all payment requests.

Evidence: The program has 28 different projects funded by grants and contracts. Performance data collected in reports varies by project, with some programs reporting more performance information than others. Data collected includes budget information, staffing changes, staff training and number of participants recruited/served (if relevant) and progress made in meeting program objectives, and other program relevant information/requirements. The program is working to make sure that all projects will be reporting on some measures by FY 2007.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: While there are clearly defined performance standards in SAMHSA program managers contracts, the program lacks specific performance standards for program partners. The program does not currently collect performance data for all of its infrastructure and technical assistance projects. Not all grantees are required to set performance targets by which there progress can be measured. Grantees compete for funds initially, but grantees lose funding only in extreme cases. Poor performance results in technical assistance to help bring grantees into compliance with program objectives. Lack of compliance with regulations can result in a grantee being designated as high risk. About six grantees in the past few years have been designated as high risk.

Evidence: SAMHSA Performance Plans and Contracts for the Administrator, Director of Mental Health Services, and program managers performance contracts contain specific goals related to PRNS program activities, including mental health system transformation, HIV/AIDS, trauma, and school violence. The program will require all partners to establish specific performance targets by which it can track grantee performance progress through the development of the new SAIS performance system.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program usually obligated funds within a few weeks after an application has been reviewed and approved. Program obligations of funds are monitored and tracked through SAMHSA's CORE accounting system. Financial reports submitted by grantees report on how an when funds are being spent. Through progress reports and periodic site visits, project officers monitor programs to ensure that funds are spent for their intended purpose.

Evidence: All PRNS applicants are required to submit a proposed budget for each year of the grant when they submit their application for a grant award. Both project and grants management staff work with grantees (successful applicants) to finalize the budget (spend plan). Any change in approved personnel, or significant change in proposed activities must be approved in advance by SAMHSA. Project Officers monitor grantee performance and expenditures through periodic progress reports. Each year grantees submit Financial Status Reports (FSRs) which are entered into the Grants Information Management System. These reports are compared against the approved budget and against the disbursements made by Payment Management.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The PRNS Program does not have procedures in place to measure efficiencies. While results have not been quantified, the program has worked to improve efficiencies by allowing most grantees to apply online at www.grants.gov. SAMHSA has also competitively sourced some of its grants review process.

Evidence: Strategic Management of Human Capital (2005) discusses efficiency plans with competitive sourcing and IT. The program is working on developing a cost efficiency measure and expects that it will be able to report on the measure beginning in FY 2008.

Does the program collaborate and coordinate effectively with related programs?

Explanation: SAMHSA works with other Federal, State, and private programs. SAMHSA was part of the President's New Freedom Commission on Mental Health, which comprised of Federal, State and private agencies. The Commission developed a report on the state of mental health care and provided advice on how to change it. The PRNS also works with the Department of Justice (DOJ) and the Department of Education (DOE) on the Safe Schools/Healthy Students Initiative, DOJ on jail diversion, and the Health Services and Resources Administration on the Collaboration to link Health Care for Homeless Programs and Community Mental Health Agencies. The program also collaborates with the Department of Housing and Urban Development and the Veterans Administration on issues relating to chronic homelessness. Finally, the program works with the National Institute on Mental Health to support the implementation of evidence-based mental health treatment practices.

Evidence: The President's Commission on Mental Health issued its final report in July 2003 (Achieving the Promise: Transforming Mental Health Care in America). The program has a collaborative agreement with DOJ and DOE on the Safe Schools/Healthy Students Initiative. The agreement streamlines the grant application process enabling applicants to apply one time for funds across program areas including: school safety, drug and violence prevention, mental health treatment, and educational reform. The PRNS Jail Diversion program is dually funded and managed by SAMHSA and DOJ.

Does the program use strong financial management practices?

Explanation: SAMHSA receives unqualified audit opinions. In addition, SAMHSA conducts pre-award accounting system reviews on new grantees to determine whether they have an adequate financial management system. This review is conducted before an award may be made and examines the grantees' fiscal policies and procedures to determine overall adequacy/ability to handle federal funds, as well as relevant financial reports to determine the solvency of the organization. Organizations that do not submit financial information are designated "high risk."

Evidence: SAMHSA received unqualified audit opinions in FYs 1998-2002. In FY 2003, SAMHSA's financial statement was consolidated with other HHS operating statements. HHS received unqualified opinions for FYs 2003-2004. Currently, six grantees have been placed on "high risk" status.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has experienced deficiencies in data collections and is in the process of developing a web-based system to collect and manage data for all of its grants programs. The web-based data system is expected to enhance SAMHSA's ability to manage the program by providing real-time data.

Evidence: The program expects its SAIS system to be fully operational in FY 2007.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Approximately 86 percent of grants in 2005 were competitively awarded. The remaining 14% of grants represent Congressional earmarks, which undergo an objective review but are not ranked. Competitively-reviewed grants are reviewed by least three outside independent peer-reviewers and are scored based on criteria laid out in the grant announcement. Scores are aggregated and top scoring applications are recommended for funding. Initial awards are made for the first year of the grant only and funding of subsequent years is not automatic. Grantees must submit a continuation application to support requests for continued funding for each subsequent budget period (usually 12 months). Approval is contingent upon the program's progress and on compliance with grant terms and conditions. SAMHSA conducts outreach through its website and public events. Grant announcements and notices of funding availability are posted on SAMHSA's website and grants.gov and are also distributed to trade and constituent groups through press releases. In addition, SAMHSA discusses funding opportunities at conferences and public meeting. SAMHSA also provides grant-writing training and technical assistance workshops for community and faith-based workshops around the country several times each fiscal year, and training videos available on its website.

Evidence: Grant announcements can be found at www.samhsa.gov/grants/index.aspx. The grant review process is discussed in SAMHSA Grants Training manual available at alt.samhsa.gov/grants/TAManual/toc_module6.htm. In addition, SAMHSA provides web-based training videos at grantstraining.samhsa.gov/

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program has oversight practices for all of its grant and contract projects. All grantees submit reports, at least annually, that include budget, progress, emerging problems/issues, future plans, and additional material required to monitor specific programs. All grantees are required to submit a proposed budget for each year of the grant award. Any change in approved personnel, or significant changes in proposed activities must be approved in advance by the program. Each year, grantees submit financial status reports that are compared against the approved budget. In addition, the program makes regular site visits to about half of its projects and on an as needed basis for the remaining projects.

Evidence: Project officers oversee grants and contracts through phone contacts, site visits, and grantee meetings. Some projects have monitoring manuals and procedures. SAMHSA's Tracking and Reporting System (STARS) is used for for SAMHSA's Post Traumatic Stress Disorder projects. These auditing mechanisms allow for the methodical examination of grantee performance to determine the need for interventions designed to increase compliance and/or improve performance, including the provision of targeted technical assistance.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program does not make grantee performance data available to the public.

Evidence: The program plans to make data available once the SAIS data collection system is available.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has demonstrated partial progress on its long-term measure for which it has data. The program does not have data to demonstrate progress on its two developmental measures.

Evidence: The number of adults reporting positively on outcomes increased from 70% to 72%, while the number of family members (for children and adolescents) doing so decreased from 63% to 60%, from FY 2002 to FY 2003 respectively.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The programs has no data to report progress on its annual measures.

Evidence: The program should be fully reporting on its annual performance measures by FY 2008.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The PRNS Program does not have procedures in place to measure efficiencies. While results have not been quantified, the program has worked to improve efficiencies by allowing most grantees to apply online at www.grants.gov. SAMHSA has also competitively sourced some of its grants review process.

Evidence: Strategic Management of Human Capital (2005) discusses efficiency plans with competitive sourcing and IT. The program is working on developing a cost efficiency measure and expects that it will be able to report on the measure beginning in FY 2008.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There have been no comparisons made of the PRNS with other publicly-funded or privately-funded mental health programs. The closest program in terms of size and mission is the Mental Health Block Grant. Once the program reports on its performance measures comparisons can be made of the two programs. However, because the program distributes grants on a competitive basis versus the formula basis of the block grant, the PRNS has the mechanism to build more accountability in its grants.

Evidence: The Mental Health Block grant underwent a PART in the FY 2005 President's Budget and received an "Adequate" rating. The PART found that the program needed to work with the States to develop performance partnerships to increase State flexibility in exchange for program performance.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Of the four completed evaluations of program activities, three demonstrated positive results. However, all of the activities that have been evaluated are no longer being funded under the program. GAO found that SAMHSA improved its collaborative efforts with other agencies but could do more to diminish administrative hurdles to applying for competitive grants.

Evidence: The Women, co-occurring disorders study, Consumer-Operated Services Program, and PRISM-E evaluations all demonstrated positive impacts as a result of the intervention. The ACCESS evaluation found that "interventions designed to increase the level of integration...did not result in better client outcomes." These previous evaluations are reflective of the earlier knowledge development and application focus of the PRNS program. Since the focus of the program has changed to emphasize mental health transformation, evaluations that are currently in progress will reflect this focus. Of programs currently being evaluated, early results show a positive impact of the Suicide Hotline. The program is in the process of developing an overall evaluation strategy that is expected to result in higher quality, outcome-oriented evaluations.