Detailed Information on the
Community Mental Health Services Block Grant Assessment

Conducting an independent evaluation of the program

Making performance data disaggregated by state available to the public

Conduct pilot on the collection of client level data across all states for National Outcome Measures

Developing a web portal to assist state planning councils in their respective state transformation efforts.

States will report on all CMHS NOMs in FY 2008. An in-depth analysis of the CMHS' Uniform Reporting System (URS) and its application to the NOMS is in progress.

Developing an efficiency measure

Measure: Rate of readmission to State psychiatric hospitals (a) within 30 days (b) within 180 days--Adults

Explanation:Readmission is useful as an indicator of the desired outcome of developing a community-based system of care. Analysis of data from 03-04 suggests that individuals who were hospitalized were likely to be those with more severe illness; also in FY 04 many states experienced a reduction in community-based services necessary to reduce readmissions. Future targets have been adjusted and will be reviewed after 05 data is available.

Measure: Rate of consumers/family members reporting positively about outcomes for (a) adults and (b) children/adolescents.

Explanation:Reporting on outcomes captures whether the person is better able to deal effectively with daily problems, control their life, deal with crisis, get along with family, do better ins ocial situations, do betterin school and/or work, and is bothered less by symptoms.

Measure: Number of people served by state mental health systems.

Explanation:The number of persons served captures the reach of the program.

Measure: Number of a)evidence-based practices (EBP) implemented per State and b) percentage of service population coverage for each (reported as percentage of service population receiving any EBP, adults and children)

Explanation:Implementation of these practices results in better quality mental health care for persons served in state public mental health systems and will also make care more cost efficient over time. The program is also exploring other ways of measuring efficiency. a)National average of EBPs per state based on 35 reporting, b) adults*, c)children/adolescents* (*not including Medication management and Self-mgt which continue to undergo definitional clarification)

Measure: Rate of readmission to state psychiatric hospitals a) within 30 days, and b) within 180 days--Children/adolescents

Explanation:Readmission is useful as an indicator of hte desired outcome of developing a community-based system of care. Analysis of data from 03-04 suggests that individuals who were hospitalized were likely to be those with more severe illness; also in FY 04 many states experienced a reduction in community-based services necessary to reduce readmissions. Future targets have been adjusted and will be reviewed after 05 data is available.

Measure: Number of Persons Receiving Evidence Based Practices (EBPs) Per $10,000 of MH Block Grant Dollars Spent.

Explanation:The measure is calculated by dividing the number of adults with SMI and children/adolescents with SED who received EBPs during the FY by the MHBG allocation for the FY in question, multiplied by 10,000.

Is the program purpose clear?

Explanation: The purpose of the Community Mental Health Services Block Grant is to provide flexible funds to states and territories by formula to support community mental health services for adults with serious mental illness and children with serious emotional disturbance. Funds are provided to state mental health agencies, which have primary responsibility for operating the public mental health system. The block grant is designed to provide resources to states to help them implement state plans to improve community-based services and reduce reliance on hospitalizations for the treatment of mental illness. The target population are those with serious illness and not those with mild disorders or those at risk of developing future disorders. Five percent of the total is used by the agency for technical assistance, data collection and other activities. The block grant funds state infrastructure to support care and treatment in the community and not only direct services.

Evidence: The block grant is authorized in section 1911 to 1920 of the Public Health Service Act. The authorization specifies eligibility, criteria for allocating resources, the content of state plans for use of funds, maintenance of effort and the establishment and maintenance of the State Mental Health Planning Council. Community mental health centers provide the majority of services funded by the block grant. Agency and Congressional reports related to the program are consistent with the program purpose as outlined in the authorizing legislation. The program was established in 1981 as the Alcohol, Drug Abuse and Mental Health Services block grant. The program is run by the Substance Abuse and Mental Health Services Administration (SAMHSA).

Does the program address a specific and existing problem, interest, or need?

Explanation: The block grant addresses the problem of providing comprehensive, community-based systems of care for individuals with serious mental illness and serious emotional disturbance who rely primarily on public mental health systems for their care. Over time, states have shifted care of people with serious mental illness from institutions to the community. The block grant is focused on services for those reliant on public mental health systems and is designed to provide resources to enable individuals to function outside of inpatient or residential institutions to the maximum extent of their capabilities. To work most effectively, the mental health service system should coordinate with many sectors, including public and private care, specialty care, social welfare, housing, criminal justice, etc. (Mental Health: A Report of the Surgeon General, 1999). States must address coordination in their state plan. Through this process, the block grant is designed to address the state-wide system.

Evidence: Of the 10 million adults who meet the criteria for serious mental illness in any given year, between 50 and 60 percent receive treatment. An estimated 4.5 to 6.3 million children in the United States have a serious emotional disturbance. An estimated 75-80% of children with serious emotional disturbance are not receiving specialty mental health services. The 1999 Surgeon General report on mental health found children with serious emotional disturbance are best served with a systems approach (SGR, 1999). Most users receive some care in private facilities and a fifth receive care in public facilities. Of the minority using inpatient care, a third receive care in public facilities (SGR). State mental health agencies are responsible for service delivery for more than 2 million people suffering from serious mental illness each year; data from 33 states indicate state agency expenditures for psychiatric hospitals dropped from 52 percent to 35 percent of total expenditures between 1987 and 1997 (GAO-01-224).

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The mental health block grant is not overly redundant or duplicative of other efforts. Traditionally, the public mental health system has been operated and funded by state and local government. The federal government has increased its involvement in this area of effort over time through Medicare, Medicaid and targeted federal funding. However, the block grant is the only federal program that provides funds to every state to develop a comprehensive, community-based system to provide services to persons with severe mental illness who are uninsured or insured but have no mental health coverage. The block grant allotment makes up between less than one percent and as high as 33 percent of each state's mental health agency expenditures, including Medicaid. The block grant also requires states to develop plans to coordinate all sources of funding.

Evidence: Evidence includes GAO-01-224. In addition to the block grant, federal programs involved in supporting mental health services today include Medicaid, Medicare, SAMHSA competitive mental health grants, SAMHSA's PATH state formula grant for homeless individuals with serious mental illness and SAMHSA's Children's Mental Health Services program. Medicaid accounted for 20 percent of all mental health spending in 1997. Medicaid covers medically necessary services and some social support services for persons with mental illness. The block grant supports services for those ineligible for Medicaid and supportive services such as employment and housing that Medicaid does not reimburse. According to a NASMHPD survey of 37 states, people served with block grant funds represent 24 percent of all persons served in the public mental health system. Plans must address health and mental health, substance abuse and other supportive services such as employment and housing to be provided to individuals with mental illness through federal, state and local funds.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The block grant is free from major design flaws that prevent it from meeting its defined objective of supporting state efforts to move care for adults with serious mental illness and children with serious emotional disturbance from inpatient care to the community. The agency is reviewing approaches to shift the program emphasis from set-asides and other state funding requirements to reporting on the outcomes of grant expenditures. The agency seeks to retain the prevention set-aside and other requirements such as screening for tuberculosis. While there are possible flaws to the distribution of funds described below, there is no strong evidence that another approach or mechanism such as competitive grants would be more efficient or effective.

Evidence: Evidence includes the draft report to Congress on transforming block grants in performance partnerships (April 2003). As initially designed, the block grant was intended to simplify federal restrictions and oversight on funds, reduce administrative expenses, increase flexibility and state authority, strengthen state capacity, increase and maintain service system capacity, allocate funds equitably and target funding to priority issues. Statute and regulations require states to report how they spent their grant funds and do not require reporting on the impact the funds have on individuals or targeted populations. By design, an emphasis on reporting on the outcomes of federal expenditures was not included.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: A strong correlation between funding distribution and prevalence is an important aspect of program targeting and improves the chances that individuals will have the same probability of getting care regardless of where they live. While the formula does not use prevalence of serious mental illness and serious emotional disturbance, agency data indicate little variation in serious mental illness by state and region, making the lack of prevalence data in the formula less meaningful. Prevalence does vary by age, gender, educational status, and urban and rural residence. In the case of serious emotional disturbance, prevalence correlates with poverty rates, which are not incorporated into the formula, but are indirectly captured by wage data. Wage data are an indirect measure and often out of date and poverty data may be more useful. State surveys confirm the block grant serves low-income individuals with serious mental illness and the maintenance of effort requirement guards against supplantation.

Evidence: The estimated 12 month prevalence of serious mental illness is between five and six percent nationally and rates do not differ among states at a 95 percent confidence interval (Federal Register 6/24/99). SAMHSA published additional definitions and data methods for serious mental illness and serious emotional disturbance (FR 5/20/93, 7/13/98). A 1995 RAND evaluation highlighted some equity shortcomings. A more narrow focus, such as the poor and uninsured, rather than age, may better serve equity goals and program purpose (RAND, MR-533-HHS/DPRC, 1995). The HHS Office of the Inspector General notes block grants often include targeting requirements for vulnerable populations, but effectiveness is unproven (OIG, OEI-01-94-00160). Prior to the most recent reauthorization, states called for an external review of the block grant formula by the National Academy of Sciences or another independent body. The 2000 reauthorization established a minimum allotment. The formula uses taxable resources, population size and age, cost of services and wage data. HHS adjusts the formula every three years.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The agency adopted new long-term outcomes measures to advance strategic planning and the conversion of the block grant to a performance partnership grant. Measures include: Rate of readmission to State psychiatric hospitals (a) within 30 days (b) within 180 days; and, Rate of consumers/family members reporting positively about outcomes.

Evidence: This first measure captures efforts to move people from state hospitals to community care; develop transition/discharge-planning systems; and establish comprehensive community-based care systems. Readmission is useful as an indicator of the desired outcome of developing a community-based system of care. Reporting on outcomes captures whether the person is better able to deal effectively with daily problems, control their life, deal with crisis, get along with family, do better in social situations, do better in school and/or work, and is bothered less by symptoms. All sixteen states do not report on each measure, and there are further variations for those that are reporting. Under the performance partnership grants, states will report on performance against agreed upon outcome goals. A notice in the December 24, 2002 Federal Register describes central elements of the proposed transition to performance partnership grants.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has baselines and targets for the long-term measures.

Evidence: The program has baseline data from 2000 for the first measure with a target year of 2008. The program has baseline data from 2002 for the second measure with a target year of 2008.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The agency has a limited number of annual measures that can demonstrate progress toward achieving desired long-term outcomes. Annual measures include: the number of people served by state mental health systems, the number of SAMHSA-identified, evidence-based practices adopted in each State and the percentage of (service) population covered, and annual increments of the two long-term outcome measures on readmission and consumer reported outcomes.

Evidence: The number of persons served captures the reach of the program. The evidence-based practices measure captures the agency's efforts to improve the efficiency and effectiveness of state-supported mental health services. The annual measures for readmission and outcomes will provide the program regular updates on progress toward meeting the long-term measures.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: The agency has baseline and targets for all but one of the annual measures.

Evidence: Initial baseline data for the evidence-based practices measure will be obtained in December 2003 through the program's URS and the remaining areas will be reported on in 2004. A pilot study will be conducted in FY 2005 on the relationship between evidence based practices and cost for baseline data.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program managers work to ensure states support the overall goals of the block grant and measure and report on performance as it relates to accomplishing goals. Beginning this year, 50 states are reporting on performance information through basic and developmental tables of the uniform reporting system. States also commit to the overall objectives of the block grant to provide community-based services when possible to adults with serious mental illness and children with serious emotional disturbance. States include descriptions of how they will meet overarching goals of the program in state plans and reports. The block grant has gone through an important transition over time from a formal application review process to more of a partnership. States are involved in the setting of goals through planning for the transition to performance partnership grants. Commitment toward the goals of the program should increase further through this transition in coming years.

Evidence: States and territories include needs assessment data in their applications and are now reporting on performance information. According to SAMHSA, the program has worked with states since its inception to improve data collection and reporting. An example of these efforts is the 16-State Project to develop uniform data and unduplicated counts of persons served. Forty-seven States have also received grants to improve data collection. A notice in the December 24, 2002 Federal Register describes central elements of the proposed transition to performance partnership grants. The state implementation reports and block grant plans already provide considerable information and commitments. The agency has also laid the groundwork for implementing new outcome measures that will enable partners to commit to and work toward the annual and long-term goals of the program.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: A Yes requires regularly scheduled objective, independent evaluations that examine how well the program is accomplishing its mission and meeting its long-term goals. The program is initiating the first of three consecutive independent evaluation studies in FY 2003. The first study will assess whether the program is working in a logical way, examine how to collect data on effectiveness, and make recommendations for program improvements. A second study in FY 2004 will be more comprehensive and will test performance indicators and examine specific program deficiencies. A final summative evaluation in FY 2005 will assess the impact of program changes made following recommendations from the first assessment. As noted in Section IV, no comprehensive and external evaluations have been completed to date on this program. By design, accountability and evaluations have been focused on compliance with statute, including set-aside requirements.

Evidence: The three studies will range from $100,000 to $1 million in cost and will be conducted by external groups through contracts. SAMHSA reports grantee efforts for evaluation, but no independent, comprehensive evaluations of the program are available. Many states also conduct evaluations, but they are not currently aggregated or reported on at the national level. RAND conducted an evaluation of the funding formula in 1995 (RAND, MR-533-HHS/DPRC, 1995). NASMHPD published a review of state spending in March 2003, including per capita spending and expenditures by group. The organization has also published reports on psychiatric hospital discharge rates and institution closings, implementation of evidence based practices and a survey of 37 states on the profile of those being served and the type of services delivered.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program does not provide a budget presentation that clearly ties the impact of funding decisions on expected performance or explains why the requested performance and resource mix is appropriate. Annual budget requests are not clearly derived by estimating what is needed to accomplish long-term outcomes. The program has different output and outcome goals and has not identified how much cost is attributed to each goal. The program is currently able to estimate outputs (number of persons served) per increased increment of dollars by dividing block grant funding by average Medicaid client cost for outpatient care. The block grant supports 17 full time equivalent staff. Other agency program management funds are budgeted separately.

Evidence: This assessment is based on the annual budget submission to OMB and the Congress.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: SAMHSA is currently undertaking a comprehensive strategic planning effort to address accountability, capacity, and effectiveness. The agency has formed a planning matrix of priorities and crosscutting principles to coordinate resource allocation across the agency and produced a draft strategic plan. The program plans to begin developing budget requests based upon average cost to serve a client in a community program. Having new measures in place will further enable the program to integrate budget planning and strategic planning and determine the level of financial resources needed to obtain long-term outcomes. The agency's efforts to develop a performance partnership grant will also facilitate commitment to and reporting on performance measures. The agency contracted with NASMHPD in 2002 to examine the ability to define and implement performance measures for the block grant. The report found promise but noted substantial work remains to make the measures comparable across states.

Evidence: The agency's restructuring plan consolidated budget formulation, planning and Government Performance and Results Act activities within one unit. As described in a December 24, 2002 Federal Register notice, the performance partnership grant is based on a shift toward greater accountability in exchange for state flexibility to design, implement, and evaluate mental health services. SAMHSA is currently working with the states to identify core measures for mental health services. With set-aside funding, the agency is also supporting a technical assistance center for evaluation of programs and systems to improve adult services under the block grant. State data infrastructure grants are being used to improve state data collection. SAMHSA indicates that it will pilot test an independent evaluation of several performance measures that will focus on multiple factors, including federal programs and funding streams and state and local resources. SAMHSA has developed an evaluation contract directed toward improving program evaluation in the block grant and other SAMHSA programs.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects performance information on an annual basis and uses the information to manage the program and improve performance. The states submit annual uniform applications that describe past, current, and intended use of program funds. The program collects annual information on state satisfaction with agency technical assistance and the grant review process. Program performance data are also collected during onsite technical reviews. SAMHSA also uses data from national surveys and contracts funded by the set-aside to guide technical assistance efforts.

Evidence: The assessment is based on agency descriptions of actions taken based on performance information, state annual reporting forms and plans, and annual budget documents submitted to OMB and the Congress. The program's Uniform Reporting System can help facilitate the transformation to a performance partnership grant to improve outcomes and focus on more effective services. The program updated the cost of services component based in part on findings from the 1995 RAND review of the formula.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: Performance plans for managers at the Division Director level and above track to management/program objectives. The program director is an SES level and has a performance contract. Managers review state compliance with the legislative requirements and monitor expenditures through compliance reviews and single audit reports, ensure that applicable financial status reports are completed, and reconcile financial status reports to the Payment Management System. Performance Based Contracting has been initiated for all new SAMHSA contractors' who hold services contracts. The transition to performance partnership grants will increase the accountability of program partners for performance results.

Evidence: The assessment is based on discussions with the agency and manager performance contracts. Employee evaluations at the agency are handled by each of the agency's three centers. One planned element of the performance partnership grants is to use corrective action plans as a means of increasing accountability for performance results and making program improvements. The monitoring visits are one week on site reviews conducted by three consultants with fiscal, management and/or clinical expertise and a federal project officer. The review covers the state agency and two or more urban and rural programs serving adults and children. The program reserves the right by statute to withhold funds for failing to fully implement the state plan.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The agency reports funds are obligated by the government on a quarterly basis, usually within two-three days after an application has been determined compliant with relevant requirements of the Public Health Service Act. States have two years to obligate and expend funds to sub-recipients.

Evidence: Evidence includes application forms and agency documents. Agency managers review annual grantee applications to determine funds are used for the intended purpose. Agency staff also examine the states' obligations and expenditures of grant funds during state technical reviews.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has some procedures in place to improve efficiencies in execution. SAMHSA has established a block grant re-engineering team to improve the efficiency of staff operations in managing the program at the federal level. The agency does rely on an HHS service clearinghouse known as the Program Support Center for many internal services. The agency is providing FAIR Act targets and appears to be making progress toward outsourcing additional services. There are also elements in the block grant that seek to limit administrative costs. For example, there is a five percent limitation on administrative costs at both the federal and grantee levels. Each state and territory uses the fiscal policies that apply to its own funds for administering the block grant. Additional steps, including adoption of efficiency measures, are needed to maintain progress in this area.

Evidence: Evidence includes the FAIR Act report, services directed to HHS' consolidated Program Support Center, and Restriction of Expenditure of Grant. In the area of technical assistance, the program provides assistance on the planning council requirements, children and families, criminal justice area, housing, and other topics primarily through contractors. In 2002-2003, 12 states received no assistance, 28 received one to two, 15 received three to five. Contractors include Bazelon, the National Alliance for the Mentally Ill, the National Association of State Mental Health Program Directors and others. The program also uses contracts for peer reviews and monitoring in the field.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The agency has transformed the relationship with states over time to a more collaborative exchange with respect to both the applications process and annual operations. Federal managers collaborate internally in SAMHSA, with other federal agencies, with national organizations and the states. At the state level, each grantee is required to have a mental health planning council to review the state mental health plan. The council must include consumers, family members, service providers and state officials. The state must also seek comments from the public on its plan.

Evidence: Evidence for this question is included in the Government Performance and Results Act report, meetings, conferences, and other documentation. Examples of specific activities include with CMS on Medicaid issues, with other agencies on the response to the Olmstead decision, with NASMHPD on the performance partnership grant planning, with states on the data infrastructure grant, with FEMA for crisis counseling and with the National Institute on Disabilities Rehabilitation and Research and DOE for research and training on children's issues.

Does the program use strong financial management practices?

Explanation: The program receives clean opinions on its audits and is free of material internal control weaknesses. SAMHSA is participating in a department-wide initiative to implement a new Unified Financial Management System. SAMHSA will in the meantime replace the current DOS-based Integrated Financial Management System with a customized government-off-the-shelf system for tracking commitment and obligation data. The Integrated Resource Management System provides for tracking of commitments and obligations and for numerous management reports.

Evidence: Discussions and documents from agency managers, audited statements from the Program Support Center; Office of the Inspector General reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program is taking meaningful steps to address management deficiencies in key areas. With respect to deficiencies highlighted in this section, the program has made performance information available from the sixteen state project on the Internet and will be able to make additional outcome data available to the public through the performance partnership grants. The program has also proposed a pilot study to test the cost efficiency of utilizing mental health interventions that have proven to be effective and the initial impact on expenditures. The program is addressing accountability for results at both the federal and grantee level. The agency has begun using performance contracts that will set specific, quantitative targets.

Evidence: The agency plans to implement performance plans for managers at the Division Director level and above that are tied to department-wide management objectives and agency program objectives in June. The agency plans to implement performance plans for all staff, which must include at least one element that tracks back to these objectives by September 30. The agency also plans to ensure program and management objectives in the SAMHSA Administrator's performance contract are incorporated into the performance plans of senior management and staffs. The Administrators performance contract is based on ten program priority areas that will eventually be incorporated into SES level, division level and branch chiefs. The use of performance measures in employee evaluations is under examination.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program does have sufficient oversight capacity. This capacity will improve with respect to outcomes of the block grant with the transition to performance partnerships. However, the program is able to document grantees' use of funds in compliance with legislatively designated categories, conducts site visits to a substantial number of grantees on a regular basis and confirms expenditures in annual reports. Through national level relationships and the work of the project officers, the program has a fairly high level of understanding of what grantees do with the resources allocated to them.

Evidence: Evidence includes agency documentation, applications and the performance plans and reports. After reviewing the state plan implementation report for the previous fiscal year, the agency also reviews whether the state completely implemented the plan approved for the previous year.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Grantee performance data are currently only available to the public at the national level and not disaggregated by state. The agency plans to make additional state information available in the near future from the Uniform Reporting System. Annual performance data are aggregated in the performance report and are available to the public through the SAMHSA web site. A conversion to a performance partnership grant will also increase the amount of information gathered on grantee performance on select outcome measures. Data from the 16-State Project are available to the public. Data are available by state and covering a number of areas, including readmission to psychiatric facilities, penetration of services and consumer reporting on access, appropriateness and positive changes resulting from services. Additional state information is available from the national association, but not through the agency.

Evidence: Assessment based on agency web site (www.samhsa.gov/funding/funding.html). Additional information is available through the National Association of State Mental Health Program Directors associated NASMHPD Research Institute (nri.rdmc.org/profiles.cfm) and from the sixteen state project at the Mental Health Statistics Improvement Program (www.mhsip.org/sixteenstate/index.htm).

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: As noted in Question 2 of the Strategic Planning section, the agency developed new long-term measures and adopted specific targets. The program has demonstrated progress in achieving outcomes related to these new measures in the annual performance plan. The related areas from existing measures that are to be dropped from the performance plan include improvements in employment, school attendance, stability of living arrangements, independent living and contact with the juvenile justice system. A small extent is given because the program does not yet have subsequent years of data to measure progress specifically on the long-term performance goals. The program will be able to measure progress in future years.

Evidence: Progress from existing measures include adult employment and contact with the criminal justice system from 1999 to 2000, improvements from 1999 through 2001 in independent living, improvements in school attendance from 2000 to 2001, improvements in stability of living arrangements from 1999 to 2001 and improvements in children's involvement with juvenile justice system in 2000 but not 2001. The program will collect additional data to show progress on the new long-term measures in the next year. Assessment based on agency planning documents, GPRA reports, SAMHSA-wide performance measures document and draft measures for the performance partnership grant. Twelve states are reporting on the percent of consumers reporting improved outcomes from services and 16 states are reporting on the percent readmitted within 180 days to any state psychiatric hospital.

Does the program (including program partners) achieve its annual performance goals?

Explanation: A Small Extent is given because the program does not have multiple years of data to show progress in achieving each of the newly adopted annual goals. The program will have additional data to measure achievement in future years. As noted in Question 4 of the Strategic Planning section, the agency has developed a baseline and adopted targets for all but one of the annual goals that support the desired long-term outcomes of the program.

Evidence: The number of persons served has increased when compared to 1992 and 1998 data from the Survey of Mental Health Organizations and General Hospital Mental Health Services. Data prior to 2000 on 30 and 180 readmissions are unavailable. However, the rate of any readmission has declined from 80 percent in 1980 to 75 percent in 1986 and 68 percent in 1997 according to data from SAMHSA and the National Institute of Mental Health at HHS. The number of resident patients has also declined. Assessment based on agency planning documents, GPRA reports, SAMHSA-wide performance measures document and draft measures for the performance partnership grant.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The agency is meeting the standards of a Yes for having incentives and procedures to measure and achieve efficiencies. A Small Extent is given because the program has not demonstrated large gains over the prior year. The program cites an increase in state expenditures per block grant dollar of $8.35 in 1983 to $38.59 in 2001 as evidence of improved efficiency from the federal perspective. While significant, increased investments at the state level do not necessarily relate to the efficiency of federal operations. Measures of reduce psychiatric hospital readmissions will provide additional data on program level efficiency improvements in the future.

Evidence: The agency's efforts to transition to a performance partnership grant are intended to reduce requirements in the block grant through an increase reliance on reporting on outcomes. The new structure should enable the program to more efficiently achieve outcome goals in mental health treatment.

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: Numerous Federal funding sources are available to support mental health treatment for adults with serious mental illness and children with serious emotional disturbance. State and local entities also invest resources in this area. However, the block grant is the only federal activity designed specifically to support state-wide services to all states in this area. No comparisons of the effectiveness of treatment services through Medicaid and treatment services supported by the block grant have been conducted.

Evidence: Evidence includes agency budget reports, GAO/GGD-98-137, SGR 1999, and agency documents.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The program has not yet had evaluations meeting the standard for this question that are at the national program level, rather than one or more partners, and focused on the program's impact, effectiveness or other measurement of performance. The program and the partners receive valuable information from state planning council reviews, but the reviews are not comprehensive evaluations with respect to this question. Similarly, state profiles provide valuable information on financing, staffing, service, information technology and other areas for managing the program, but are not independent evaluations. Research confirms the efficacy of mental health treatment more broadly. As noted in Section II, additional steps are also being taken to support evaluations in the future.

Evidence: The agency conducts reviews of state activities through on-site reviews, reviews of applications, and reviews of financial audit reports. Annual program reviews are also conducted by State Mental Health Planning Councils. However, GAO notes that the councils generally lack expertise in evaluation and reviews are not consistently accompanied by back-up information (GAO/GGD-98-137). The agency reports that since the GAO report these reviews have become more sophisticated. RAND has examined the formula and GAO has examined the federal involvement in this area overall, but neither have performed comprehensive evaluations of the program. The state technical reviews provide information on the states' obligations and expenditures in accordance with the statute, service delivery by modality, quality improvement for clinical services and management, and opportunities for improvement and targeted technical assistance.