Detailed Information on the
Childrens Mental Health Services Assessment

Tracking cost efficiencies.

Completing an evaluation of the performance of the national evaluation team that is responsible for the Children's Program program evalulation activities.

Launching a continuous quality improvement initiative to assess the effectiveness and appropriateness of the delivery of technical assistance (TA) to funded grantee sites. Results of initiative will be reviewed on an annual basis.

Conducting an internal study to evaluate the effectiveness and appropriateness of the efficiency measure adopted by the program in 2002. The current measure "Decrease in inpatient care costs per 1,000 children served" was approved by OMB on 5/10/06.

Determining if the program is making lasting improvements in the care of children with serious emotional disturbance. The program will track how well children's behavioral and emotional symptoms improve and how well funded communities sustain their systems of care beyond the period of federal funding.

Developing Efficiency Measure

Measure: Percent of funded sites that will exceed a 30 percent improvement in behavioral and emotional symptoms among children receiving services for six months

Explanation:This measure tracks the clinical impact of funded sites on children receiving services as measured by scores on a standardized child behavior checklist (Child Behavior Checklist (CBCL, Achenbach, 1991). Scores are obtained at entry into services and after six months of services. The measure is the percentage of funded sites where children average more than a 30% improvement in score after receiving services for six months.

Measure: Percent of systems of care that are sustained five years after Federal program funding has ended

Explanation:Although the 2004 baseline was 100%, the data were based on only four grants initially funded in 1993, and thus the long-term target has not been raised. No additional data will become available until December 2009. A five-year follow-up is not planned for grantees funded in 1994, and no grants were awarded in FY 1995 and 1996. The next cohort, funded in FY 1997, was funded for six years; thus assessment at five years post-funding for thise grantees will occur in 2008 and will be reported in FY 2009.

Measure: Average reduction in the number of days per client spent in inpatient/residential treatment

Explanation:The performance target for reduction in days of inpatient care (measure 3.2.14) was set at an approximate target level, and the deviation from that level is slight. The FY 2007 target was nearly achieved. However, there was almost 80 percent improvement, which is equal to a reduction of .78 days as compared to the result obtained in FY 2006. Grantees funded in FY 2005 serve proportionately larger numbers of very young children who generally have shorter and less frequent hospitalizations. Given this change in populations served, and the sensitivity of the measure to the length of hospitalization prior to service intake, the targets for this measure remain stable through 2009.

Measure: Decrease in inpatient care costs per 1,000 children served. (New measure, added February 2008)

Explanation:One of the main goals of the program is to provide least restrictive services to children and youth served by the grantees. More restrictive services, like inpatient hospitalization, are also among the most expensive to provide. The proposed indicator meets the OMB definition of efficiency as a program's ability to implement its activities and achieve results relative to resources since it reflects per unit changes in costs. The efficiency measure reflects per-unit changes in costs. The performance target for measure 3.2.17 was set at an approximate target level, and the deviation from that level is slight. The FY 2007 target for reduction in costs of inpatient care was nearly achieved. However, there was almost 73 percent improvement as compared to the result obtained in FY 2006. The 2007 result may be due to the reduction in in-hospital days as reported in measure3.2.14. Since that indicator may vary, as discussed above, targets have been kept level.

Is the program purpose clear?

Explanation: The program purpose is to make grants to public entities to support comprehensive community mental health services to children with a serious emotional disturbance. The legislation specifies competitive grants will be used to establish systems of care for children with a serious emotional disturbance that provide specific minimum mental health services. The legislation also clearly outlines the term and matching requirements of the grants. The purpose is commonly shared by interested parties.

Evidence: Comprehensive Community Mental Health Services for Children and Their Families was authorized in 1992 (section 561 to 565 of the Public Health Service Act). Agency and Congressional reports related to the program are consistent with the program purpose as outlined in the authorizing legislation. The program is run by the Substance Abuse and Mental Health Services Administration (SAMHSA).

Does the program address a specific interest, problem or need?

Explanation: The program is designed to support and improve mental health services in the community for children with serious emotional disturbance. The agency defines the target population as "children and youth with a serious emotional disturbance from birth to age 21 who currently have, or at any time during the past year had, a mental, behavioral, or emotional disorder of sufficient duration to meet diagnostic criteria specified in the Diagnostic and Statistical Manual for Mental Disorders, Fourth Edition (DSM-IV), that resulted in functional impairment that substantially interferes with or limits one or more major life activities."

Evidence: An estimated 4.5 to 6.3 million children in the United States have a serious emotional disturbance. The 1999 Report of the Surgeon General on mental health found children with serious emotional disturbance are best served with a systems approach; and 75-80% of children with serious emotional disturbance are not receiving specialty mental health services. Prior to managed care, some state community mental health centers offered no children's mental health services. There are no data on the number of communities that have implemented a system of care approach.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The program is reaching a relatively limited number of individual communities and the national impact in the context of all other factors is not fully known. With an emphasis on changing the mental health system and a required graduated match from grantees, the program is designed to have a significant and lasting impact in individually funded communities. The program provides incentives for systems reform and provides seed money for developing new community-based mental health services and enhancing existing services. The program also includes a national public information and education campaign to increase public awareness that began in 1994, though the impact of this campaign is unknown.

Evidence: The program provides grants to local entities and from its inception has reached 8% of the nation's counties. The program has funded individual grantees in 43 states. Some state governments have adapted the program's approach to additional communities within the state, but in general the impact of the Federal investment is confined to those communities receiving funds. The program has leveraged an estimated $200 million from state, local and private sources, nearly one third of the Federal contribution. The program estimates at current levels it would take 16 years to reach one quarter of the nation's communities.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: Children's Mental Health is the only Federal funding source targeted to support comprehensive, community-based mental health services for children with serious emotional disturbance. There is little evidence of widespread state or local investment in establishing systems of care.

Evidence: The Robert Wood Johnson Foundation supported a program with similar goals in the 1980s that served as a foundation for Children's Mental Health. The Foundation also supported a replication program in 1993.

Is the program optimally designed to address the interest, problem or need?

Explanation: The program is administered through cooperative agreements with communities and provides direct contact to influence system changes at the community level.

Evidence: There is no evidence that providing support through a block grant or other mechanism would be more effective or efficient than competitive awards direct to communities.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has adopted long-term outcome goals focused on measuring performance and sustainability of funded communities. Program grants are designed to enable a community to establish a systems of care approach to children with serious emotional disturbance and support mental health services. Clinical improvement in child behavior after treatment is a key measure of program impact. Sustainability of systems of care after the end of the grant cycle provides information on the effectiveness of the community by community approach. An additional goal on program cost is under review to provide evidence of program efficiency beyond the sustainability of new systems of care.

Evidence: The long-term outcome measures will track the clinical impact of funded sites on children receiving services as measured by scores on a standardized child behavior checklist. The program provides support to transform a mental health system, which relies on the participation of juvenile justice, education and other service sectors. The legislation requires matching funds in order to broaden the reach of the program and increase the likelihood that the new system will be maintained after the conclusion of the six year grant cycle. A second measure adopted by the program will track the percent of systems of care that are sustained five years after program funding has ended.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: The program has a limited number of annual performance goals that are quantifiable and relevant to the mission. The annual goals relate directly to the long-term outcomes and purposes of the program. The goals address both individual outcomes for children receiving services and the performance of systems of care within funded communities.

Evidence: Children's Mental Health annual goals include: 1. Decrease average days in inpatient or residential facilities; 2. Increase percentage of referrals from juvenile justice system to system of care; 3. Sustain at least 80% of systems of care five years after they have stopped receiving Federal funds through the program.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: The program's direct grantees provide performance data on the program's annual goals to the agency. Each award recipient is required to report performance on a quarterly basis to an evaluation contractor. The evaluation contractor conducts a cross-site national evaluation. The agency also works with award recipients to use performance data for their own strategic planning.

Evidence: Award recipients dedicate two FTE for the evaluation system. Performance data are entered directly into a computer and are reported to the national evaluation contractor quarterly through a web-based system. These data are compiled and reported in the program's annual report.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: The program collaborates and coordinates at both the grantee level and the Federal level. At the local level, collaboration between education, juvenile justice, and the mental health system is central to the program goal to integrate services at the local level. Federal level collaboration takes the form of meetings, funding for technical assistance, and reimbursable agreements.

Evidence: At the grantee level, projects are required to develop collaborative relationships across child-serving sectors in the community including education, child welfare, juvenile justice, and mental health. At the Federal level, the program collaborates with the National Institute of Mental Health, the Health Resources and Services Administration, the Administration on Children and Families and the Department of Education.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: As required by the authorizing legislation, the program supports an annual evaluation to demonstrate the effectiveness of the systems of care approach supported by the program. The evaluation is focused on program goals and is conducted through a private contractor external to the program and funded sites. Outcome data are collected from each funded site beginning in the third year of the six year grant period. The evaluation measures the effectiveness of the program and presents recommendations for program improvements. The program produces an annual report to Congress on evaluation results. The latest report focuses on 31 grant communities that established systems of care for approximately 40,029 children and their families.

Evidence: Each site is visited three times during each six year award cycle. Evaluated elements include the extent to which systems of care develop and improve over time, type and amount of services children receive, cost of services, improvements in clinical and functional outcomes and family life, duration of improvements, attribution to systems of care approach, and relative effectiveness of the intervention. The evaluation consists of a study of the demographic and functional characteristics of children and families at intake, child and family outcome study, a measures of the incorporation of the systems of care approach into service at the clinical and systems levels, and a study of the cost-effectiveness of the program.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: Annual budget requests are not clearly derived by estimating what is needed to accomplish long-term outcomes. The program has different output goals and has not identified how much cost is attributed to each goal. The program is able to estimate outputs (number of communities funded and children served) per increased increment of dollars. Program management funds are budgeted separately.

Evidence: This assessment is based on the annual budget submission to OMB and the Congress.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The deficiency highlighted in this section relates to program budget alignment with program goals. Through this process, the program has adopted new long-term goals that capture intended outcomes of the program. The program is estimating the likely outcomes of the program based on past performance. Having these measures in place will further enable the program to integrate budget planning and strategic planning and determine the level of financial resources needed to obtain long-term outcomes.

Evidence: The program has adopted new long-term goals. The agency also reports developing performance based budgeting to strengthen the links between performance and budget. The agency's restructuring plan consolidated budget formulation, planning and Government Performance and Results Act activities within one unit.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects performance information on an annual basis and uses the information to manage the program and improve performance. Cross-site data have been collected since 1995 when the program's national evaluation was first implemented.

Evidence: For example, when data showed a decrease in referrals from child welfare and education systems in FY 2001, the program increased technical assistance to grantees to emphasize interagency collaboration at the local level through expertise in child welfare, education, juvenile justice and primary care.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: Federal managers are not held accountable for results through employee evaluations or other mechanisms. The program manager is responsible for ensuring that Project Officers exercise adequate surveillance and quality control over the activities of grantees and contractors. The agency does use annual performance data to hold funded communities accountable for their results. The program also uses performance contracts to monitor the performance of its evaluation and technical assistance contractors.

Evidence: The assessment is based on discussions with the agency and program manager vacancy announcements. Employee evaluations at the agency are handled by each of the agency's three centers.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program obligates funds on schedule and monitors use for the intended purpose. Award recipients typically spend awards during the single fiscal year. Federal managers review expenditures for contracts on a monthly basis and approve or disapprove reimbursement items.

Evidence: The assessment is based on apportionments, program evaluation forms and financial status reports. The agency is also working on establishing waves of grant announcements to improve the distribution of obligations through the fiscal year.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program can take additional steps to improve administrative efficiency, but does have some incentives and procedures in place. The program operates with a relatively limited number of Federal staff. The agency relies on an HHS service clearinghouse known as the Program Support Center for many internal services. The agency is providing FAIR Act targets and appears to be making progress toward outsourcing additional services. Outsourced activities include accounting, graphics, human resources, and property management. The program contracts out evaluation, technical assistance, public education, and logistics. Performance data are collected electronically and reported through a web-based system known as the Interactive Collaborative Network. Federal staff also review proposed budgets to identify excessive or inappropriate costs.

Evidence: The assessment is based on discussions with the agency, FAIR Act reports, and the description of services directed to HHS' consolidated Program Support Center.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: The program is unable to cost out resources needed to achieve targets and results. The program does not capture all direct and indirect costs borne by the program agency, including applicable agency overhead, retirement, and other costs budgeted elsewhere, or include informational displays in the budget that present the full cost of outputs. FTE and administrative expenses are not tied to annual program budgets. The program does not have a financial management system that fully allocates program costs and associates those costs with specific performance measures. The program does develop annual budget proposals that include associated FTE costs.

Evidence: The assessment is based on annual program management budget requests to OMB and Congress.

Does the program use strong financial management practices?

Explanation: IG audits of the agency's financial management have identified no material internal control weaknesses. The agency's fiscal monitoring of grant awards is conducted through the SAMHSA Grants Information Management System (SGIMS), which tracks awards and obligations, carry over and submission of quarterly reports, application renewals and final reports.

Evidence: The assessment is based on conversations with the agency, audited statements and Office of the Inspector General reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The main deficiencies include use of performance data to enhance accountability and the ability to identify changes in performance with changes in funding levels. Most significantly, the agency reports taking additional steps to hold staff accountable for program performance.

Evidence: The agency has begun rolling out performance contracts as part of an overall management reform plan that will set specific, quantitative targets. These contracts are to include outcome elements focused on program goals. The agency's restructuring plan consolidated budget formulation, planning and Government Performance and Results Act activities within one unit.

Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process?

Explanation: A central office within the agency organizes and conducts independent review of grant applications for agency programs. Applications for this program are peer reviewed based on clear criteria and awards are made based on merit as judged through the peer review process.

Evidence: Assessment based on grant review procedures, Federal Register Notices. Congress does not include earmarks for this program.

Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process?

Explanation: The program encourages participation of public entities that have never been funded before. The program is designed to establish sustainable changes in funded communities that will not require Federal funding once the six year grant period has ended. The program also funds grantees in new geographic regions of the country. The program also provides technical assistance to prospective applicants and those that have applied but not received an award.

Evidence: Since its inception, the program has funded 67 grants in 43 states and eight Native American Tribes. The FY 2002 grant announcement introduced set-asides for territories and cities of 500,000 or more to encourage grant applications from areas which have not received funding.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Federal staff serving as project officers receive data on grantee activity quarterly through the agency's SGIMS system. Project officers visit each funded site accompanied by agency consultants in years two and four of the grant cycle and as needed. The national evaluation contractor also conducts site visits three times during the grant period. Project officers review and approve annual budgets and monitor non-federal match funding. Grantees report annually on performance.

Evidence: The assessment is based on copies of grantee reports, and site visit protocol documents.

Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Data are collected and compiled through the national evaluation of the program conducted since 1995. Annual performance data are summarized in the performance report and made available on the agency web site. Additional steps could be taken to make performance data by state or community available to the public.

Evidence: Assessment based on agency GPRA reports and web site (www.samhsa.gov). Additional data outside of GPRA are reported through the agency's mental health web site (www.mentalhealth.org) and through annual reports to Congress on the program, which are also available on the agency web site. On a more ad hoc basis, performance data are conveyed through journal articles and at professional and grantee conferences and meetings.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: The program has adopted new long-term outcome goals that are ambitious and relate to the mission of the program. The measure of clinical effectiveness is based on the number of communities that exceed a 30 percent improvement in behavioral and emotional symptoms among children receiving services for six months. Program impact is also measured by the percentage of funded communities maintaining systems of care five years after no longer receiving Federal support. Currently, the oldest cohort of grantees is only three years out from receiving Federal support and 80% of these communities have maintained a system of care approach to children's mental health. An additional goal is under consideration to measure program efficiency, such as a measure of average cost of treatment before and after implementing a system of care approach. A possible third measure is under review as a means of capturing the reduction of more costly treatment modalities realized from a system of care approach. These data are already tracked for the annual measure.

Evidence: The improvement in behavioral and emotional symptoms is derived from a calculation of the Reliable Change Index (RCI, Jacobson & Truax, 1991) for the intake and six month scores of the Child Behavior Checklist (CBCL), a standardized measure of behavioral and emotional symptoms (Achenbach, 1991).

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program sets annual targets and is meeting those targets. The annual goals provide information on program progress toward meeting its long-term outcomes. One measure related to system efficiency is the average number of inpatient or residential days. This measure captures both improvements in system approaches and also provides a rough indication of potential reductions in overall costs to the system associated with more expensive mental health care services. This measure was not adopted as a long-term outcome because only 5% of children served by the program enter the system from a residential care treatment facility, and the measure is insufficiently representative of the program's total long-term outcomes. The annual measure will also track system sustainability after the conclusion of Federal funding.

Evidence: Data on program outcomes are collected from a multi-site outcome study that uses self-reported delinquency surveys. Reductions in inpatient treatment are tracked by comparing data from grantees with a restrictiveness of living environments scale. Sustainability data have been collected by contract using a checklist of key system components.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The agency is meeting the standards of a Yes for having incentives and procedures to measure and achieve efficiencies and has realized some improved efficiencies at the Federal program level. The agency is taking further steps to improve efficiency through reductions in deputy manager positions and consolidation of smaller offices. The average number of children served in the second year of the grant shows some upward movement from the 1997 to 1999 grantee cohorts. However, the average number of days in residential treatment has crept upward from 1998 to 2001. A Large Extent or Yes would require additional data on improvements in efficiencies and cost effectiveness in achieving program goals in the last year.

Evidence: Assessment is based on annual performance reports, agency restructuring plans, and discussions with agency managers. The average number of children receiving services in the first operational year increased from 23 to 36 between 1998 and 1999 and in the second operational year from 105 to 179. The average number of days in residential treatment is below the 1997 baseline, but increased from 143 in FY 1998 to 152 in FY 2001. Improved efficiency data are needed.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: As noted in Section I, Children's Mental Health is the only Federal funding source targeted to support comprehensive, community-based mental health services for children with serious emotional disturbance.

Evidence: The performance of this program is similar to a Robert Wood Johnson Foundation Demonstration program and a predecessor program at the National Institute of Mental Health, but not to any existing Federal programs.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The results of the program's annual evaluation indicate the program is effective and achieving results. Data are reported in GPRA, but the most comprehensive reporting of program performance is found in annual reports to Congress. The 1999 report presents data accumulated through August 1999 from 22 grant communities initially funded in either FY 1993 or FY 1994 and 9 grant communities first funded in FY 1997. The evaluations have found that children are able to function better in school, at home and in society than when they first started in the program. After two years of services, 42 percent of the children showed a significant reduction in severe behavioral and emotional problem symptoms and an additional 48 percent of the children were stabilized. The children have fewer behavioral and emotional problems, their behavioral and emotional strengths improve, and their level of impairment decreases. Effected families as a whole are functioning better than when they first started to participate in systems of care programs.

Evidence: Selected findings in the most recent report include: regular school attendance increased from 85.9 percent at entry into services to 89.4 percent after 1 year; the percentage of children who had scores below 40 on the Child and Adolescent Functional Assessment Scale more than doubled, from 13.5 percent to 29 percent, indicating these children are no longer clinically impaired in their social functioning; and law enforcement contacts were reduced by 25 percent among children who remained in services after 1 year.