Detailed Information on the
Resident Opportunities and Self Sufficiency Assessment

Reduce timeline for obligation of grant awarded funds through new competition design.

Complete the development and implementation of performance tracking systems.

Develop annual and long-term, outcome oriented performance measures that measure success in service enriched housing for the elderly.

Add measure to compare annual increase in earnings of participants to the annual cost of the salary of service coordinators.

Measure: Increase the number of families or individuals being served.

Explanation:As time passes and the redesigned ROSS program becomes more established the number of families and/or individuals will increase.

Measure: Increase the number of elderly people or persons with disabilities who are able to live independently and/or age in place.

Explanation:Increase the number of elderly or disabled persons that are reported as being able to age in place (remain in thier units) as a result of the services provided.

Measure: Number of families participating in the program that increase their average annual earned income.

Explanation:Increase the number or percent of participants who report an increase in earned income.

Measure: The combined earned income increase of all participating families will be greater than the Service Coordinator funded salary.

Explanation:

Is the program purpose clear?

Explanation: The purpose of the Resident Opportunity and Self-Sufficiency program comes from Section 34 of the United States Housing Act of 1937, as amended by the Quality Housing and Work Responsibility Act of 1998 (QHWRA). Section 34 states that grants may be made "for the purposes of providing a program of supportive services and resident empowerment activities to provide supportive services to public housing residents or assist such residents in becoming economically self-sufficient." Eligible activities are broad, including service coordinators, provision of services directly to residents, and a services program for elderly and/or disabled residents. These activities have been operationalized in many ways over the life of the program, but the underlying purpose has always been the same. In the FY08 Connecting with Communities Users Guide to HUD Programs and the 2008 SuperNOFA Process, the ROSS program is described as such: ROSS- Service Coordinator program is to provide funding to hire and maintain Service Coordinators who will assess the needs of residents of conventional Public Housing or Indian housing and coordinate available resources in the community to meet those needs. The Service Coordinators will assist families to move to economic self sufficiency and/or elderly and disabled residents to maintain independent living. Eligible applicants are Public Housing Authorities (PHAs), Indian Tribes and Tribally Designated Housing Entities (TDHEs), Resident Associations and non profit entities. At the program's inception, the operationalization of the statute resulted in programs that had more of a focus on resident capacity building, resident business development, resident conflict resolution and the like. A large portion of the grants went directly to Resident Councils and Resident Management Corporations. However, over time, the paradigm has changed to, under the same statute, offering self-sufficiency and services coordination programs instead. Resident Councils have received fewer grants over the years, in part because of the Grants.gov process. However, the statute currently states that 25% of grant funds must go to qualified applications from Resident Organizations. It is hoped that with the changes to the ROSS program, more Resident Organizations will be able to apply and be successful. Special attention will be given to these new Resident Organization grantees to ensure their capacity and to evaluate the current state of Resident Organization capacity for this new grant model. Consideration may be given in the future to recommending to Congress a change in the statute that will remove the 25% mandate.

Evidence: The purpose of the Resident Opportunity and Self-Sufficiency comes from Section 34 of the United States Housing Act of 1937, as amended by the Quality Housing and Work Responsibility Act of 1998 (QHWRA), Section 34 states that grants may be made "for the purposes of providing a program of supportive services and resident empowerment activities to provide supportive services to public housing residents or assist such residents in becoming economically self-sufficient."

Does the program address a specific and existing problem, interest, or need?

Explanation: Conventional public housing is currently the housing of last resort. If a family is residing in conventional public housing and that family is able to work, that family may need support services to assist them to move along the spectrum of housing self-sufficiency. If the family is elderly or disabled, they may need services to help them maintain independent living and not be transferred un-necessarily to a nursing home. Typically, social services are funded through multiple agencies and delivered in a piece-meal fashion. This often results in duplication of some services and some needs remaining un-met. Service Coordination, however, streamlines the needs assessment and provides a central point of contact for a family to enable them to access wrap-around services in the community to better meet their immediate needs and longer-term self-sufficiency goals. In addition, when Resident Services programs are offered in the context of publicly assisted housing, they have been shown to benefit occupancy and rent collection functions as well as reduce evictions and maintenance costs. (Proscio, 2006) "Good" management of public housing extends beyond the physical up-keep of property. Stewardship of the property and attention to human needs go hand-in-hand to wards successful, thriving public housing. The ROSS program assists in the human side of that equation.

Evidence: Research has shown that service coordinators benefit occupancy and rent collection functions as well as reduce evictions and maintenance costs. (Proscio, 2006) "Good" management of public housing extends beyond the physical up-keep of property. Stewardship of the property and attention to human needs go hand-in-hand towards successful, thriving public housing. The ROSS program makes possible the human side of that equation. Furthermore, coordination of services streamlines the needs assessment and provides a central point of contact for a family to enable them to access wrap-around services in the community to better meet their immediate needs and longer-term self-sufficiency goals.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The ROSS program is the only competitive program available for supportive services for the vast majority of public housing residents. Supportive services dollars are included in HOPE VI grants, however, these only serve a small portion of PHAs across the country. The PHAs could use other federal funds or rents collected for these activities but, traditionally, these formula-driven funds and rents have not been used for service coordinators or economic self-sufficiency activities. The new design of the ROSS program is specifically intended to reduce duplication of services by providing the service coordinator position and not the services directly. The services will then be coordinated from among the other offerings in the community. Public Housing Authorities are often prohibited from applying for other supportive services programs that may be available to providers in the community because PHAs are not generally 501(c)(3)s. Therefore, the ROSS program presents a unique opportunity to serve residents of public housing directly and exclusively. In addition, the structure of the program, in fact, serves to improve the capacity and coordination of the services offered in a community on the whole, because as the PHA creates its Program Coordinating Committee, it becomes a catalyst for collaboration for serving all people in need in the larger community. Those Housing Authorities that receive the Elderly/Disabled Services Grant through the Operating Subsidy have not been eligible to apply for the ROSS-Elderly/Persons with Disabilities grant in the past and starting in FY08 will have the option of ceasing their EDSC funding if they would like to opt for the ROSS-SC program instead. This is yet another way that limited program funds are targeted to the PHAs most in need and are not redundant. However, it should be mentioned that even with the current funding available, funds are only able to serve approximately ten percent of the Public Housing Authorities across the country.

Evidence: The ROSS program is the only competitive program available for supportive services for the vast majority of public housing residents.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program has struggled for several years over the best way to achieve its purpose. As a result, the program was recently modified administratively to the single eligible use of service coordinators. The direct delivery of services is no longer funded. It will take some time to prove that this new model overcomes the previous design flaws. HUD believes that this new Service Coordinator model for the ROSS program will resolve any prior limits on effectiveness by setting up a program whereby the funds can be used in the most efficient manner possible. This decision was arrived at through intensive interviewing and discussion with the grantee community and industry groups across the country as well as through analysis of other current supportive services programs. It should be noted, however, that the program, as a competitive program, and funded at the current levels, does not meet the existing need. Current research shows the positive impact of resident services not only for the residents themselves, but for the health and well-being of the community and physical housing stock. Finally, the program statutorily requires 25% match. This, and the structure of Services Coordination, requires that grantees effectively leverage the capacity and resources available in the community. It is a highly efficient return on investment for HUD dollars. Currently, the statute states, "The Secretary may not make any grant under this section to any applicant unless the applicant supplements amounts made available under this section with funds from sources other than this section in an amount equal to not less than 25 percent of the grant amount??" That is currently operationalized by the applicants having to request letters certifying in-kind (or cash, but usually in-kind) match from their community partners. Historically, documenting this was difficult, certification of match is now required to be maintained by the Executive Director of the housing authority. This program office inherited the ROSS program in 2005 with $34 million of carry-over due to program NOFA structure and competition processing challenges in previous years. In successive years, the amount of un-used funds rolled-over has been reduced each year, until the point that, in FY07, all but $800,000 in program funds were granted. It is anticipated that even this last $800,000 may be granted upon appeal. Also, each year, the number of grantees has increased from 212 in 2004 to 268 in 2007. During the last year, the Department has aggressively pursued the timely obligation and expenditure of ROSS funds. The program structure that will begin in FY08 has streamlined the application and competition process such that the funds should be awarded several months earlier than in years past. In addition, the program office has been working intensely with the Office of Departmental Grants Management and Oversight to streamline the logic model and reporting requirements so that grantees will be able to determine and report on cost/benefit efficiency questions as well as a Return on Investment figure.

Evidence: The program has been modified recently. It will take time to prove this new model overcomes previous design flaws. The new Service Coordinator model should resolve any prior limits on effectiveness by setting up a program whereby the funds can be used in the most efficient manner possible and better align specific resident needs with the cooresponding service. Since the program office inherited the program the number of grantees has increased from 212 in 2004 to 268 in 2007 and un-used funds from prior years have been drastically reduced.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: 1) The only eligible uses of funds are on activities that directly support the program's purpose. The purpose is broad, but is deliberately so because a single mother with a teenager and two younger children will have much different needs than an elderly person with mobility impairment. The program is designed for local assessment and prioritization that best serves the needs of the residents of a particular Housing Authority. After extensive consultation with the applicant and grantee community, the current program structure seems to be the best way to meet the program goals and remain flexible enough to serve the spectrum of residents that live in Public Housing. In addition, as the population of public housing ages, each PHA will have the flexibility to move the focus of their Service Coordinators from family services to elderly services if that is the area of greatest need. 2) The amount of funding provided is not adequate to meet the need in the public housing community. However, the program is streamlined to allow the grantee PHAs to identify the families most in need/most motivated to take advantage of the services. In addition, the streamlining of the application and competition process will enable HUD to make obligations in a more timely manner and for the grant cycle to be shortened, allowing the grantees more time to simply coordinate services and requiring less staff time for applications and less waiting time. This will also allow for more efficient budgeting for the PHAs. 3) The only eligible direct beneficiaries of the program are residents of conventional public housing, and PHAs must demonstrate through case files that only conventional public housing residents are being served directly with program funds. However, the structure of the ROSS program will actually benefit a much broader community than simply the public housing residents. As the Service Coordinators become more adept at their functions, their services coordination and the existence of the Program Coordinating Committee will benefit the service provider community as a whole. The PHA will serve as the catalyst for stronger and more efficient collaboration across community providers, resulting, again, in less duplication of services and the increased ability of the service providers to specialize, knowing that all service needs of the community are being met by other providers. 4) With the pro-ration of PHA Operating subsidies and the competing priorities of the Capital Fund dollars, these competitive funds are generally the only source of funding that are used for resident services. If any other PHA funds are used for resident services, it is only because the need is so great that both sources are necessary. What's more, however, is that although there may be a plethora of services available in a community, the typical resident of conventional public housing either cannot or has not been able to access those resources. The Service Coordinator makes it possible for the residents to more effectively access the needed services that may have been available, but not taken advantage of, all along.

Evidence: The program is designed for local assessment and prioritization that best serves the needs of the residents of a particular Housing Authority. The only eligible use of funds are on activities that directly support the program purpose.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: 1. Increase the number of families or individuals being served (output and potentially also an outcome) - As time goes on and Service Coordinators and Program Coordinating Committees become more established and communities learn to better collaborate on services provision, the Program Office would hope that the number of families being served increases, either by more families being served by each coordinator or by more coordinators being hired by the PHAs because their worth to the bottom line of Asset Management has been proven. 2. Number of elderly people or persons with disabilities who are able to live independently and/or age in place as a result of being connected to services through this program 3. Number of families participating in the program that increase their average annual earned income.

Evidence: These measures will provide the needed information that show if the participants receiving services provided become self suffiicient and also indicate if elderly participants are able to better age in place as a result of services received.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program had program goals in past iterations of the Annual Performance Plan (APP). These goals were for the former ROSS-Homeownership Supportive Services program and concerned numbers of families being assisted with homeownership counseling as well as the number of households that purchased a home as a result of the program. HUD met and exceeded these goals. As a result of the streamlining of the APP and other performance planning documents, there are no longer APP goals for programs as small as ROSS. Our current targets, as proposed for the PART are to: 1. Increase the Number of Families Being Served - once we can establish a baseline based on the FY08 logic model reporting, our goal will be to increase the number of families being served each year by 5% 2. Increase the Number of People able to Age-In-Place - once a baseline is established based on FY08 logic model reporting, our goal will be to increase the number of elderly people or people with disabilities that are reported as being able to age-in-place in their units with the assistance of services provided by community partners and coordinated through this program. This will allow them to thereby avoid moving to a nursing home and will save taxpayer funds that would have been spent on full-time nursing care through Medicare. We would like to show that elderly residents that participate in this program are able to age-in-place for longer than similar elderly in non-service-enriched housing. 3. Increase the Number of People who Increase their Average Earned Income - once a baseline is established based on FY08 logic model reporting, our goal will be to increase the number or percent of participants in the program who report an increase in earned income after one year. This will not only benefit the individual or family themselves, but will also benefit the PHA because as income increases, the rent paid to the PHA will increase (unless the family is under the Earned Income Disregard or participating in a program like FSS that establishes an escrow account with the difference in rent.) This may be hard to measure, as, if the program functions properly, those with increasing incomes will move on and out of public housing, being replaced again by those who may need to start at the remedial end of the self-sufficiency spectrum. We will assume that the program enrollees will be families that do span the spectrum and that as the program is established, families will be able to take advantage of services more rapidly. Each program will be given a rating score. ?? Outstanding = 60% or more of families participating in the program report an increase in average earned income. ?? Satisfactory = 40% ?? Needs Improvement = 20% or less The program's goal will be that within 6 years of implementation, 80% of all programs are satisfactory or better. When it comes time for renewals, we will use this measure as a criterion for renewal.

Evidence: The previous goals have been removed. The new goals mentioned above, including a) number of families served, b) number of people able to age in place, and c) increase in earned income, are proposed as measures/targets that capture the long-term effectiveness of the program while setting ambitous targets.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: 1. Increase the Number of Families Being Served - Baseline of FY08 grants established in early 2010 2011 - baseline x5% 2012 - 2011 x5% 2013 - 2012 x5% 2014 - 2013 x5% Etc. 2. Increase the Number of People able to Age-In-Place - once a baseline is established based on FY08 logic model reporting, HUD will establish goals for improvement. 3. Increase the Number of People who Increase their Income - once a baseline is established based on FY08 logic model reporting, HUD will move incrementally towards our long-term goal, increasing the percentage of satisfactory or better by 10 % each year until we meet our long-term goal. 4. The annual increase in earnings by participants will be at least twice the cost of the salary of the service coordinator (efficiency/outcome measure).

Evidence: See above. The goals will provide the necessary evidence the program demonstrates progress towards it long-term goals and overall aims of the program. Data is not yet available, as baselines need to be established.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: 1. Increase the Number of Families Being Served - Baseline of FY08 grants established in early 2010 2011 - baseline x5% 2012 - 2011 x5% 2013 - 2012 x5% 2014 - 2013 x5% 2. Increase the Number of People able to Age-In-Place - once a baseline is established based on FY08 logic model reporting, HUD will establish goals for improvement. 3. Increase the Number of People who Increase their Income - once a baseline is established based on FY08 logic model reporting, HUD will move incrementally towards our long-term goal, increasing the percentage of satisfactory or better by 10 % each year until we meet our long-term goal. 4. The annual increase in earnings by participants will be at least twice the cost of the salary of the service coordinator (efficiency/outcome measure).

Evidence: These are new measures. Baselines need to be established and the proper instruments are in place to establish such. Once the baselines are in place the targets relating to each are ambitious.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Grantees are given these management questions at the beginning of their grant and they know that they will be asked to report on them. They are also given a limited number of activities/outputs/outcomes from which to choose on the logic model that shape the type of program they will present. They must report on these activities/outputs/outcomes semi-annually. They also understand that their reporting is the contingency upon which future program funding will be based. In addition, the Grants Management Center at HUD that processes the applications is committed to the streamlining of the competition process and has been and will work with the Program Office to continue to improve the competition process so as to meet some of the long-term goals that pertain to grant making. The Field Operations Office at HUD is the liaison between the program office and the field offices, who directly manage the grants and are immediately responsible for the collection of the logic model reporting and for delivering those reports to HQ to be aggregated. They have already been a strong partner with the program office and ODGMO in this process. They are committed to, along with the program office and ODGMO continuing to provide guidance, training, and TA to the field offices to enable them to better meet these requirements. ODGMO is the originator of the logic model and the reporting process. They, too have a commitment and vested interest in assisting the program office and the field office in increasing efficiency of grant making and reporting. One thing that we could do to codify these relationships and commitments among HUD staff is to integrate performance review criteria into the HUD performance management system for involved employees such as GMC staff, Office of Field Operations staff, and Field Office Grant Managers. This could include, for example, timelines for grant execution and collection and forwarding of logic model reporting. However, the program office would not have the oversight to make this type of mandate.

Evidence: Grantees are given these management questions at the beginning of their grant and they know that they will be asked to report on them. They are also given a limited number of activities/outputs/outcomes from which to choose on the logic model that shape the type of program they will present. They must report on these activities/outputs/outcomes semi-annually, and the reports will affect the award of future grants.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: It is too early to have completed any independent evaluations on this redesigned program. The program is working with HUD's Planning, Development, and Research staff to develop a plan for a rigorous evaluation of the program in the future. Many ROSS programs do contract for evaluations of their own services programs. These have not been collected by HQ and analyzed, but could be, if required. The most exciting and recent research, though, has been and continues to be undertaken by Enterprise Community Partners, Inc., which has recently established an organization called the National Resident Services Collaborative (NRSC) (www.residentservices.org) The National Resident Services Collaborative was established by several national, regional and local community development organizations to improve and increase the delivery of resident services for families in affordable housing. To this end, they have been sponsoring research on the efficacy of services coordination in public housing. Two recent publications are 'Building Opportunities for Families into Affordable Housing' located at: http://www.practitionerresources.org/cache/documents/54273.pdf and 'More Than a Roof and Walls' located at: http://www.practitionerresources.org/cache/documents/56278.pdf. As a result of this study, they are now undertaking a larger study of similar subjects. Their recent Policy Symposium, in which ROSS program staff took part, produced several discussion papers that provide direction on policy and future research on the topic. http://residentservices.org/policy.asp The Harvard University Joint Center on Housing Policy supplies a plethora of research papers that address the topic of resident services in Public Housing (http://www.jchs.harvard.edu/publications/rental/index.html) Here is but one example: The purpose of all these different forms of housing subsidies is, first and foremost, to help assist people afford decent rental housing, and local authorities are, by and large, achieving that mission. So why bother with matters of work and self-sufficiency, which are traditionally the responsibility of other government systems? There are at least three reasons, examined below: (1) to reach many of the nation's work-capable poor in need of employment support, (2) to counter possible negative work influences associated with housing assistance, and (3) to increase access to a limited supply of housing assistance by cycling recipients through the system more rapidly. (Riccio, 2006)

Evidence: It is too early to have completed any independent evaluations on this redesigned program. The program is working with HUD's Planning, Development, and Research staff to develop a plan for a rigorous evaluation of the program in the future. Many ROSS programs do contract for evaluations of their own services programs however these have not been collected by HQ and analyzed, but could be if required. Additionally many practitioners and other expects are conducting extensive research on the issue of self sufficiency that address the topic of resident services in Public Housing, provide direction on policy and future research on the topic and overall improve the delivery of services as they realize the importance and benefits these programs bring.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: As the annual and long-term goals have only recently been established, the budget requests have not, as yet, been tied to the accomplishment of those goals. The budget requests have, for the most part, remained consistent, though dropping when the program had significant carry-over funds. However, in anticipation of fully utilizing available funds in FY09, the Department requested a figure consistent with prior funding minus the rescission factor. In the future, it is hoped that as the program continues to show its efficiency in grant-making, as data collection shows the efficacy of program implementation and as research continues to show that supportive services in a housing setting has a positive impact on the residents as well as the hard-side project budget, the Department and Congress will consider increasing the funding for this program.

Evidence: As the annual and long-term goals have only recently been established, the budget requests have not, as yet, been tied to the accomplishment of those goals.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has undertaken a significant re-design for the FY08 grant cycle. This new design is based on the successful FSS program, designed to maximize leverage and streamline the application and competition process for both applicants and HUD. In addition, the program has moved to undertake specific, ambitious long-term and short-term performance goals to better measure the efficiency of the program and continued improvement.

Evidence: See above.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: 1) The agency collects semi-annual reporting and logic models from grantees. These have in the past gone no further than the grant managers in the field offices. Starting with FY2006 grantees, however, these logic model reports are coming to HQ to be aggregated. Please see above for aggressive, specific goals. 2) This process will be used to evaluate the performance of the overall program and improve performance where possible. It will also be used to assess the need for further training or technical assistance for field offices and/or grantees. In addition, an analysis of services connected will inform the program office of possible federal-level partnerships to be pursued. 3) All of the data considered is from the grantees, our program partners, the field offices, the Field Operations division, the Office of Departmental Grants Management and Oversight, or the Grants Management Center. 4) HUD has some of the baseline data in place, and are still in the process of collecting what will be our baseline performance data for our meaningful, ambitious targets.

Evidence: The Department now collects semi-annual reporting and logic models from grantees containing information on who the program is serving, services available and progress of the client. Up to this point the Department has not had the mechanism to collect national level data. The mechanism has been established and beginning in FY '08 the data collection is now achievable.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: 1)There is one program office team consisting of a director and a program manager who are ultimately responsible for program results. 2) The program office team has accountability measures for making the grants and executing the grants written into their personal performance review tools. These include measures on the efficiency making the grants and making sure that the grants are executed in a timely manner in addition to measures on providing training and guidance to grantees and field office personnel. In the future, the manager's performance evaluation will be tied to the annual goals for the program. These are the only established performance standards, though many other tasks exist. 3) The Logic Model is the current performance standard and accountability tool for the grantees of the program. Currently, this is a semi-annual requirement. Starting with FY08 grantees, it will be annual, but more in-depth. In addition, the grantees are made aware that there is a three-year time-frame for expenditure of the grant funds. Extensions are made on rare occasions, however, notice has been given in the NOFAs for the past two years that failure to expend grants within the grant program timeframe may result in an impact on the Past Performance review on future competitive grant applications.

Evidence: The program office team has established accountability measures for making the grants and executing the grants written into their personal performance review tools. Program partners/grantees are held accountable for specific performance and accountability via the reporting requirements of the logic model.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: 1) Between FY04 and FY07, the program had challenges in the timely obligation of funds due to difficulties with writing the Notification of Funding Availability (NOFA) and processing applications in such a way that a sufficient number of applicants could successfully qualify for funding. This program office inherited the program at the end of the 2005 SuperNOFA process with a significant carryover of unawarded funds. Each year that this program office has had the grant, the amount of carryover funds has been reduced and the number of grantees has increased. In FY07, all but $800,000 was awarded (and it is anticipated that this amount will be awarded in appeals) to a record number of grantees. 2) The schedule for the obligations is subject to outside factors such as the SuperNOFA process and the timeline of the Grants Management Center. However, the timing of obligations has been fairly consistent from year to year. With the onset of the new program design, the program hopes to considerably shorten the timeline of obligations - within the confines of the other factors. Please see annual and long-term goals. Also, with the new program model, it is anticipated that there will be fewer requests for no-cost extensions of grants, based on current rates of requests between the FSS and ROSS programs. 3) Expenditures are tracked by the field office grant managers. This includes a review of the semi-annual financial report. The Program Office is currently in the process of developing guidance and training for Field Office grant managers on this topic. Field offices do participate in addressing audit findings and unresolved audit findings are taken into consideration in the grant-making competition process. 4) In sum, we believe the corrective actions will result in a yes answer in the future but more time is needed to be certain.

Evidence: HUD believes the redesigned program will overcome the historical problems in obligated funds in a timely manner. More time is needed to demonstrate that this new design will correct the deficiency.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program is working to develop and implement measures of cost effectiveness but they are not yet in place. 1) As of FY08, the main eligible expense for the program will be the salary of a service coordinator. All services must be donated from partner agencies. Internally, the costs of running the program are being decreased along with the streamlining of the program. The program has only one 75% staff person and one 20% director. The program is supported by ODGMO and the Office of Field Operations by employees who serve ROSS along with a plethora of other duties. The new model of the program will also decrease funds needed for contract staff at the Grants Management Center to run the competition. The decentralized grant management system whereby the grants are managed at the field office level enables the Field Office grant managers to have a more intimate knowledge of the grantee as a whole and the ROSS program specifically, and better enables them to detect problems and provide targeted technical assistance when required. 2) Please see the annual and long-term goals above. Note that a program efficiency and performance measure will be added, when available, where the increase in earnings of participants will be compared with the annual costs of the program. The output in terms of the increase in the particpants' earnings should exceed the costs of the inputs.

Evidence: The program is working to develop but does not yet have procedures in place to ensure the cost effectiveness of this self-sufficiency program.. An efficiency/effectiveness measure will be added to compare the outputs in terms of the increase in the earnings of particpants to the annual cost of the inputs. As of FY08, the main cost of program will be the salary of a service coordinator. All direct services must be donated from partner agencies.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Program staff is highly collaborative with other HUD programs that provide services to public housing and Section 8 residents through one-on-one meetings and a newly established internal working group. Program staff is also involved in coordinating councils convened by DOJ, HAY, Education and the Federal Mentoring Council and is working toward making the NOFA process more collaborative across federal agencies. In addition, because HUD is re-directing their funding for direct services, program staff is working on these coordinating councils to encourage other agencies to open their services funding programs to application by or partnership with Public Housing Authorities. By its very nature, the Service Coordinator program collaborates and coordinates with local programs that provide services to people in need in the area. These programs can be faith- and community-based non-profits or the local branches of federal, state, or city funded assistance programs such as Department of Labor Workforce Investment Board One-Stops and local TANF agencies. The program is designed specifically to augment, not overlap or replace other service systems and programs.

Evidence: Collaboration occurs regularly with internal HUD staff along with other federal agencies including DOJ, DOE, HAY and Federal Mentoring Councils to coordinate service resources and encourage service provider participation.

Does the program use strong financial management practices?

Explanation: Financial management is conducted at the field office level. Grantees are required to submit financial reports semi-annually. Field Office grant managers monitor draws through the LOCCS system. Field Office grant managers conduct remote and on-site monitoring reviews as is possible. HUD would like to provide more training/guidance for field office grant managers on this type of monitoring in the future. HUD has simplified and streamlined financial management by limiting the competitive grants to paying the salaries of qualified service coordinators.

Evidence: Program financial records are monitored through the Line of Credit Controls System (LOCCS) and the Field Office level. Both remote and on-site reviews are conducted to insure grantees are properly using funds. HUD has simplified and streamlined financial management by limiting the competitive grants to paying the salaries of qualified service coordinators.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Seeing the large amount of carry-over in the past several years, the program office has undertaken significant administrative changes in NOFA writing, outreach, and training, which have had the ultimate result of diminishing the amount of carry-over to currently almost $0. The program has been redesigned to limit fundng to service coordinators. HUD plans to monitor the competition and implementation of this new program closely so as to circumvent any future managerial challenges.

Evidence: The program redesign has addressed prior management deficiencies, particularly in efforts to reduce carryover funds from prior years and focus funding on what is needed and will deliver results. The general streamlining provides a central point of contact for a family to enable them to access wrap-around services in the community to better meet their immediate needs and longer-term self-sufficiency goals.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The NOFA process is clear and competitive. The Grants Management Center follows the NOFA to the letter during the competition. 1) The SuperNOFA process is fair and open. All eligible applicants may apply through grants.gov. The grants.gov process does provide its own challenges to the capacity of otherwise strong applicants and may have resulted in otherwise qualified applicants not meeting the applications requirements. 2) 100% of the awards are distributed according to the competitive process outlined in the NOFA. 3) The SuperNOFA is a public document, published in the Federal Register. Program staff responds to all requests for training at statewide or regional conferences, having presented at eleven in FY07 and anticipating nearly as many in FY08. Program staff does a webcast available to all applicants. In addition, the Program Manager is available for all technical assistance calls and emails, responding to over 600 inquiries in each of the last two funding cycles. Each year includes both grantees who have had the grant before as well as new grantees. Past grantee status is not a consideration in the competition. 4) This program does not currently include renewals, though may in the future.

Evidence: The NOFA process is open, clear, and fully competitive.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The grants are managed at the local field office level, so the Field Office grant managers are able to have intimate knowledge of each grantee, their activities and expenditures. The grantees are required to report semi-annually using the logic model and a financial report. Site visits are encouraged, but not required, determined for the most part by Field Office funding (and field offices often do not have enough funding to conduct even the required site visits for other programs.) The field office grant managers monitor spending through draws from the LOCCS financial system and the system is set up to lock if semi-annual reports are not received. As PIH moves toward Asset Management, there will likely be a re-alignment of the field office staff and staff responsibilities. This may mean that the role of ROSS Grant Manager will be doled out among the portfolios of all of the PHRSs in the Field Offices. This will be good, in that each PHRS will become an expert on their PHAs. However, the challenge for ROSS will be to maintain consistency in grant management and to communicate a more dispersed workforce.

Evidence: Grants are managed at the Field Office level and staff has intimate knowledge of each grantee and their activities and expenditures. Management at the local field office level insures greater oversight and better communication between HUD and the grantee.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The grantee performance information is not yet accessible but will be in the future. The amount of funds appropriated and carry-over are published with the NOFA in the Federal Register. The grant awards are published by press release on the HUD website with a list of all grantees and funds awarded. Starting with the FY08 SuperNOFA cycle, the highest ranking application will be posted for viewing on the HUD website. All former APP goals and outcomes are published in the Performance Accountability Report. The program and budget justifications are published in the Congressional Justifications for 2009 Estimates. Once the logic model reporting is collected and aggregated, the plans are to publish that outcome data on a program level, on the HUD website. There are also plans, apparently, to publish each program's logic model goals and outcomes on the HUD website as well.

Evidence: The grantee performance information is not yet accessible but will be in the furture. 1) Amount of funds appropriated and carry-over are published with the NOFA 2) The highest ranking application will be posted for viewing on the HUD website 3) Once the logic model reporting is collected and aggregated, the plans are to publish that outcome data on a program level, on the HUD website. There are also plans, apparently, to publish each program's logic model goals and outcomes on the HUD website as well.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Prior to FY '08 the Department has not had the mechanism to collect and aggregate data at the national level. Beginning in FY '08 that mechanism has been established and from this point forward the program will be much more effective in demonstrating progress towards long term goals. The program has decreased the carryover of funds each year for the past three years and has increased the number of grants made each year during that same time. Each year, the NOFA has been streamlined, and, for FY08, major program deficiencies have been addressed by a significant re-design of the program NOFA. In addition, the logic model reporting process has been put into place, first by introducing the logic model as a tool and then by requiring a national-level collection and aggregation of the data. This streamlining of the program model as well as the implementation of this second stage of data collection has set the stage for achieving the aggressive annual and long-term goals set by the program. So, as of yet, the long-term goals have only just been established. It will take time to show adequate progress towards those goals.

Evidence: Prior to FY '08 the Department has not had the mechanism to collect and aggregate data at the national level. Beginning in FY '08 that mechanism has been established and from this point forward the program will be much more effective in demonstrating progress towards long term goals. With program redesign and the establishment of new long-term goals new data collection efforts will likely take until 2010 before results can be demonstrated. This streamlining of the program model as well as the implementation of this second stage of data collection has set the stage for achieving the aggressive annual and long-term goals set by the program.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Prior to FY '08 the Department has not had the mechanism to collect and aggregate data at the national level. Beginning in FY '08 that mechanism has been established and from this point forward the program will be much more effective in demonstrating progress towards annual goals. The program successfully awarded all of its appropriations and nearly all of its program funds for the FY07 grant cycle. This represents the culmination of several years of movement towards this goal. The baselines have been, or are currently being set based on the grants made during this process. They will enable the program to better assess its progress and modify the course or program if necessary. So, as of yet, the annual goals have only just been established. It will take time to show adequate progress towards those goals.

Evidence: Prior to FY '08 the Department did not have the mechanism to collect and aggregate data at the national level. That mechanism has now been developed and beginning in FY '08 data will be collected at the national level and the new goals established and mentioned in the previous sections will be tracked and reported on. With program redesign and the establishment of new long-term goals new data collection efforts will likely take until 2010 before results can be demonstrated.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: This self-sufficeincy program doesn't measure cost effectiveness yet. Plans are underway to add a cost effectiveness measure that will measure the outputs, in terms of the increase in annual earnings of its participants, to the annual cost of the program. The baselines for efficiencies or cost-effectiveness have only just been established. It will take time to show adequate progress towards those goals.

Evidence: See above. The baselines have been, or are currently being set based on the grants made during this FY07 process. They will enable the program to better assess its progress and modify the course or program if necessary.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: This program, modeled after other self-sufficiency programs, should be highly valuable; however, it is too early to compare with similar programs. Being one of the few programs to serve residents of public housing in this way, it is hard to assess. There are other programs that serve similar residents (e.g. the multi-family service coordinators program that assists residents of HUD-assisted housing or the Family Self-Sufficiency programs that serve Housing Choice Voucher recipients) and the ROSS program has been modified for FY08 to be more like those programs with the intention that the funding of only the coordinators will streamline the application/competition and make the federal cost/benefit ratio greater. This program serves a broader range of residents, though, than most other programs, in that it is designed to be flexible enough to serve both families and elderly/disabled residents. This is a unique aspect of the ROSS program and makes it ideal for public housing, which serves residents all along the age and ability spectrum.

Evidence: It is too early to evaluate the performance of this program with other similar programs.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: While the ROSS program specifically has not been the subject of an independent program evaluation, there is a great deal of research being undertaken in the housing community on this topic currently. The most exciting and recent research, has been and continues to be undertaken by Enterprise Community Partners, Inc., which has recently established an organization called the National Resident Services Collaborative (NRSC) (www.residentservices.org) The National Resident Services Collaborative was established by several national, regional and local community development organizations to improve and increase the delivery of resident services for families in affordable housing. To this end, they have been sponsoring research on the efficacy of services coordination in public housing. Two recent publications are Building Opportunities for Families into Affordable Housing http://www.practitionerresources.org/cache/documents/54273.pdf and More Than a Roof and Walls http://www.practitionerresources.org/cache/documents/56278.pdf. As a result of this study, they are now undertaking a larger study of similar subjects. Their recent Policy Symposium, in which ROSS program staff took part, produced several discussion papers that provide direction on policy and future research on the topic. http://residentservices.org/policy.asp Several recent studies have shown that when residents' economic or family emergencies are addressed in a timely way, they are less likely to fall behind on rent, vacate their apartment or be evicted, or create other problems for managers. When children are constructively occupied and well-supervised during the out-of-school hours, they are less likely to disrupt other residents or do harm to themselves or the building, and they are more likely to grow up healthy and well-adjusted. (Procio, 2007). In addition, family affordable housing properties perform better when resident services are provided. A successful resident services program can improve a property's overall performance by directly affecting the following key indicators of a property's success: the occupancy rate, rent collection rate, eviction rate, maintenance costs, turnover rate, net operating income and cash flow. (Tassos, 2007) The Harvard University Joint Center on Housing Policy supplies a plethora of research papers that address the topic of resident services in Public Housing (http://www.jchs.harvard.edu/publications/rental/index.html) Here is but one example: The purpose of all these different forms of housing subsidies is, first and foremost, to help poor people afford decent rental housing, and local authorities are, by and large, achieving that mission. So why bother with matters of work and self-sufficiency, which are traditionally the responsibility of other government systems? There are at least three reasons, examined below: (1) to reach many of the nation's work-capable poor in need of employment support, (2) to counter possible negative work influences associated with housing assistance, and (3) to increase access to a limited supply of housing assistance by cycling recipients through the system more rapidly. (Riccio, 2006) Procio, Tony. 2006. More Than Roof and Walls: Why Resident Services Are and Indispensable Part of Affordable Housing. Enterprise Community Partners, Inc. http://www.practitionerresources.org/cache/documents/56278.pdf (www.residentservices.org) Riccio, James. 2007. Subsidized Housing and Employment: Building Evidence about What Works to Improve Self-Sufficiency. Harvard University Joint Center for Housing Studies http://www.jchs.harvard.edu/publications/rental/revisiting_rental_symposium/papers/rr07-6_riccio.pdf Tassos, James. 2007. State Leadership in Encouraging Family Housing Enriched with Resident Services: An Assessment of 2007 Housing Credit Allocation Policies. Enterprise Community Partners, Inc. http://www.practitionerresources.org/cache/documents/657/65731.pdf

Evidence: Currently a great deal of independent research is being undertaken and have shown that when residents' economic or family emergencies are addressed in a timely way, they are less likely to fall behind on rent, vacate their apartment or be evicted, or create other problems for managers. When children are constructively occupied and well-supervised during the out-of-school hours, they are less likely to disrupt other residents or do harm to themselves or the building, and they are more likely to grow up healthy and well-adjusted. In addition, family affordable housing properties perform better when resident services are provided. A successful resident services program can improve a property's overall performance by directly affecting the following key indicators of a property's success: the occupancy rate, rent collection rate, eviction rate, maintenance costs, turnover rate, net operating income and cash flow.