Detailed Information on the
Nursing Education Loan Repayment and Scholarship Program Assessment

Test methodology to identify critical nursing shortage facilities

Develop an Information System to replace BHCDANET (legacy system) to better manage data on scholars, loan repayers and sites.

Will maximize the impact of the program by targeting providers to nursing homes, hospitals that serve a disproportionate number of low-income patients under Medicare and Medicaid, and other priority health facilities.

Will conduct an evaluation of the program's impact, develop outcome measures, and begin to track performance against newly adopted benchmarks by developing a baseline and refining performance targets.

Make performance data available to the public.

Improve program efficiency in processing applications and making awards by replacing the current Citrix-based application system with a web-based (.Net) system.

Measure: Increase the number of individuals enrolled nationwide in nurse education and training programs.

Explanation:Baseline: Students in all pre-licensure R.N. programs in the 2003-2004 academic year.

Measure: Percentage of program participants that serve in nursing homes, hospitals that provide care to a disproportionate number of low-income patients under Medicare and Medicaid, and public health departments and clinics compared with 2003.

Explanation:*The FY 2010 target was established in PART before baseline information was available.

Measure: Percentage of participants who remain employed at the health facility for at least a year after completing their federal service contract.

Explanation:The 2004 result is based on a survey of nurses who had completed NELRP service. Revised targets were established, but are lower than the reported 2004 pending additional analysis of the data.

Measure: Increase the proportion of program participants who extend their service contracts and commit to work at a critical shortage facility for an additional year.

Explanation:

Measure: Increase the proportion of NELRP participants who remain employed at a critical shortage facility for at least one year beyond the termination of their NELRP service

Explanation:

Is the program purpose clear?

Explanation: There is a general consensus that the specific purpose of the program is to increase the number of nurses serving in facilities that face challenges with recruitment and retention. Program managers view the program as a way to help place nurses in specific facilities where they are most needed to improve care. Some views expressed by interested parties indicate a more broad purpose of addressing a nursing shortage. The program repays up to 85% of the principal and interest of any qualified nursing education loans loan for nurses in return for up to three years of service in an eligible health facility. Eligible nurses are those who received a baccalaureate or associate degree in nursing, a diploma in nursing, or a graduate degree from an accredited school of nursing. New 2002 authorizing legislation broadens the type of facilities eligible to receive a nurse supported by the program, establishes a scholarship component and renames the program the Nursing Education Loan Repayment and Scholarship Program. The program is part of a Department of Health and Human Services Secretarial Initiative.

Evidence: The Nursing Education Loan Repayment Program was amended in 1998 and again in 2002 (section 846 of the Public Health Service Act). Agency and Congressional statements related to the program are consistent with the program purpose as outlined in the authorizing legislation. The program had been authorized to place registered nurses specifically in community health centers, Native American and Native Hawaiian health centers, public hospitals, rural clinics, and public or private nonprofit health facilities with a critical shortage of nurses. The Nurse Reinvestment Act of 2002 expanded eligible facilities to include any health care facility with a critical shortage of nurses, including private for-profit facilities, and gives preference only based on the financial need of the applicants. The new legislation also establishes a nursing scholarship program. The program is run by the Health Resources and Services Administration (HRSA).

Does the program address a specific interest, problem or need?

Explanation: The program is designed to address the problem of nurse vacancies and low ratios in health facilities. The program addresses the problem of shortages by providing funds directly to nursing graduates who agree to provide care in a facility with a critical shortage of nurses. Shortages are a fairly subjective measure, however, evidence from the program and others suggest an insufficient number of nurses in place and in training. There is evidence of more acute shortages in specific health care facilities. By giving preference by financial need, the program maintains an equity element to support those nurses with the greatest financial burden.

Evidence: Nursing is the single largest health profession. Projections from HRSA indicate a shortage of 110,000 registered nurses in 2000 (a national supply of 1.89 million nurses and demand of 2 million) and an estimated shortage of 800,000 by 2020. The Bureau of Labor Statistics is also projecting a shortage of nurses. The American Hospital Association reports 75% of hospital vacancies are for nurses. GAO reports the national unemployment rate for RNs was 1% in 2000. Nursing impacts the quality of care. Researchers have found higher levels of nursing care provided by registered nurses are associated with better care for hospitalized patients.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: In its current form, the program is not designed to have a significant impact on the problems of nursing distribution and supply. However, the weighting of this question is reduced because the main impediment to having a significant impact is the program's size, which is not merely a factor of program design. The program provides a direct financial incentive for registered nurses with student loans to enter service in any health care facility facing a nursing shortage. By placing nurses in facilities facing a shortage, the program could have an impact on the problem of the distribution of nursing professionals. The program was first designed to be relatively small. The agency does not have data on the number of facilities that are eligible under the current authorization, but the new design captures a broader list of potential entities. For example, the new definition can include hospice centers, nursing homes, and other facilities in addition to hospitals, health centers and other clinics.

Evidence: According to the National Sample Survey of Registered Nurses, hospitals, public and community health settings, ambulatory care settings, and nursing homes and extended care facilities are the main employment settings for nurses. Facilities with a critical shortage of nurses that have been added to the new authorization include: public health clinics, ambulatory surgical centers, home health agencies, hospices and skilled nursing facilities. For profit entities are eligible until 2007 under the new authorization. This list captures thousands of health care facilities. In its current form, the program is supporting only 560 contracts. The program received approximately 6,000 requests for applications in FY 2002. Multiple other factors nurses report as reasons for leaving the profession that the program is not designed to address include direction over patient care, workload, support staff, salaries and hours. The program is now authorized at such sums as necessary, but was authorized at $5 million in FY 1993 and $6 million in FY 1994.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: The Nursing Education Loan Repayment and Scholarship Program is the only Federal program that is designed to provide a financial incentive directly to registered nurses to send them into shortage facilities as a means of improving access to health care in public and private settings. The NHSC supports advanced practice nurses that serve as primary care providers, and not RNs in direct nursing. Private foundations and professional associations, along with some state governments, offer scholarships to encourage students to enter study in nursing. The focus of this program is to improve the distribution of the existing registered nurse workforce.

Evidence: Nurse practitioners and certified nurse-midwives are also eligible for support through the National Health Service Corps (NHSC) loan repayment and scholarship programs in exchange for service in a shortage area. The Department of Veterans Affairs' National Nursing Education Initiative offers scholarships for registered nurses who return to school to attain baccalaureate and advanced degrees, but only for those nurses in service to the VA. The Army, Navy and Air Force also support nursing scholarships, but in exchange for service in the military.

Is the program optimally designed to address the interest, problem or need?

Explanation: The program provides direct payments to registered nurses in exchange for serving in a facility facing nursing shortages. Aspects of the program design will have an impact on the focus and efficiency of the program. Unlike the National Health Service Corps (NHSC), the program is focused on eligible health care facilities with a shortage of nurses, rather than geographic areas, and does not focus on sites that serve patients with multiple barriers to care. However, unlike the NHSC, the program authorization leaves the allocation between scholarships and loan repayments up to the discretion of the Secretary.

Evidence: There is no evidence that a block grant to states or other mechanism would be more efficient or effective in addressing the problem.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has adopted new long-term measures that are useful and capture important elements of program impact. Selecting a measure of impact on a large problem is difficult for a small program such as this one. As the program matures, further work may be needed to improve the measurement of key outcomes. The program's first measure tracks the impact of the program on increasing student enrollment in nurse training programs. Increasing enrollments in nursing is important to stave off an anticipated nursing shortage and help improve shortages within specific types of health care facilities. The addition of a scholarship component may serve as an additional incentive to encourage students to pursue careers in nursing. The program's second and third goals track placement and retention, and by themselves do not constitute true outcome measures.

Evidence: The program's long-term measures are useful but focus mostly on outputs. The third measure relates to program impact by capturing the portion of participants who continue to serve after the end of the contract. The program's long-term measures include: 1) Increase by 10% by 2010 the number of individuals enrolled in nursing training programs; 2) Increase to 25% by 2010 the proportion of program participants working in priority shortage facilities such as: disproportionate share hospitals for Medicare and Medicaid, nursing homes, public health departments (state or local) and public health clinics contained in these departments; 3) Increase to 12% by 2010 the proportion of program participants who remain employed at a critical shortage facility for at least one year after they have fulfilled their service contracts.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: During the assessment process, the program has adopted annual output measures that would demonstrate progress toward desired long-term outcomes. These goals are in addition to the GPRA goal previously used that tracked the annual number of contracts.

Evidence: The annual goals measure the number of nurses supported by the program, the percentage who extend their contract, and the percentage who remain in service a year after no longer receiving support.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: Program partners include the participating RN's, the facilities that are identified as eligible for the program, lending agencies, and the loan verification and application contractors. Nurses supported by the program commit to a period of service in a health care facility in return for financial incentives in the form of loan repayments, and in the future also scholarships. The program maintains contact with recipients, verifies loan balance information with lending agencies and verifies employment with employing agencies through the contract period.

Evidence: The employer verification is completed by the employer every six months and indicates employment status and salary. Loan verification includes an initial credit check for Federal loan defaults and a status check with the lender every six months. The program is examining ways to measure retention, which is a good indication of recipient support of the long-term goals of the program.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: The program does not yet have evidence of collaboration leading to meaningful actions in management and resource allocation.

Evidence: There are examples of the program working with other Federal activities. The program is collaborating with other HRSA units on a nursing HPSA designation and learning from the National Institutes of Health Loan Repayment Program to improve processes to monitor retention.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: No independent evaluations of the program have been conducted. As described below, the program is to develop an evaluation plan to meet a Congressional requirement to report on results and to further support program improvements.

Evidence: The program is relatively new and until FY 2001 was funded at less than $2.3 million. No funds were expended out of this amount to contract out an evaluation and no third parties have conducted comprehensive evaluations on their own.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The program can estimate the associated cost of each nurse placement, which is directly associated with the program's desired outcomes. The program budget structure is fairly straightforward and clear and does not vary markedly from program goals. While the program's annual budget display does not meet all standards of alignment, the program's ability to attribute cost to the key output is sufficient to meet the standards of this question. The agency is working to tie budget planning to strategic planning. The program can estimate outputs (number of placements) per increased increment of dollars. The program does not yet survey retention rates and cannot, however, estimate the impact of funding changes on the total directly supported and retained workforce. Program management funds are budgeted elsewhere. The addition of a scholarship component to the program would require additional effort to align budgeting and planning for the program.

Evidence: This assessment is based on the annual budget submission to OMB and the Congress, and other information provided by the agency.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The main deficiencies related to this section are in collaborating with other programs and having planned evaluations. Now that the program has grown, the program plans an independent evaluation of program participants in FY 2004, using data from FY 2001 and FY 2002 awards as the baseline. The agency's electronic data system can also improve the use of performance information in budgeting and planning. The Division of Nursing has a long history and is experienced in collaborating with other Federal programs and the program plans to increase collaboration now that the program is funded at a larger level. The agency overall is making organizational changes which will further integrate budget and performance planning. Additional work is also underway to consider improved long-term outcome measures.

Evidence: The assessment is based on discussions with the agency. Under the new authority, the program is required to submit a report to Congress within 18 months of enactment that describes numerous aspects of the program's performance, including an evaluation of the overall costs and benefits of the program. The legislation also calls for a report from the Comptroller General within four years of enactment on nursing shortages and hiring practices according to the type of facility, as well as on the impact of the new scholarship program on enrollment in schools of nursing. The program expects the new data collection efforts will allow for an analysis of other program elements such as location of practice, types of facilities served, and retention rates of the nurse recipients.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Overall, the program is regularly collecting performance information useful for management. Managers confirm program requirements are being met on a regular basis and this information is current enough to be useful. Collecting timely and credible performance information will be especially critical as the program develops the new scholarship component in order to make resource allocation decisions between the two instruments to maximize program performance. Critical performance data on retention after the two to three year service agreement is not being collected. This data would provide the program useful information on how well it is meeting its long-term goals.

Evidence: Program staff are responsible for verifying loan payment and employment every six months.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: The agency's senior managers will be held accountable for program operations, including results, through their annual performance contracts. The nursing provider only needs to provide a payment history showing the Federal award has been applied to his or her loans in the event of an amendment contract, however, the program also confirms payments directly with the lender. Performance information could be extended to program staff performance evaluations or contracts.

Evidence: If the contract is breached, participants will be liable to pay the total amount of loan repayments paid plus interest. Unlike the National Health Service Corps, there is no penalty for breach of contract. Currently, payments are distributed monthly through an electronic funds transfer to a checking or savings account that the participant designates. It is the participant's responsibility to see that loan payments are made to the lenders. The program is examining the option of making payments directly to the lender, which could improve accountability.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Loan repayments are awarded annually with sufficient time to shift awards to alternates in the event a potential recipient declines the award. The program confirms with lenders that loan repayment awards are spent for the intended purpose and is exploring the option of making payments directly to the lender.

Evidence: Assessment based on apportionment requests and annual budget submissions.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has developed a web-based application that it expects will improve program efficiency, data collection, and oversight and analysis by the program staff. As noted in Section IV, these changes have enabled the program to increase their approved contracts to staff ratio. With respect to achieving its goals, the agency predicts a continual increase in tuition costs, which will drive-up the average cost of placing a nurse through the program. Managing the impact on placements per Federal dollar will be an important factor if tuition costs rise quickly.

Evidence: The program has contracted out specific services, including the development of a web-based application. The agency is exploring competitive sourcing options. The program is also using a new database to collect information on educational preparation, types of facilities, correlation between award and financial need, and the geographic distribution of placements.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: The program does not have a financial management system that fully allocates program costs and associates those costs with specific performance measures. The program has not developed a procedure for splitting overhead and other costs between outputs. The program does not capture all direct and indirect costs borne by the program agency, including applicable agency overhead, retirement, and other costs budgeted elsewhere, or include informational displays in the budget that present the full cost of outputs. Formulation and execution are also not driven by performance goals. Given a budget total, the program can estimate the number and average cost of two and three year loan repayment contracts. For example, with an average cost of a new contract for FY 2003 at $21,000, the program can estimate the number of new contracts that can be funded at a given level.

Evidence: The assessment is based on annual budget submissions to OMB and Congress. The program does not have an agency program budget estimate that identifies all spending categories in sufficient detail to demonstrate that all relevant costs had been included or a report that shows the allocation of overhead and other program costs to the program. Program managers budget for grants, grant review, travel and technical assistance. Staffing, space, and overhead are budgeted for within the agency program management budget line.

Does the program use strong financial management practices?

Explanation: HRSA received its first clean audit in 1999.The 2000-2001 agency financial statements showed no material weaknesses. HRSA financial statements are conducted by the Program Support Center. The IG found in a 2002 audit of HRSA's travel, appointments, and outside activities that there was no evidence of substantive violations, but that there are technical lapses requiring improvement. The agency disagrees with the breadth of the problem and has re-issued guidance to improve oversight.

Evidence: The assessment is based on agency financial statements and IG audits. Applicants are not eligible if they have a judgment lien against their property for a debt owed to the United States, have breached an obligation for professional service to a Federal, State, or local government entity, are in default of a Federal debt (e.g., student loans, delinquent taxes, etc.) or are not considered by their creditors to be in good standing.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The purpose of this question is to register credit where a program does not meet the standards of individual questions in this section, but is taking meaningful steps to address those specific deficiencies. The main deficiencies in this section relate to the development of the full annual cost of operating the program to achieve desired performance and the availability of performance data to the public. The program is actively engaged in developing new goals, which is a key first step for the program to develop the full cost of meeting performance levels. Tracking performance on key outcomes will also enable the program to make meaningful performance information available to the public.

Evidence: The program is reviewing program policies and procedures related to grant application materials, application review, repayment awards and compliance with program requirements. The program is adopting long-term and annual performance goals, and will be in a better position to advance the alignment of the budget with those goals. The program is also implementing a new electronic on-line application and data reporting system and has taken steps to improve the compatibility of the system with software used by program applicants.

Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process?

Explanation: Because it provides loan repayments directly to individual nurses, the program does not use a peer review process for making loan repayment awards. However, the process is competitive and fair and is based on clear criteria including those established by law.

Evidence: "The criteria used to determine the eligibility of a health care facility to receive a nurse now includes only that the health care facility face a nursing shortage. The program also emphasizes financial need of the nurse professionals in the application process. "

Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process?

Explanation: By design, the program encourages the participation of new and first-time nurses. The application is open to all nurses with student loans who meet program requirements, and the majority of awards are made to first time applicants.

Evidence: The program provides application materials on the Internet and allows recipients to submit the application on-line. Nurse professionals are only eligible for a second contract with the program if they have returned to school in nursing and have incurred new student loans. The program received approximately 6,000 requests for applications in FY 2002.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program confirms annually that obligated nurses are serving in approved facilities, and monitors loan debt to insure funds are used for paying down loans.

Evidence: The program confirms payments directly with the lender and checks with employers to monitor whether the program recipient remains in service in an approved facility.

Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The only annual performance data currently made available to the public is the number of contracts awarded by the program. Important aspects of program performance to be collected and made public in the future include retention rates after the two to three year period of required service, the correlation between actual awards and financial need, and the distribution of nurse professionals by facility and geography.

Evidence: Assessment based on agency GPRA reports and web site (www.hrsa.gov).

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: The program has adopted new long-term goals for the program. Additional steps may be needed to capture the program's impact on the national problem or on targeted facilities. An additional goal measuring program efficiency is also being considered. A Small Extent, Large Extent, or Yes will require outcome data. The program's existing performance measures provide relatively limited data on past performance toward meeting its long-term goals. However, with the adoption of new annual and long-term goals, the program will be in a better position to track performance in the future. The program includes health departments in the list of key facilities because of the importance of these entities and the critical need for nursing staff there. Retention is an important indicator of program outcomes. The program will track progress on the third measure using a survey of recipients similar to that used by the National Health Service Corps.

Evidence: The baseline year for these goals is 2001 and in most cases 2002 data are not yet available. The target year for the long-term goals is 2010. Once baseline data are available, the 2010 targets may need to be adjusted.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program's existing performance measures provide relatively limited data on past performance toward meeting its annual goals. However, with the adoption of new annual goals, the program will be in a better position to track performance in the future. A Large Extent will require additional data to indicate progress on the annual measures.

Evidence: Relevant data that are currently available include the number of contracts supported. The program supported 170 contracts in 1998, 202 in 1999, 195 in 2000, 443 in 2001 and 560 in 2002.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The program met the standards for a Yes in Question 4 of Section III due to steps taken to improve the efficiency of Federal administration. The program is implementing a new electronic on-line application and data reporting system and has taken steps to improve the compatibility of the system with software used by program applicants. There is no evidence of improved efficiency per Federal dollar at the actual loan repayment contract level. The program emphasizes that improved efficiencies per Federal investment will be difficult given rising tuition costs. Efficiencies can be improved with increased retention rates after the period of service.

Evidence: In 2001, roughly eight staff reviewed 600 applications and awarded 200 loan repayment contracts. In 2002, roughly 18 staff reviewed 5,900 applications and awarded 560 contracts. The only data currently available in the program's annual performance report has been the number of loan repayment contracts. Due primarily to rising tuition costs, the Federal cost per contract increased from roughly $11 thousand in 1999 and 2000 to roughly $18 thousand in 2000. By tracking data on third year extensions and retention beyond the service contract, the program will be better able to measure changes in efficiency in the future that go beyond the increasing size of recipient loan burdens.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: The program is not involved in the Federal government's Health Common Measures (for information on these measures see www.whitehouse.gov/omb). There are no programs of similar size available for comparison. Another agency program, the Health Professions, has as one of its goals the placement of health professionals. A third agency program, the National Health Service Corps, provides a closer comparison in that it also works to place health care providers in key areas by providing a financial incentive directly to the provider. The Nursing Education Loan Repayment and Scholarship Program has a lower per provider unit cost than the NHSC. However, the unit cost is difficult to compare given the variation in provider type and the program is not yet able to show retention of its clinicians in eligible facilities.

Evidence: Nursing loan debt is on average lower than that of physicians and the program can place more practitioners per Federal dollar than the NHSC. According to the most recent data available, in 2000 the average cost per placement was $77,400 for the Health Professions, $47,900 for the NHSC, and $11,700 for this program. However, the type of professionals supported by the Nursing Education Loan Repayment and Scholarship Program do not compare with the other two programs. According to the National Conference of State Legislatures, most state scholarship and loan repayment programs have not been evaluated, and thus have no evidence of their effectiveness.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: No comprehensive evaluations have been conducted. An evaluation of the program's impact could be useful to help target resources and make other management and budget decisions.

Evidence: Until FY 2001, the program was funded at less than $2.3 million and did not use any of these funds to evaluate the program impact at this level. In addition to a comprehensive evaluation, new data to be collected will include the number of applications received, number of awardees, distribution by state, level of education, ethnicity and gender, awards by facility and the number of recipients who default.