Detailed Information on the
Leadership Capacity Assessment

The Center for Leadership and Capacity Services will assess its current performance measures jointly with OMB to identify the vital measures that demonstrate quality, effectiveness and efficiency of its programs and program outcomes.

CLCS will standardize its course evaluation process, methodology and analysis of results across training centers and programs to improve overall program performance.

Measure: Individual Impact-Perceived Learning

Explanation:The measure reflects self-assessments by participants using a survey before and after the training to evaluate how much they have learned from CLCS training. The intent of the measure is to evaluate more than simply participant satisfaction (Kirkpatrick Level 1). The measure assesses the extent to which students perceive an advance in skills, knowledge, or attitude. Using the Kirkpatrick Evaluation Model described above, this measure evaluates the participant's perception percentage (%) change in learning (knowledge after less the knowledge before divided by knowledge before) on a five point scale.

Measure: Organizational Impact-Perceived Behavioral Change

Explanation:This new measure relies on a survey of participant supervisors to assess the perceived impact of the CLCS training on an organization. An "organization" would be a specific work-unit or element within the agency that is responsible for a specific function that contributes to the whole and collective function of the agency.

Measure: Annual Full Cost Recovery

Explanation:This measure is a part of OPM's Strategic Plan 2006-2010 (E-21). It requires that organization achieve full cost-recovery annually for each revolving fund. Full-cost recovery is expressed as a the amount of retained earnings divided by the total earned revenues. As part of its long-term goal, the CLCS program must meet its annual targets through yearly milestones.

Measure: Number of Hired PMFs

Explanation:The purpose of the Presidential Management Fellows Program is to attract to the Federal government talented individuals with advanced degrees who are committed to leading and managing public policies and programs. The rate of hire of these individuals within the Federal government is therefore a critical measure of the degree to which the PMF program is impacting leadership talent in Federal Government and fulfilling the need. This measure, tracks the number of PMF finalists hired in a given year. Agencies may hire until March of the following year, so the final FY 2006 result may increase. Provided the organization has the budget, if the PMF Program is effectively providing needed talent to the organization recruiters, they will express their satisfaction by hiring the PMF candidates. There were approximately 550 PMF finalists in 2006.

Measure: Government-wide Reach

Explanation:This measure is calculated as follows: (Total Federal managers receiving CLCS training) / (Number of Federal managers). CLCS will count only those with a "supervisor or manager" designation (CPDF designation of "2"). This attempts to measure the CLCS market share for Federal training, and as such gives an indication as to the overall impact of CLCS. The wider the reach, the more impact on Federal Government CLCS is having, and the better it is achieving its purpose.

Measure: Percent of New Customers - Individual

Explanation:This metric tracks the yearly percent of total Open Enrollment customers that are new to CLCS, or returning after an extended absence. This measure is calculated by taking the total number of Open Enrollment course participants and subtracting the number of repeat participants (see below) and dividing the difference by the total number of unique participants for the period. CLCS strives to increase the number of Federal managers who attend its courses for the first time, or have returned after an absence. Each new registered customer is an opportunity for CLCS to provide training and assist agencies in meeting their leadership, management and succession planning needs.

Measure: Percent of Repeat Customers - Individual

Explanation:Repeat customers are defined as individuals registered for courses who have taken a CLCS course within the past 35 months. This measure tracks the percent of all individual customers who are repeat customers. It is calculated as the total number of repeat customers divided by the number of potential repeat customers over the past 2 years (prior to the current year). Repeat customers are an indication that CLCS has delivered effective training at a good value.

Measure: Overall Customer Satisfaction -- Individual

Explanation:Students complete course evaluations at the end of training to rate their overall learning experience. This metric would be the overall score for the evaluation based on a 5-point scale. It will include food and lodging, customer service, logistics, course content, instructor, before and after knowledge, etc. While there is no strong correlation that satisfied course participants increase their competencies, there is empirical evidence showing that dissatisfied participants do not learn as much. Higher customer satisfaction will also drive increased repeat customers.

Measure: Percent of Agencies Served within the Last Twelve Months

Explanation:This measure is calculated as follows: (Total organizations receiving custom CLCS training) / (total number of organizations). This measure tracks the percent of organizations that used custom CLCS training during the previous 12 month period. According to CLCS data, there are approximately 650 applicable government organizations. The increasing percentage is an indication of market penetration among Federal agencies, and as such gives an indication as to the overall impact of CLCS. The wider the reach, the more impact on Federal Government CLCS is having, and the better it is achieving its purpose.

Measure: Percent of New Customers - Agencies

Explanation:This measure is calculated by taking the number of new (non-repeat, see below) custom client organizations and dividing by the total number of custom clients for the period. Each new organization partnership provides CLCS the opportunity to impart public service values, decrease competency gaps and foster an effective civilian workforce. This metric will track the yearly increase in new organization custom course clients.

Measure: Percent of Repeat Customers - Agencies

Explanation:Repeat customers are defined as organizations receiving custom courses using a new (unique) funding contract (the measure excludes agencies registered for a series of courses using the same funding contract) who have also been customers within the 2 years prior to the current year. This measure tracks the percent of repeat customers in relation to the number of potential repeat customers during a rolling 35-month period that return to CLCS to receive additional custom training programs. Repeat agencies are an indication that CLCS has delivered effective training and a good value.

Measure: Overall Customer Satisfaction -- Agency

Explanation:CLCS has developed a CSS to rate nine core dimensions of service quality related to the training experience. The survey will evaluate customer agencies' perception of the individual's learning experience and the impact of CLCS training on their agency on a 5-point scale.

Measure: Five Year Retention Rate of PMFs

Explanation:This tracks retention rates of PMF's after 5 years of employment. The Program seeks to attract and recruit committed individuals and enhance that commitment through training/ developmental opportunities and real-life immersion in the culture of government. Tracking the 5-year retention rate of PMF hires and comparing that to data for a similar pool will provide the program with valuable insights as to how well it is performing.

Measure: PMF Customer Satisfaction

Explanation:In order to best meet the needs of its customers, CLCS must be able to satisfy their needs. CLCS surveys customers using an instrument with a 5-point scale. A positive customer satisfaction score is an indicator that CLCS is meeting their needs efficiently and effectively. If Customers are more satisfied then this may positively influence the number of hires converted into permanent positions. *Class year, not fiscal year.

Measure: Earned Revenue (in millions)

Explanation:This is the amount of earned revenue earned by the program. If customers are satisfied and CLCS is fulfilling a need in the marketplace, it should be able to sustain and or grow its revenue. This measure is tracked by business line.

Measure: Direct Costs

Explanation:This measure, expressed as a percent, measures the amount of direct costs (defined as all expenditures directly incurred by the preparation of the program for delivery to the customer, including labor) for a product divided by the revenue received for that product. The lower the percent of direct costs, the more the program can contribute to retained earnings.

Measure: Indirect Costs

Explanation:This measure, expressed as a percent, measures the amount of indirect costs (defined as the costs of all activities and non-capital investments charges not adding direct value to the products for which customers pay, such as OPM Overhead, HRPS Overhead, and CLCS Front Office Overhead) accrued in the period divided by the total revenue for that period. Excessive indirect costs can reduce the program's retained earnings and/or lead to price increases to customers. To reach its long term target level of retained earnings, the program must meet its annual targets which represent milestones in reaching the long-term target.

Is the program purpose clear?

Explanation: The purpose of the Center for Leadership Capacity Services (CLCS) is to offer to Federal executives, managers, and supervisors high-quality training where they can (i) acquire knowledge and tools required to meet the leadership, management development, and succession planning needs of their organizations and (ii) develop or renew their commitment to public service values. CLCS programs provide an opportunity for networking and knowledge sharing on leadership development topics and issues for attendees. CLCS offers its courses and services on a fee-for-service basis. The core CLCS programs are based on OPM's Executive Core Qualifications (ECQs) and Leadership Competencies which are designed to define the competencies and characteristics needed to build a federal corporate culture that drives for results, serves customers, and builds successful teams and coalitions within and outside the organization. ECQs are required for entry to the Senior Executive Service. They are used to guide Federal leadership development efforts and are often used by departments and agencies as in the selection, performance management, and leadership development for management and executive positions. CLCS programs and services are divided into three tracks: Open Enrollment; Custom-designed; and Leadership Succession Planning. Each of these programs and services aim to support the leadership, management development and succession planning needs of Federal agencies. The Open Enrollment programs include interagency residential programs that last one to two weeks in duration, at one of the three training centers: the Federal Executive Institute in Charlottesville, Virginia; the Eastern Management Development Center, in Shepherdstown, West Virginia; the Western Management Development Center in Aurora, Colorado. CLCS will soon conduct open enrollment workshops at different cities throughout the country. CLCS's Custom Designed Leadership Programs are tailored to the needs of a specific agency. Program Directors at CLCS consult with agency staff to understand the unique issues, and then create targeted leadership programs. Working with agency staff, CLCS will design a program series and individual seminars for large groups, small teams or through one-on-one forums. Custom Designed Leadership Programs are delivered at CLCS sites, at the agency, or at an alternate site. Leadership Succession Planning helps to build an agency's leadership capacity by focusing on recruitment and development. CLCS Leadership Succession Planning programs include the Presidential Management Fellows Program (PMFP) that recruits men and women with graduate degrees from a wide variety of academic disciplines who have an interest in, and commitment to public service. The Senior Executive Service (SES) Federal Candidate Development Program (Fed CDP) recruits candidates from government agencies, the private sector, and academia into a Federal Government position with the goal of developing the executive leadership of each candidate.

Evidence: Authorities to provide training and development programs to Federal employees by OPM on a fee-for-service basis are listed below: (1) Title 5 of the United States Code (U.S.C.) chapter 41 gives agencies authority to plan, develop, establish, implement, evaluate, and fund training and development programs designed to improve the quality and performance of the workforce. (2) Executive Order (EO) 11348 (as amended by EO 12107) requires OPM to "provide leadership and guidance to ensure that the policy. . .is carried out." Further the EO gives OPM authority to "establish training facilities and services as needed." (3) Title 5 United States Code (U.S.C.) sections 1301-1307 provide special authorities for OPM (including the establishment of a revolving fund). Section 1304(e) provides OPM with the authority to sign reimbursable agreements with other agencies to conduct training and provide related personnel management services. Section 1304, along with title 31 U.S.C. Section 6505, authorizes OPM to perform services for state and local governments. (4) EO 12008 established the Presidential Management Intern Program (PMI) in 1977. This program was amended by EO 12645, in 1988 and superseded in November 2003 by EO 13318, which renamed the PMI the Presidential Management Fellows Program (PMFP). The purpose of the PMFP is "to attract to the Federal service outstanding men and women from a variety of academic disciplines and career paths who have a clear interest in, and commitment to, excellence in the leadership and management of public policies and programs (5) 5 U.S.C. 3396 requires OPM to "establish programs for the systematic development of candidates for the Senior Executive Service [SES] and for the continuing development of senior executives." Title 5 CFR (5 CFR 412 - Executive, Management, and Supervisory Development) addresses developing executive, managers, and supervisors.

Does the program address a specific and existing problem, interest, or need?

Explanation: The need to hire, develop and retain leaders to effectively manage is increasing -- especially in the light of the projected wave of Federal retirement during the next ten years. The Federal civil service, like the military and postal service and many large corporations, requires a top quality, federally-run enterprise-wide management and executive development program such as CLCS. Both current employees and new hires require training to prepare them to lead the government of today and tomorrow. The successful delivery of effective leadership-development training is an essential part of the Government's efforts to transform the management of federal human capital as outlined in the President's Management Agenda. Leadership training also supports improving mission accomplishment by agencies, and the successful implementation of new and effective personnel systems, such as those for the Department of Homeland Security and National Security Personnel System. As many corporate or non-profit organizations have their unique organizational cultures, the Federal government has its own culture and values. The CLCS programs are designed to transfer the latest knowledge and reinforces the Government values and practices. CLCS-trained leaders can then demonstrate and instill in their staff -- new workers as well as upcoming leaders -- the desired values and culture. Leadership Succession Planning helps to build an agency's leadership capacity by focusing on recruitment and development. The Presidential Management Fellows Program (PMFP) recruits men and women with graduate degrees from a wide variety of academic disciplines who have an interest in, and commitment to public service. The Senior Executive Service (SES) Federal Candidate Development Program (Fed CDP) recruits candidates from Federal agencies, the private sector, and academia into a Federal Government position with the goal of developing the leadership potential of each candidate.

Evidence: The Government Accountability Office has looked at the expected wave of retirements by the generation of "baby boomers" born in the years following World War II, and provided recommendations. They addressed the issue in Senior Executive Service: Retirement Trends Underscore the Importance of Succession Planning, GAO/GGD-00-113BR (May 12, 2000) and in Senior Executive Service: Enhanced Agency Efforts Needed to Improve Diversity as the Senior Corps Turns Over" GAO-03-34 (January 2003). Several years later, they provided recommendations to mitigate detrimental effects of large scale retirements in HUMAN CAPITAL - Selected Agencies Have Opportunities to Enhance Existing Succession Planning and Management Efforts, GAO-05-585 (June 2005). While predicting the exact timing of the expected retirement wave remained uncertain, developing solution was the focus of several studies by recognized experts in the human capital field. The National Academy of Public Administration and Human Capital Solutions conducted a Symposium in February 2006 on the topic: "Can Government Grow Great Leaders?" They have also reported on "Leadership for Leaders - Senior Executives and Middle Managers", published in August 2003. Another source of Leadership recommendations comes from the American Management Association with Leading into the Future: A Global Study of Leadership 2005-2015, published in 2005.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The federal market for leadership development is very large (there are approximately 1.8 million civilian employees) and requires many service providers. There are numerous private and public sector organizations which provide leadership training. However, the CLCS program has clear boundaries and mandates set by law and Executive Order which ensure its contribution to the further development of leadership in all ranks of the Federal government. CLCS leadership development services help shape the federal workforce by training its leaders. Those who are trained pass along what they have learned to their staff and use it to improve their agency's effectiveness. The efficiency and effectiveness of Government depends on consistency among agency leaders in pursuing their mission, and living the values that creates a culture of public service and continuous improvement. CLCS programs provide uniformity and set standards for leadership training over the long term, while providing an infrastructure that reduces the overall cost to Government for this training. CLCS is unique as the only government-wide federal provider for leadership development, and as such provides the most federal leadership development experience and expertise. It provides the Federal Government with these unique benefits: ?? Providing the best value in the market (lower prices, high quality) and savings for the Federal Government totaling in the millions of dollars. Unlike the private sector, CLCS prices only according to a full cost-recovery model; ?? Protecting the best interest of the Federal Government and the public (rather than shareholders), as well as providing the market with at least one provider that is not "at risk" of bankruptcy, market withdrawal, consolidation leading to a monopoly, etc.; ?? Being the only government wide federal entity to train government leaders on important policies and programs; ?? Ensuring access to the latest Federal Human Resources policies and Human Capital leadership guidance through its close relationship OPM's Strategic Human Resources Policy Division (SHRP) and Human Capital Leadership and Merit System Accountability Division (HCLMSA); ?? Maintaining unmatched subject matter expertise in public leadership development with its merit-based, legal framework. This includes training on the unique values and culture of the public sector, built upon CLCS's long history of federal HR work, institutional knowledge, relationships with leading experts and firsthand knowledge as Federal employees; and ?? Setting standards for best-in-class leadership training programs that include merit system principles, promote a common language (competencies), and allow Government-wide standardization. Standardization is provided through valid, standardized leadership assessments and certificate programs designed around government competency and core qualification standards.

Evidence: Evidence of CLCS's contribution to development of Federal leadership ranks include articles and studies by private sector firms, GAO and the Washington Post ?? "CLCS and Other Federal Manager-Executive Development Programs - Assessment of Competitive Landscape" by National Analysts, Inc. (October 2005) ?? Senior Executive Service Candidate Development Program (SESCDP) www.doiu.nbc.gov/sescdp.html ?? "Leading into the Future "A Global Study of Leadership: 2005-2015 by the American Management Association www.amanet.org/research/pdfs/HRILeadershipSurv05.pdf ?? "Improved Leadership Needed for Better Performance" - U.S. General Accounting Office, HRD 91-88 (August 1991) http://archive.gao.gov/d19t9/144590.pdf ?? "MANAGING FOR RESULTS - Federal Managers' Views Show Need for Ensuring Top Leadership Skills" - U.S. General Accounting Office, GAO 01-127 (October 2000) http://www.gao.gov/new.items/d01127.pdf ?? "Defense Department Reconsiders Senior-Level Career Development" - By Stephen Barr, Washington Post Wednesday, March 15, 2006 "a world-class leadership structure." www.washingtonpost.com/wp-dyn/content/article/2006/03/14/AR2006031401763_pf.html

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The current design is free from major flaws that would limit the program's effectiveness or efficiency. Boundaries set for the program are appropriate, providing significant savings to agencies, while allowing the flexibility to respond to agencies' special requirements and requests. No significant programmatic flaws have been identified either by internal or external sources. In 2003, the OPM Director realigned all reimbursable programs. This set the framework for increased integration of delivery of strategic human capital services to departments and agencies, reductions in redundancy of OPM offerings and better leverage of overhead support and costs. Reimbursable products and services were consolidated into one division: Human Resources Products and Services (HRPS). Two programs within HRPS provide the bulk of reimbursable products and services geared towards helping agencies improve their human capital management, CLCS and the Center for Talent Services (CTS). CLCS has a strategic partnership with CTS on a Government-wide leadership assessment project (CTS provides assessment design expertise and support for recruitment, assessment, and selection of candidates). The revolving fund authority provides an appropriate mechanism for delivering products and services on a reimbursable, cost-recovery basis. CLCS is able to provide agencies unparalleled federal leadership development expertise and best practices that help agencies build high-performance organizations as well as OPM support of Government-wide leadership programs. Examples include the Presidential Management Fellows Program, the Federal Candidate Development Program, Project Management certification, PMA-related seminars and partnerships with prestigious universities. These products and services are delivered by a combination of highly qualified and experienced Government personnel and private-sector adjunct faculty who offer customized training solutions. The current design allows for private sector support with a limited core of federal experts to provide program management, and training development and delivery. This design assures price and performance competition, scalable capacity to support increasing or decreasing demand, and provides an experienced cadre of federal employees to ensure leadership training is focused on the unique Federal environment, while letting CLCS run leadership training delivery programs that meet statutory and customer demands. While the program is free of design flaws, it could operate more efficiently and effectively if it operated in a more entrepreneurial environment dedicated to providing reimbursable services. CLCS's efficiency is limited by its inability to exercise maximum delegated authority to streamline contracting and financial management processes and systems. It also could have greater flexibility in establishing mutually beneficial partnerships with Non-governmental Organizations (NGO's) and private sector organizations (Cooperative Research and Development Agreements for example).

Evidence: Title 5 USC Sections 1301-1307 provide special authorities for OPM (including the establishment of a revolving fund). Section 1304(e) provides OPM with the authority to sign reimbursable agreements with other agencies to conduct training and provide related personnel management services. That section, along with title 31 USC 6505, authorizes OPM to perform services to state and local governments.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The CLCS program is designed to reach the highest possible number of federal leaders and managers. For the SES, there is the Federal Executive Institute (FEI). For managers at GS levels 9-15, there are the Management Development Centers. For new potential federal leaders, there is the Presidential Management Fellows program and the Candidate Development program. This program design offers something for every type of federal leader and manager. For agencies and organizations, CLCS develops custom leadership and manager development programs. CLCS targets federal leaders and managers, which are the appropriate beneficiaries according to its founding legislation. All of these programs are reimbursable, meaning that the prices for CLCS products and services reflect the real cost of operation. Since CLCS conducts no activities outside of its purpose, this ensures that there are no unintended beneficiaries and resources are being used directly and effectively to meet the program's purpose.

Evidence: The CLCS Open Enrollment and Custom programs use dedicated Program Directors (PDs) who are responsible for developing programs and seminars, financial analyses and evaluation methodologies to support learning for an effective Federal leadership, management and executive cadre. PDs design curriculum, recruit adjunct faculty, teach and evaluate the effectiveness of CLCS programs. They also conduct research relative to advances that could potentially affect executive, management, leadership development and public policy programs. They act as speakers or resource persons on leadership and public policy education in the Government and in meetings with officials from customer organizations. The Leadership Succession Planning Program encompasses both the Presidential Management Fellows (PMFP) Program and the Federal Candidate Development Program (FedCDP). Program Managers are dedicated to program implementation, program management, recruiting and administrative activities that support the leadership succession planning programs. The PMF Program staff implements program-wide policy and guidance regarding the recruitment, examination and employment of PMFs. The FedCDP Program Manager manages all aspects related to the design, delivery and evaluation of training for FedCDP candidates, in addition to newly appointed SES members and Schedule C appointees. Administrative staff facilitates cross-functional activities to support all CLCS lines of business. Administrative staff includes Business Operations Managers, IT Manager, Business Development Manager and HR Manager. Each of the CLCS lines of business has a specific target audience that can benefit from leadership training. Open Enrollment courses target Federal executives, managers and supervisors who require training in executive leadership development, individual assessment and development, management and supervisory skills and organizational leadership. Custom Services target agencies and organizations that desire to address a one-time need or want to establish a strategic partnership to address organizational challenges. The target audience for Succession Planning programs includes recent university graduates who have an interest in leadership and management of public policies and programs through the PMF Program and Federal agencies who seek individuals to fill their managerial and executive ranks through the program. Senior managers are targeted though FedCDP, and SES and Schedule C briefing. The CLCS catalog provides detailed course descriptions, in addition to summaries about relevant competencies emphasized, the target audience, benefits of the course, and college credit, if applicable. The website (http://www.opm.gov/Products_and_Services) has been updated to highlight new and interesting courses. In addition, the marketing brochure is distributed to students and at federal conferences. Sales and marketing efforts use existing and new data to target opportunities and address training needs of the Federal Government and increase agency penetration. In addition, CLCS works with every major Federal agency and many smaller agencies to promote its courses and identify potential participants. CLCS has also strengthened its relationship with OPM's Chief Human Capital Officers Council, so that they can act as a resource to refer agencies needing assistance to improve their performance and results. CLCS operates as a Revolving Fund. As such, all services must be funded prior to performance. The organization provides its services primarily to federal agencies. Since agencies must pay for services, CLCS's resources are effectively targeted. There are no unintended beneficiaries, with the exception of a limited number of internal OPM projects for which CLCS receives less than full reimbursement.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: CLCS's four long-term measures reflect its purpose to provide Government leaders with the knowledge and tools to help organizations meet their leadership, management development, and succession planning needs. These long-term measures are key indicators of the program's performance in executing the CLCS strategy and OPM's Strategic and Operational Plan - 2006 - 2010. The long term measures are: ?? 1.0 Individual Impact-Perceived Learning. CLCS measures the impact of its training on developing individual competencies and skills. Surveys are completed by class participants after each class, to provide an opportunity for self-evaluation of learning transfer and to allow students to rate their knowledge before and after attending the training. Actual testing on class materials is not used; this measure is based on the individual student's perception of the increase in knowledge precipitated by training attendance. CLCS measures the percent change in before and after knowledge as rated in the survey by the participant to gauge the impact of its training on individual participants. ?? 2.0 Organizational Impact-Perceived Behavioral Change. A new measure has been developed to track supervisory assessment of the impact of training on participants from their respective organizations. CLCS will measure the perceived transfer of knowledge, skills and attitudes from the training context to the workplace using a survey of supervisors. CLCS has piloted this measure at one Federal agency and is in the process of standardizing this measure for all applicable custom engagements. ?? 3.0 Full-cost Recovery. This measure is a part of OPM's Strategic Plan for 2006-2010. Full-cost recovery is expressed as the amount of earned revenues divided by the total operating expenses. If CLCS is meeting the needs of government and operating efficiently, it will achieve full cost recovery. It has a long term retained earnings target. To achieve the target, the CLCS program must meet its annual milestones. ?? 4.0 Number of Hired PMFs. The purpose of the Presidential Management Fellows Program is to attract to the Federal government talented individuals with advanced degrees. This measure tracks the number of screened candidates (PMF finalists) hired by agencies in a given fiscal year. The rate of hire of these individuals within the Federal government is therefore a critical measure of the degree to which the PMF program is fulfilling the leadership talent need. If the PMF Program is effectively providing needed talent to the agency recruiters, they will express their satisfaction by hiring the PMF candidates.

Evidence: The measures align with the key strategic outcomes CLCS has mapped: Improved Government Leadership; Promulgate Public Service Agenda, Values and Culture; effectively execute strategy; and Best Learning Value. 1.0 Individual Impact-Perceived Learning- CLCS measures training effectiveness using the Kirkpatrick Evaluation Model. The Kilpatrick Model's four levels of evaluation are: reactions, learning, transfer and results. According to this model, evaluation should always begin with reactions (level 1), and move sequentially through learning (level 2), transfer (level 3), and results (level 4). Information gathered from each prior level serves as a base for the next level's evaluation. The measure is associated with level 2 in Kilpatrick's model. The measure reflects self-assessments by participants using a survey before and after the training to evaluate how much they have learned from CLCS training. The intent of the measure is to evaluate more than simply participant satisfaction (Kirkpatrick Level 1). The measure assesses the extent to which students perceive an advance in skills, knowledge, or attitude. Using the Kirkpatrick Evaluation Model described above, this measure evaluates the participant's perception percentage (%) change in learning (knowledge after less the knowledge before divided by knowledge before) on a five point scale. 2.0 Organizational Impact-Perceived Behavioral Change . This new measure relies on a survey of participant supervisors to assess the perceived impact on an organization. An "organization" would be a specific work-unit or element within the agency that is responsible for a specific function that contributes to the whole and collective function of the agency. For instance, organizations within the Department of Agriculture include the National Park Service and the Office of Surface Mining. CLCS identifies organizations using the Central Personnel Data File, OPM's central personnel database, with some additional codes for those agencies that have only one code in the CPDF. The measure will be used with custom-designed programs to evaluate the change in behavior (Kirkpatrick Level 3, see above). Current plans call for a pre- and post-program survey of supervisors. One Federal agency has piloted this measure and CLS will standardize this measure for all applicable custom courses during FY 2007. Plans are to develop data in FY 2007, and adjust targets appropriately for FY 2008 and beyond. 3.0 Full Cost Recovery Full Cost Recovery is an OPM Strategic Objective (Strategic and Operational Plan 2006-2010, E-21). At present, CLCS does not fully recover its costs. CLCS is now setting a long term target level for its retained earnings and ambitious annual targets designed to achieve the long term target level of retained earnings. Efforts to achieve full-cost recovery include the recent increase in the charge to agencies for participation in the PMF Program, and tracking of expenses and revenue by course [Expenses & Revenue by Course]. 4.0 Number of Hired PMFs with the OPM Director's approval, the program can now actively recruit as many Fellows as the program can manage, yet still maintain the integrity of the program. Each year, the Director will make a determination on the number of PMF finalists selected from an applicant pool. The number of PMF hires indicates success the program is having at attracting candidates which fulfill the needs of federal agencies.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: CLCS has set challenging targets for its long term measures. For the Individual Impact-Perceived Learning measure, its course participants in FY 2006 indicated that they have received a 40% increase in subject matter knowledge, consistent with prior year's data. Direct benchmark data is difficult to obtain, but one private sector firm has provided data to CLCS indicating that 44% of 1,422 leadership trainees indicated they experienced significant on-the-job performance improvements. CLCS has targeted a 1 percentage point increase in average participant knowledge transfer for FY 2007 to 41% and a further 2 percentage point increase in FY2008 (a target of 43%). This is ambitious because CLCS has never attained a 43% knowledge transfer average for its courses, and achieving this target requires constant course improvements. For the Organizational Impact-Perceived Behavioral Change measure, CLCS piloted the measure in one of its clients in FY2006. The result was a 4.54 on a 5 point scale. The result indicates an already high level of transfer of knowledge, skills and attitudes from the training context to the workplace. It has set a target of 4.55 for FY 2007 and FY 2008. Of the four long-term measures, the most challenging may be for CLCS to achieve its full-cost recovery targets. In FY 2004, CLCS became a 100% fee-for-service organization and was grouped with the PMF program which previously received a significant level of appropriated funding. In FY04 and FY05, it did not recover its costs. This improved marginally in FY06, but the program still ran at a deficit. CLCS's target for FY07 is to recover all of its costs, which is very ambitious given its prior year results. It requires a combination of increasing revenues and controlling costs. In FY08, the target is to add to its retained earnings by a significant amount. The long-term measure for the PMF Program will track the number of PMF hires who complete their fellowship and convert into permanent Federal positions, to provide a pool of talent for the Government's future leadership requirements. In FY 2006, 355 finalists were converted to Federal employees and the goal is to increase the number of PMF finalists gaining a permanent employment status each year. The target for FY2007 is 400 hires, a 13% increase over FY2006 levels. The target for FY2008 is 413 hires, or a 16% increase over FY2006 levels.

Evidence: CLCS long-term measures are included in a variety of OPM's official documents including OPM's Strategic Plan, and OPM's FY2008 CBJ submission. Results are tracked on CLCS's Mission Measures Matrix (M3) tracking tool ands are reported at least quarterly. Results for the Organizational Impact-Perceived Behavioral Change measure are currently derived from a variety of survey instruments, but a standardized survey will be used in FY 2007 to gather baseline data for setting targets in FY 2008. The long-term measure is new, and CLCS will gather additional baseline data during FY 2007 to establish targets for FY 2008. This current baseline is derived from a pilot program between FEI and a partner agency to assess the impact of training on organizational performance. Beginning in FY 2004, CLCS began tracking expenses and revenue by sub-program, leading to the conclusion that an increase in prices for the PMF Program was an essential step to eliminate the FY 2004 shortfall in CLCS cost recovery. This price increase was approved effective during FY2006, and will contribute significantly, along with cost reduction measures, to achieving full cost recovery in FY 2007. Emphasizing the strategic importance of the PMF Program, CLCS has recently established targets for the number of PMF participant conversions into permanent Federal positions.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: A total of thirteen annual measures provide a comprehensive measure of cost-effectiveness and agency and organizational impact. The Individual Impact-Perceived Learning (1.0) is tracked through four annual measures. ?? 1.1 Government-wide Reach. This measures the CLCS market share for Federal training. This measure is calculated by dividing the total of Federal managers receiving CLCS training during a 12-month period by the number of Federal managers in a given year. ?? 1.1a Percent of new customers - Individual. CLCS strives to increase the number of Federal managers who attend its courses for the first time, or have returned after an absence. This measure is calculated by comparing the number of new open enrollment course participants to the participant lists from the previous 35 months. ?? 1.1b Percent of repeat customers - Individual. The measure is calculated as the total number of repeat customers, defined as individuals who have registered for a class within the last 35 months, divided the number of potential repeat customers over the past 2 years prior to the current year. ?? 1.2 Overall customer satisfaction - Individual. Students complete course evaluations at the end of training to rate their overall learning experience. This metric will be the overall score for the evaluation. It includes quality, customer service, logistics, course content, instructor, before and after knowledge, etc. The long-term Organizational Impact-Perceived Behavioral Change measure (2.0) is tracked through four annual measures. ?? 2.1 Percent of organizations served within the last 12 months. This measure tracks the percent of organizations that used custom CLCS training during the previous 12 month period. The measure is calculated by dividing the total of organizations receiving custom CLCS training by the total number of potential client organizations. ?? 2.1a Number of new customers - Organization. This measure is calculated by taking the number of new (non-repeat) custom client organizations and dividing by the total number of custom clients for the period. This metric will track the yearly increase in new organization custom course clients. ?? 2.1b Percent repeat customers - Organization. This measure tracks the percent of repeat customers in relation to the number of potential repeat customers during a rolling 35-month period that return to CLCS to receive additional custom training programs. Repeat agencies are an indication that CLCS has delivered effective training and a good value. ?? 2.2 Overall customer satisfaction - Organization. CLCS is replacing program-specific surveys with a standardized Customer Satisfaction Survey in FY 2007. This will be sent to organizational customers, as opposed to class participants. The survey is able to be benchmarked against the American Customer Satisfaction Index (ACSI). The first survey was completed in FY2006 and serves as CLCS's benchmark for FY07 and FY08 targets. The long-term Full-cost Recovery measure (3.0) is tracked through three efficiency measures: ?? 3.1 Earned revenue. This is the amount of revenue earned by the program. ?? 3.2 Direct costs. This measure, expressed as a percent, measures the amount of direct costs for a product divided by the revenue received for that product. ?? 3.3 Indirect costs. This measure, expressed as a percent, divides the amount of indirect costs by the total revenue for that period. The PMF Hires Converted to Permanent measure (4.0) will be tracked through two annual measures: ?? 4.1 5-year PMF retention rate. Tracking the 5-year retention rate of PMF hires and comparing that to data for a similar pool will provide the program with valuable insights as to how well it is performing. ?? 4.2 PMF Customer Satisfaction. A positive customer satisfaction score is an indicator that CLCS is meeting their needs. If customer agencies are more satisfied, then this may positively influence the number of PMF hires converted into permanent positions.

Evidence: For all the long-term measures, an "agency" is defined as a department or other unit of government created by legislation to administer the law in a particular area of public concern. Examples of agencies include the Office of Personnel Management, the Department of Agriculture, and the Department of the Interior. An "organization" would be a specific work-unit or element within the agency that is responsible for a specific function that contributes to the whole and collective function of the agency. For instance, an organization within the Office of Personnel Management would be the Human Resources Products and Services Division, or the Office of Congressional Relations. For 1.1, Managers are defined as employees in positions with a "supervisory" or "management" designation in the Central Personnel Data File (CPDF). For 1.1a, the measure is calculated by comparing the number of new open enrollment course participants to the participant lists from the previous 35 months. If no match is found, the new participant is counted as a new customer and the sum is divided by the total number of participants that year. Two years prior to the current year is deemed appropriate because leaders and managers should receive training at least once every two years and course curricula can change significantly over that time. For 2.1b, repeat customers are defined as organizations receiving custom courses using a new (unique) funding contract (the measure excludes agencies registered for a series of courses using the same funding contract) who have also been customers within the 2 years prior to the current year. For 3.2, direct costs are defined as costs that can be specifically identified with the production and delivery of the output including: Salaries and other benefits for employees who work directly on the output; Materials and supplies used in the work; Various costs associated with office space, equipment, facilities, and utilities that are used exclusively to produce the output; costs of goods or services received from other segments or entities that are used to produce the output. For 3.3, indirect costs are defined as the costs of resources that are jointly or commonly used to produce two or more types of outputs but are not specifically identifiable with any of the outputs. Examples of indirect costs include the cost of general administrative services, OPM common services, general research and technical support, security, rent, and operating and maintenance costs for buildings, equipment, and utilities. Traditionally, CLCS has reported performance based on training centers and programs, i.e., the Federal Executive Institute (FEI), Western and Eastern Management Development Centers (WMDC and EMDC), the President's Management Fellowship (PMF) Program and the SES Federal Candidate Development Program (FedCDP). Effective with Fiscal year 2007, the program-specific customer satisfaction surveys will be replaced with a standardized Customer Satisfaction Survey. In addition, the survey results will be presented by business lines to more accurately reflect the organization's performance, namely: 1. Open Enrollment; 1a. Executive Leadership; 1b. Management Development; 2. Custom-designed Programs; 3. Leadership Succession Planning; 2a. Presidential Management Fellows Program; 2b. Federal Candidate Development Program. Furthermore, CLCS is now implementing a Learning Management System to track the full range of enrollment data and generate reports of program information which had previously been tracked in its training server application, including incurred costs, earned revenue and other metrics for performance measures, along business lines. CLCS is replacing program-specific surveys with a standardized Customer Satisfaction Survey in FY 2007. Both the FY 2007 CLCS operating budget and OPM 2008 CBJ include PART measures.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: ?? 1.1 Government-wide Reach. CLCS is targeting a sharp reversal of present downward trend in managers trained in FY2007, and expects to train 3.3% of all government managers, an increase of 26% over FY06 reach. In FY 2008 it is targeting 3.5% reach, a 34% increase over FY06 reach. ?? 1.1a Percent of new customers - Individual. In FY 2007 and FY 2008, CLCS is targeting 81% of its customers to be new customers. It is more in line with FY2004 and FY 2005 results of 83% and 82% respectively. ?? 1.1b Percent of repeat customers - Individual. Of its pool of potential repeat customers [there were 646 potential customers in 2006], CLCS trained 8.6% in FY06. This was an increase in the percentage of repeat customers from 6.9% in FY 2005. CLCS is following a strategy of increasing the number of repeat customers, and has set targets of 9.9% in FY07 and 10.5% in FY08. This represents increases of 14% and 21% over FY06 results respectively. ?? 1.2 Overall customer satisfaction - Individual. For FY 2007 and FY 2008, CLCS is targeting 4.70 and 4.72. If achieved, this would be the highest level of customer satisfaction in recent CLCS history. ?? 2.1 Percent of organizations served within the last 12 months. CLCS is targeting an ambitious increase in penetration for this line of business, rising to 16.6% in FY 2007 and 17.3% in FY 2008. ?? 2.1a Number of new customers - Organization. CLCS is targeting a decrease in number of new customers to 69% in FY2007 and 68% in FY2008. ?? 2.1b Percent repeat customers - Organization. CLCS has set targets to increase the percent of repeat customers to 16.7% in FY2007 and 17.5% in FY2008. This would put the percent of repeat customers at its highest levels. ?? 2.2 Overall customer satisfaction - Organization. CLCS is targeting an increase in customer satisfaction to 4.60 in FY 2008. Its targets are well above the ACSI benchmark of 71.3. The Full-cost Recovery measure (3.0) is tracked through three efficiency measure targets: ?? 3.1 Earned revenue. For FY 2007, CLCS is targeting a sharp turn-around, with revenue growth of 8% to $47.1 million. In FY08, it is targeting a further 7.5% increase to $50.7 million. ?? 3.2 Direct costs. Direct costs as a percentage of revenue was 71.7% in FY06, almost a full percentage point less than the 72.6% in FY 2005. In order to achieve its full cost recovery targets, it is targeting a substantial decrease in direct costs as a percent of revenues to 66.0% in FY 2007 and 66.3% in FY 2008. ?? 3.3 Indirect costs. CLCS's indirect costs were 31.5% of revenues in FY 2006, essentially the same as the 31.4% in FY 2005. For FY 2007, CLCS is targeting 33.1%, which is reflective of spending in sales and marketing as well as in facilities and technology designed to increase revenues, maintain customer satisfaction, and improve program performance. The target for FY 2008 is 30.8%, which would be the lowest level since the 2003 reorganization. The PMF Hires Converted to Permanent measure (4.0) will be tracked through two annual measure targets: ?? 4.1 5-year PMF retention rate. The five year retention rate of PMF hires was 69% in FY 2006. CLCS is targeting a 1 percentage point increase in the retention rate for FY 2007 to 70% and another 1 percentage point increase in FY 2008 to 71%. ?? 4.2 PMF Customer Satisfaction. Customer Satisfaction among PMF customer agencies was 4.08 on a 5 point scale. This is above the ACSI benchmark and CLCS is targeting an increase to 4.1 in FY07 and to 4.15 in FY 2008.

Evidence: CLCS used FY 2004 and FY2005 data for its baseline in establishing its targets for FY 2007 and FY 2008 for all of its annual measures. For 1.1, CLCS's reach into the number of potential federal managers it has trained has been declining since FY2004, and reached a low of 2.6% in FY2006. For 1.1a, Similar to its government-wide reach results, the percentage of new customers has been declining since FY2004, with FY 2006 results being 78%. This number combined with a decline in total customers is one which CLCS is targeting to improve. For 1.2, Customer satisfaction is very high among CLCS course participants. On a 5 point scale, for FY 04, 05 and 06 CLCS scored 4.69, 4.61 and 4.67 respectively. For comparison, the American Consumer Satisfaction Index (ACSI) is 71.3 (aggregate on a 100 point scale). Private sector data indicates that 82% are very satisfied with the overall program (Knowledge Advisors, n=17,059). For 2.1, the percentage of the 646 potential government organizations CLCS has served through its custom leadership training business line has been consistently around 15% each year since FY2004. For 2.1a, the percent of CLCS custom business line customers who are new customers has consistently been around 70% since FY2004. For 2.1b, The percent of CLCS custom business line customers who are repeat customers has dropped from 16.6% in FY04 to 15% in FY06. For 2.2, CLCS sent its first standardized Customer Satisfaction Survey organizational customers in FY 2006. The survey was designed and conducted by OPM's Center for Talent Services and will be repeated each year. It is able to be benchmarked against the ACSI. The FY06 survey yielded a very high result of 4.55 on a 5 point scale. For 3.1, CLCS earned revenue fell by almost 2% from FY 2005 to FY 2006. For 3.2, the largest reduction in direct costs is expected in the Succession Planning line of business, where a change in the PMF assessment methodology is expected to yield large savings. The Open Enrollment and Custom business lines are also targeting ambitious direct cost reductions. The private sector benchmark in the business training services industry for direct costs as a percent of revenues is 64.9% (Bizminer, June 2006). Regarding the increase in indirect costs (3.3) for FY2007, WMDC is going to upgrade four of its classrooms and breakout rooms with more current IT equipment and furnishings. The Center will also be investing in its efforts to re-negotiate/re-compete the current Food/Lodging contract. EMDC indirect costs are projected to increase in FY 2007 due to an expansion project at the Center to add square feet of office space to the facility. The additional space will support the National Sales Team and provide up to four additional breakout rooms.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: CLCS' partnerships extend the quality and scope of its ability to achieve the strategic objectives in its Strategy Map, and four long-term performance goals: Individual Impact-Perceived Learning (1.0); Organizational Impact-Perceived Behavioral Change (2.0); Full Cost Recovery (3.0); and Number of hired PMFs (4.0). Within OPM CLCS partners with the Human Capital Leadership and Merit System Accountability Division (HCLMSA), Strategic Human Resources Policy Division (SHRP), and Center for Talent Services (CTS). CLCS often works with these groups to keep abreast of the latest HR policy developments so that it can provide relevant HR products/services and build high-performing organizations and a high-quality workforce. CLCS also informs these partners about projects it is working on with other Federal agencies, so that other parts of OPM are fully aware of Federal agencies' needs and challenges and the ways in which CLCS is assisting agencies in meeting their Human Capital objectives. CLCS and CTS have recently completed an initiative to standardize the customer satisfaction survey that will be used to collect data for annual measures: Overall Customer Satisfaction - Individual (1.2) and Overall Customer Satisfaction - Agency (2.2). CTS also entered into a service level agreement with CLCS to implement an on-line application process for the Presidential Management Fellows (PMF) program. CLCS external partnerships include Federal and state agencies, academia, and private-sector organizations that support the OPM mission to "ensure the Federal Government has an effective civilian workforce." Examples of CLCS' external partnerships include: University Partnerships; Executive in Residence (EIR) Program; Pacific Leadership Academy (PLA); Custom Designed Leadership; Programs; and Contractors. Regarding contractors, CLCS partners with some of the leading experts in leadership development training. It shares the results of the course evaluations with every contractor and works together with these partners to make improvements. In its performance-based contracts, CLCS has the ability to affect payment for poor evaluations.

Evidence: University Partnerships -CLCS has worked to make many of its courses be certified by the American Council on Education (ACE) for college or graduate credit. CLCS has also partnered with American University and the University of Colorado at Denver. These partnerships provide enhanced learning opportunities, because CLCS clients can interact with colleagues from other public agencies??state, and local??as well as those from the nonprofit sector. Through the partnership with the Graduate School of Public Affairs (GSPA) at the University of Colorado at Denver, individuals can earn an Executive Masters in Public Administration (MPA). The program offers a combination of courses taken from the Management Development Centers (MDCs) (up to 18 credit hours) with courses completed online. Through the partnership with American University (AU) in Washington, DC, MDC courses can be applied toward university credit. MDC courses can count for up to 12 credit hours toward American University's Master of Public Administration (MPA) or Master of Public Policy (MPP) degree, or toward AU's Key Executive MPA program. EIR Agreements - The Executive In Residence (EIR) program invites executives from Federal agencies to serve as visiting full-time teaching staff at the Federal Executive Institute (FEI). These executives work with a team of FEI permanent and adjunct faculty members to develop the values and competencies that are the foundation of public service, transcending individual professions and missions. EIRs help FEI: create, share, and apply knowledge and skills to address the challenges faced by public sector organizations; develop the values and competencies that are the foundation of public service, transcending individual professions and missions; and offer state-of-the-art learning experiences in world-class learning environments. PLA agreement - The PLA is sponsored by the Honolulu-Pacific Federal Executive Board in partnership with the U.S. Office of Personnel Management, Western Management Development Center (WMDC). CLCS's WMDC worked with (FEB) to open the PLA. The FEB reviews applications from agency nominees for the nine-month competency-based leadership program, which is based on OPM's leadership competencies for SES certification. Custom Designed Leadership Programs - In the Custom designed leadership programs, CLCS Management Development Centers (MDCs) and the Federal Executive Institute (FEI) partner with individual Federal agencies. Partnerships include custom seminars, team development programs, facilitated planning programs, consulting, organizational and individual assessments, executive coaching, succession planning and executive development partnerships, all of which help individual agencies develop their current and future leaders. CLCS's partnership with the Center for Executive Leadership (CEL) at the U. S. Agency for International Development (USAID) provides a good example of how CLCS works with individual agencies to develop their current and future leaders. This is a multi-year leadership development partnership that provides an ongoing, comprehensive skill-building program for USAID. Facility Services Agreement - CLCS has a partnership agreement with two of its facility providers that base payments, in part, upon participant evaluations of the quality of food, lodging and service. CTS Partnerships - The Center for Talent Services (CTS) has provided support to CLCS for a standardized customer satisfaction survey and to automate the Presidential Management Fellows (PMF) program application process through the USA Staffing system.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Academic Evaluations - As discussed in the response at 2.5, CLCS has had many of its courses evaluated and certified by the American Council on Education (ACE) so participants can receive college or graduate credit for their work. These certified courses demonstrate CLCS' ability to produce a catalog of training courses that meets academic standards. Customer Evaluations - The individual and agency feedback on training experiences with CLCS programs and the impact the training has on individual and organization performance, are the key drivers of change and improvement for CLCS offerings and services. CLCS receives evaluations from course participants at the conclusion of every product and service delivery. Cumulative survey results are consistently high at 4.65, 4.60 and 4.60 out of 5 for FY 2004 through FY 2006. A new standardized agency satisfaction survey of CLCS training programs has been implemented. Programs covered are the Federal Career Development Program (FedCDP); Federal Executive Institute (FEI); Management Development Centers (MDCs); and Presidential Management Fellows (PMF). First results were received in September 2006. Overall Customer Satisfaction - Agency (2.2) Program Score FedCDP 3.38 FEI 4.50 MDCs 4.58 PMF - Universities 3.31 Individual survey results and the new 2006 agency results were used to establish FY 2007 targets for annual measures: Overall Customer Satisfaction - Individual (1.2) and Overall Customer Satisfaction - Agency (2.2), with full implementation scheduled for FY 2008. Other evaluations - Neither CLCS, nor predecessor organizations with responsibility for its programs have received regular, high quality evaluations from non-biased, independent organizations. The CLCS organization was created as the result of an overall OPM reorganization evaluation and CLCS will contract in FY 2007 for an independent evaluation of its program operations and results, as well as potential opportunities for improvement.

Evidence: American Council on Education (ACE) Accreditation -Through the ACE accreditation process employers can gain national recognition for their training. The American Council on Education's College Credit Recommendation Service (CREDIT) connects workplace learning with colleges and universities by helping adults gain access to academic credit for formal courses and examinations taken outside traditional degree programs. Government, private-sector and non-profit organizations participate in this program that provides course equivalency information to facilitate college credit award decisions. Course accreditation is a rigorous process that takes approximately one year to complete. It includes submitting an application (including all course material), onsite course review, annual course audit and annual audit of instructional sites. Twenty six CLCS courses were submitted and approved for ACE accreditation, which provides under-graduate and graduate college credit. Courses approved include the Executive Development Seminar, Strategic Leadership, Team Building, etc., in addition to the Project Management Series, which culminates with the Project Management Institute's (PMI) certification exam. Customer Satisfaction Survey - In an effort to improve organizational effectiveness, OPM's Center for Leadership Capacity Services (CLCS) is partnering with the Assessment Services Branch, Center for Talent Services (CTS), to develop and administer customer satisfaction surveys. Personnel research psychologists from the Assessment Services staff of the Center for Talent Services have developed a standardized Customer Satisfaction Survey (CSS) to assess the quality of services provided by public-sector organizations. The survey is geared to agency training coordinators, supervisors and university coordinators. Based on nine core dimensions of service quality, the survey will help organizations identify their strengths and diagnose sources of dissatisfaction. The nine core dimensions are: Access, Courtesy, Knowledge, Timeliness, Reliability, Choice, Tangibles, Recovery, and Quality. The survey was benchmarked by GAO in 1999 as a valid and reliable measure to meet requirements of the Government Performance and Results Act (GPRA). The CSS has been used by a number of Federal agencies and by each office in CTS. If the same standardized service quality items are used, the results of the CLCS surveys can be benchmarked with the results of other organizations. Other Evaluations - Executive and leadership development programs have had extremely limited reviews by the Government Accountability Office (GAO). Reports were issued on August 15, 1984 (GAO/GGD-84-92) [http://archive.gao.gov/d6t1/124947.pdf] and in September 1989 (GAO/GGD-89-127) [http://archive.gao.gov/d26t7/139989.pdf], but there are no recent GAO examinations or testimony. Likewise the OPM Office of Inspector General has not conducted audits, investigations or reviews of leadership development programs.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: CLCS is a 100% reimbursable organization. All of CLCS' resources come through reimbursements from its customers. CLCS currently sets its budget levels based upon past performance, performance targets for the upcoming year, and a reasonable projection of future costs and customer demand for its products and services (see evidence). CLCS begins its process with a review of its current performance and performance targets for the upcoming year. Strategic initiatives for the upcoming year designed to improve performance metrics (for example, the current initiative to standardize on-line course evaluations across CLCS) are proposed, costed, ranked and then selected for implementation by the leadership team. CLCS publishes its course catalogues before the beginning of the fiscal year. For courses previously offered, it looks at the past number of attendees and then makes adjustments based upon the current environment (for example, changes in congressional budget levels, new administration policies, etc.) and planned sales and marketing initiatives. Once the volume of attendees is estimated by the business operations managers (BOMs) and Program Directors (usually completed in the last quarter of the previous fiscal year), CLCS projects its indirect costs and adds to it the target level of retained earnings and resources needed for the selected strategic initiatives and other investments. The BOMs then propose course prices based upon the historical gross margins designed to generate the needed revenues at the projected volumes. These prices are debated internally, especially with sales and marketing, and agreed upon. Once prices are set, the earned revenue targets are firmly established and adjustments are made to the budgeted levels for indirect costs, initiatives and investments. Targets for CLCS's performance measures were set by analyzing the historical performance, comparing performance to benchmarks (where applicable), analyzing the environment opportunities and threats, and agreeing on a vision of where the organization wanted to be at the end of the year in terms of its performance metrics. The suggested targets and their methodology were presented to CLCS leadership and discussed. Adjustments were made and the targets were finalized. A similar process will be repeated for upcoming fiscal years.

Evidence: CLCS is implementing a performance management process to track current performance against goals. The CLCS performance management process also incorporates current and projected financial information using recent, current and future budgeting information. The CLCS financial data shows the ratio of direct and indirect costs to revenue has been about 72% and 32% in FY 2004 through FY 2006 with a net loss of about 4%. For FY 2007, the planned ratio is 66% and 33% for net revenue of 1%. Different assumptions were used in developing the FY 2007 participant numbers and the corresponding revenue. The assumptions used are provided below. It should be noted that this was a much different methodology for projecting revenue and participants than had been used in the past. OE Methodology: ?? Starting with the draft calendar prepared by the Program Directors, CLCS took the current course catalog and plugged back in the programs already scheduled for the 1st quarter of 2007. ?? CLCS then looked at the current registrations for the 1st quarter classes to see if there was anyone registered and whether the class should be canceled due historical sales demand. ?? CLCS started looking at the core programs and the sales data for each class to determine the total number of sessions that should be conducted for the year. CLCS looked at the total number of participants for a year and the average enrollment at each location. CLCS also tracked 2006 participant registration against 2005 levels. ?? Then CPDF demographics were used to determine the location of target audiences. ?? In order to provide maximum scheduling options for its participants without over scheduling, CLCS went back and looked at the average number of participants over the past four years and set more realistic caps for each program. ?? The decision was made not to exceed caps and when all the spaces have been sold, the Web will be updated to reflect the course has been sold out. Participants will then be steered to other sessions. ?? Next CLCS looked at the breakeven point for each program to ensure full cost recovery and retained earnings. ?? In looking at prices for each class, CLCS looked at the expenses to determine whether an increase was needed. Its competitors' (CCL, AMA, MCI) pricing structure for similar programs was also considered. ?? 1st and 2nd quarter revenue was calculated using the first quarter discounted price multiplied by the program cap. This provided for a very conservative revenue estimate through the 2nd quarter because CLCS knows that not all participants receive the discount. Also the number of discounts offered in the 2nd quarter is less than the 1st quarter. The FY 2007 calendar/schedule reflects a different mix between one and two week programs, the addition of short 3-day programs, on-the-road programs, and more on-line programs. Thus the use of this methodology directly affects the overall number of participants and revenue for Open Enrollment Programs. By using historical sales data and trends, and keeping to the guiding principles, future projections should be much more predictable and reliable. Custom Methodology. The custom revenue amount CLCS has been increasing each fiscal year since its inception back in 2001. However, for FY07 CLCS is projecting flat revenues for its custom business line. The current forecasts for FY07 were based upon most known existing contracts with agencies that CLCS has worked with over the past few years. Organizations have conferred with CLCS and have given it dates for delivering training in FY07. CLCS agrees that there will be additional, one-time custom programs throughout the year, but CLCS needs to be careful as to how it projects out this revenue for this business line. The Succession Planning line of business polls its customers at the beginning of the year as to their planned PMF and CDP hiring needs. It then adjusts its revenue forecasts accordingly.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In 2003, OPM reorganized its organizational management structure to create the Division for Human Capital Leadership & Merit System Accountability (HCLMSA), the Strategic Human Resources Policy Division and the Human Resources Products and Services Division (HRPS), of which CLCS is a part. The reorganization improved strategic planning by putting all leadership development activities into CLCS, including the Western and Eastern Management Development Centers (MDCs), Federal Executive Institute (FEI) and the candidate development programs (President's Management Fellowship and Candidate Development Plan). This allowed CLCS to offer organizations a comprehensive solution to their succession planning, leadership development and competency gap challenges. One challenge presented by the reorganization was the fact that the appropriated funds supporting the PMF program prior to the reorganization were no longer provided, and it took almost three years to get approval for an appropriate price increase to cover the deficit. OPM's new Strategic and Operational Plan 2006-2010 builds upon this reorganization by setting straightforward and identifiable goals tied to each program with dates for clear accountability. Subsequent budget submissions will tie resource requests to these (and other) operating goals. OPM has also created a new, comprehensive governance process. Progress reviews with senior program management (Center Directors) are held on a regular basis to review the progress in achieving the operational goals detailed in the new strategic plan. These Directors are tasked with cascading organizational goals down throughout their organizations. Through the Office of the Chief Financial Officer, there are quarterly operations reviews, including budget and performance data, targets and operational plans. Performance results and issues are discussed at weekly management meetings which include all higher level managers. CLCS is currently in the process of updating its strategic plan based upon the new OPM strategic plan and the vision documented at an organizational off-site conducted in early 2006. CLCS is currently facing a retained earnings shortfall and is implementing steps to mitigate its projected fiscal shortfall, including increasing prices for its products and services where possible, improving management control, and increasing attention to strategic performance measures.

Evidence: Prior to the reorganization, CLCS was known as the Office of Executive and Management Development and was organized as separate training centers. The reorganization created funding problems discussed above, but the new OPM Strategic Plan and more recent budget submissions have mitigated these problems. For example, CLCS participated in OPM's strategic goals for Professional Development (B-1 and B-2) and plays an ongoing role in Strategic Goal C, "Federal Agencies will be employers of choice" particularly with regard to effective leadership. It also contributes to improved financial management and internal controls by achieving full cost recovery annually (E-21). This has provided additional incentive for CLCS to focus on its mission and increase its impact on federal government.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program regularly collects performance information to manage the program and improve performance, including data on the performance of its partners, and uses the data to make informed resource and program decisions. For all business lines, CLCS tracks its performance by (1) closely monitoring contractor products and services for compliance with performance-based contract clauses and (2) soliciting suggestions for improvements and future needs through its customer surveys. CLCS uses a variety of customer surveys to determine the quality of present service and future needs. CLCS class participants complete surveys on basic satisfaction with services (class content, facilities, etc) and knowledge transfer (Kilpatrick Level 1 Measure). CLCS also surveys training participant supervisors to determine the impact of the training on the organization Thus, CLCS continuously monitors customer feedback, which allows CLCS management to address issues immediately by bringing them to the attention of staff. Financial performance data on all CLCS sessions and contracts are collected and analyzed through CLCS's own internal tracking systems (so-called CLCS "cuff" systems. Information is collected in real time and used as the basis for ensuring project cost recovery, setting annual revenue and cost targets, measuring specific project outcomes, and prioritizing allocation of resources. Finally, CLCS communicates with its clients and other stakeholders through forums and contacts made in person or by phone or email. In these ways, CLCS identifies needs for new programs. On a monthly basis, actual spending is compared to budgeted spending with the goal of ensuring efficient and effective use of resources. The operating budgets are reviewed quarterly with the OCFO, and any variances are addressed.

Evidence: Some examples of CLCS survey types and questions include ?? The PMF orientation survey ask question like "I developed practical coaching skills as a result of this training" and scored using a five point scale ranging from 5 (strongly agree) to 1 (strongly disagree) ?? The SES Briefing survey asked more open ended questions like "If the briefing wasn't effective what can we do to improve?" in addition to a few general questions graded on a five point scale ranging from 5 (excellent) to 1 (poor) ?? The CLCS course evaluations ask a variety of questions about course design, instructor's delivery and course impact. The survey is graded using a five point scale ranging from 5 (strongly agree) to 1 (strongly disagree)

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The program's executives and managers are held accountable for performance results through the performance appraisal process. CLCS holds its program partners (contractors and facility providers) accountable through its performance contracts. These are built into task orders, and CLCS continuously monitors partner performance. Where possible, CLCS utilizes fixed price contracts. OPM's Director holds Associate Directors (AD) accountable for achieving OPM's strategic and operational goals through monthly reviews. The HRPS AD holds the Deputy Associate Director, CLCS accountable through his performance plan. In turn, individual CLCS managers' performance plans include expectations that are linked to operational goals and strategic objectives. All manager performance plans are currently being reviewed in light of OPM's new strategic plan, and OPM's operational goals are being incorporated into individual performance plans to contain quantifiable performance standards. Program managers are held accountable for cost, schedule, and performance results through the annual performance review, mid-year performance review, and quarterly financial and performance metric reviews. Managers in turn hold their employees accountable through their performance plans for providing work products that meet standards for quantity, quality, and timeliness.

Evidence: Evidence of managerial accountability include managerial Performance agreements include the SES annual performance plans (which are tied to OPM's strategic plan), financial targets, and/or annual work plans as key elements on which CLCS managers and supervisors are evaluated. As mentioned earlier, sample contracts provided by CLCS all contain performance clauses and detailed project plans and deliverables along with delivery dates. CLCS managers and staff use a number of reports from OPM's central financial and payroll IT systems, such as Summary Reimbursement Reports which present revenues and expenses by initiative and project/contract); and year-to-date payroll. To provide a comprehensive status and accounting of a particular project, business line, or CLCS as a whole, managers and staff combine the data from various reports to produce customized reports.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: CLCS ensures that funds are obligated and spent in accordance with program requirements. The program is managed consistent with the goals and priorities outlined in OPM's strategic plan. No funds allocated to the program are redirected to other OPM programs. Together with the OCFO, formal quarterly assessments are conducted with the Director of actual and projected expenditures. This assessment covers costs associated with salaries and benefits, travel, and other objects. This process facilitates resource planning and management, allowing for adjustments as needed to accomplish identified program needs. To ensure that funds are obligated and spent as intended, the program analyzes OPM's Government Financial Information System (GFIS) transactions to identify and resolve any problems pertaining to funds being obligated and expensed. OPM budget staffers establish budgetary resource levels in GFIS for each fiscal quarter against which the program charges appropriate business expenses, most of which are payroll costs, contractor expenses, facility management fees, and overhead charges.

Evidence: CFO reports, including income statements and operating plan budget reports including travel expenses, indicate that funds are obligated in a timely manner, spent appropriately and accurately reported. CLCS's Quarterly Director's reviews demonstrate that OPM monitors CLCS obligations closely. CLCS also uses reports from OPM's central financial and payroll IT systems, such as Summary Reimbursement Reports, which present revenue and expenses by project and object class. Various customized reports are also used by staff and managers to provide a comprehensive status and accounting of a particular project or center.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Projected improvements in efficiency and effectiveness are an important part of making program decisions. The 2003 OPM organizational realignment allowed CLCS to group together its leadership development activities and focus on a few key strategic objectives for improved effectiveness and more efficient execution. Projected improvements included an increase in the price charged for the PMF Program and periodic review of course profitability. All major contracts are rigorously competed. Where feasible, CLCS utilizes performance-based or firm-fixed price contracts to achieve cost efficiencies.

Evidence: CLCS has several established procedures for improving efficiency. For example, it monitors the profitability of each session, monthly income and expenses, and the extent to which it achieves full cost recovery. As a fee-for-service organization, CLCS also monitors competitor costs regularly to ensure its products and services remain competitive and it regularly reviews its pricing models and practices. CLCS managerial performance agreements include financial targets as key elements on which managers and supervisors are evaluated. An example of a rigorously competed, performance-based contract is FEI's adjunct faculty recruitment process. This is a major solicitation process every 5 years whereby CLCS selects these contractors for their services based on best value (past performance and pricing issues). After award, these contractors are set up under a Blanket Purchase Agreement. There is no guarantee of work, only that CLCS may use their services.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Internally, the Eastern and Western Management Development Centers (EMDC and WMDC) collaborate on curriculum such that programs offered at both Centers are the same. A student attending a Seminar at Eastern has the same material and requirements as one who attends the same seminar at Western. Where EMDC and WMDC offer different Seminars, they are careful not to overlap or undercut one another's offerings. Together, EMDC and WMDC coordinate with Federal Executive Institute (FEI) and its Center for Executive Leadership (CEL) program to avoid duplication of effort and overlapping of offerings]. Externally, CLCS collaborates with American University (CLCS's partner in the AU/OPM program) to recruit students, direct financial resources to the appropriate institution, and schedule seminars to accommodate the time needs of Masters of Public Administration (MPA) students in a cohort-based program. CLCS also has a "custom" line of business at both the FEI and MDC levels to work with agencies individually to help solve their leadership development challenges. It collaborates with them to develop custom solutions. The entire PMF Program is a collaborative effort between OPM, academia, and agencies participating in the program to recruit, hire and develop top new talent.

Evidence: Evidence of CLCS's collaborative approach include ?? Collaborative outreach and recruitment efforts at hundreds of participating universities. Avenues of communication include the President's Management Fellowship (PMF) Campus listserv, correspondence to rented lists of deans and career professionals in graduate programs, emails to prior academic nominating officials, and periodic meetings with academic partners. ?? Collaboration with hundreds of agency hiring officials facilitated by hosting an annual Job Fair, providing online resume searches and job postings; training offerings, agency support through a network of Agency PMF Coordinators; and an extensive set of online agency resource materials. Some of the specific partnerships with related programs include: ?? Partnering with the Eastern Management Development Center in Shepherdstown, West Virginia to develop and support the residential PMF Orientation sessions. ?? Working with the Assessment and Training Assistance Services (ATAS) branch of OPM's Center for Talent Services to develop a Customer Satisfaction questionnaire that they will independently administer to agency clients. ?? Partnering with OPM's Examining and Consulting Services Group (ECSG) for services associated with application to the PMF Program and subsequent assessment for the purpose of selecting Finalists. ?? Partnering with OPM's Technical Services Group (TSG) for services related to development and support of our website; development of custom software programs used in program operations, including a division-wide LMS database system to manage training and event registration, and Individual Development Plan (IDP) information, hosting and archiving of databases containing information about our applicants and finalists; and PMF Agency Coordinators. ?? Working with CLCS on a division-wide LMS database system to manage training and event registration, as well as IDP information. ?? Scheduling periodic meetings with the academic community to gain their unique perspective on how CLCS can best improve the program. They are an important stakeholder because they provide the talent pool that the PMF Program selects Finalists from to market to agencies. CLCS is currently working to develop e-mail updates and other communication strategies that will increase the communication to this community.

Does the program use strong financial management practices?

Explanation: OPM's annual consolidated financial statements are independently audited by a well-recognized international CPA firm. These auditors reported no material internal control weaknesses from their audit of OPM's financial statements for FY 2005. CLCS is funded through OPM's Revolving Fund account. The auditors identified no material weaknesses. In addition, the program has procedures in place to ensure that payments are made properly for the intended purpose, and OPM has identified no erroneous payments. OPM's financial management systems meet statutory requirements and financial information is recorded accurately and timely.

Evidence: The program utilizes weekly, monthly and quarterly financial and performance data to support its day to day operations. For example, monthly financial reports track spending versus budget, quarterly performance reports track the participant training days, etc. CLCS produces an internal resource summary and create both monthly and quarterly reports and projections. There is also a costing spreadsheet that is used across all centers.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: During FY 2006, OPM under the leadership of its new Director undertook an extensive review of its mission and strategic plan. As a result, it recently issued a new Strategic and Operating Plan 2006-2010. In the early fall of 2004, EMDC underwent a management review. Subsequent actions were undertaken by program leadership to address deficiencies surfaced by this. An EEO Climate Assessment was conducted with suggestions for improving areas that need improvement. The staff also participates in a yearly retreat to share their views and concerns. OPM continues to focus its efforts to substantially improve the agency's financial management, internal controls, operations, and reporting. OPM's CFO has received the agency's sixth consecutive unqualified ("clean") opinion on the agency's FY 2005 annual consolidated financial statements. The OCFO identified that it was not meeting the requirements of the prompt payment act. It reviewed its procedures and made changes to improve payment processing. It generates monthly reports of prompt payment metrics, reviews the results and continues to modify its processes and procedures as needed. As a result, payments made within guidelines have increased. The program has a system for identifying and correcting program management deficiencies and uses the system to make necessary corrections within agreed upon timeframes. Specifically, monthly financial reports are communicated to program managers and monthly meetings are held with program and financial management to review the reports and identify inaccuracies and other issues. In addition, the OPM Director conducts quarterly financial and performance reviews with all OPM executives. Finally, OPM's Center for Internal Control and Risk Management oversees OPM internal certification process for the Federal Managers Financial Integrity Act and tracks corrective actions regarding all recommendations accepts from independent auditors and 3rd party reviews of OPM programs. CLCS's PMF program has taken the following steps to enhance its reach and effectiveness: ??Facilitated the transition of key leadership personnel. A major management challenge is the relatively small size of its operational staff and the turnover of key personnel. The program utilized a shared computer drive and filing system as an important knowledge management strategy. ??Established training programs and reference materials for PMF Agency Coordinators that include a revised online PMF Guide for Agencies, regular agency PMF Coordinator training, and ongoing technical support for agency partners. Trained personnel handle myriad requests for assistance that are made to the PMF Program Office staff. ??Provided transition guidance, training and individual technical assistance to agencies for implementation based on the new EO and regulations. ??Revamped the assessment process, which, in addition to cost effectiveness, will allow it to hold its Job Fair event earlier in the year to lock in top talent before they get other offers. This reform will also alleviate time pressures inherent in the current process) [3.7.6 New Assessment Process]. ??Redesigned and streamlined the development programs, which will reduce costs and labor demands on the Program Office [3.7.7 Streamlined Development Program]. ??Provided more stringent financial management through improved fee collection to correct a long-standing management deficiency within the Program [3.7.8 Improved Fee Collection]. ??Partnered with USA Staffing for its application process to eliminate labor intensive processes involved in the current requirements for PMF application notice approval 3.7.9 Application Outsource].

Evidence: CLCS and the OPM have addressed management deficiencies with: ??OPM's Strategic and Operating Plan 2006-2010, ??OPM's Performance and Accountability Report (PAR), ??management review reports; Prompt payment reports; detailed remediation plan to overcome the FY 2004 audit findings and current status update, and ??Status of Corrective Action Plan for FY 2005 Consolidated Financial Statements

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: CLCS expects to meet all of its long-term performance targets for those measures which had targets in FY2006 (one did not because it is a new measure). These long-term measures are key indicators of the program's performance in executing the CLCS strategy and OPM's Strategic and Operational Plan - 2006 - 2010. ?? 1.0 Individual Impact-Perceived Learning. CLCS targeted a 40% increase in the before and after knowledge of its training class participants. Based upon the FY06 result of 40%, CLCS achieved its target and can say that it had, on average, a significant learning impact on individuals who participated in CLCS training. ?? 2.0 Organizational Impact-Perceived Behavioral Change . A new measure has been developed for impact of training on organizations. CLCS has piloted this measure at one Federal agency and is in the process of standardizing this measure for all applicable custom engagements. ?? 3.0 Full-cost Recovery. This measure is a part of OPM's Strategic Plan for 2006-2010. Due to anticipated shortfalls in its Candidate Development Program, CLCS targeted recovering 96% of its costs. In FY2006 CLCS exceeded its target and recovered 97% of its costs, as structural and pricing changes began to show in the program's financial results. CLCS is targeting recovering 101% of its costs in FY2007, and is well on its way to achieving that ambitious goal. ?? 4.0 Number of Hired PMFs. CLCS targeted 375 PMF hires for the class of FY2006. Agencies may hire from that class of finalists up until March 31, 2007. To date, 355 PMF finalists have been hired, and based upon historical trends (188 of the 366 class of 2005 hires have EOD dates after October 1, 2005), CLCS anticipates meeting or exceeding its target for the class of 2006. This is a clear indication that, provided agencies have the budget, the PMF Program is effectively providing needed talent to the agency recruiters. PMF is targeting 400 PMF hires for the class of 2007, and has all ready instituted a new agency marketing program to ensure that target will be met.

Evidence: Please see measures tab. Evidence supporting CLCS's long-term goals can be found in OPM Strategic and Operational Plan 2006-2010, OPM's Performance and Accountability Report and OPM's Congressional Budget Justification. CLCS also relies on the OPM's Central Personnel Data File (CPDF) and participant and organizational surveys to support the long-term measures.

Does the program (including program partners) achieve its annual performance goals?

Explanation: CLCS has 13 annual measures of which 7 are new measures for which it did not have performance goals in FY2006. Of the remaining 6 measures, CLCS met one third of targets, and came very close to achieving another one thirds of its targets. ?? 1.1 Government-wide Reach. CLCS had targeted reaching 3.5% of eligible federal managers in FY06, and final results show that the program reached 2.6% of federal managers. This result was due to a number of course cancellations and other factors (see evidence). CLCS is targeting 3% reach in FY2007, and has already instituted changes aimed at mitigating the factors under its control which led to FY2006's disappointing results. For example, it issued its FY07 course catalogue much earlier than for FY2006 and plans an update in mid-FY07, has begun a telemarketing and direct sales call (visit) campaign with agencies, and will be running more one week programs and short programs (3-4 days), all designed to increase participant numbers. ?? 1.1a Percent of new customers - Individual. CLCS strives to increase the number of Federal managers who attend its courses for the first time, or have returned after an absence. CLCS targeted 83% of its customers to be new customers, and results in FY06 showed that 78% of its participants were new customers. It has targeted 81% for FY 2007, and with the new customer efforts mentioned above it expects to achieve this target. ?? 1.1b Percent of repeat customers - Individual. Repeat customers are defined as individuals registered for courses who have taken a CLCS course within the past 35 months. CLCS targeted 8.8% of its pool of past participants to become repeat customers. Final FY06 results showed that CLCS achieved 8.6%, very close to the targeted level. And well above its FY05 result of 6.9%. Repeat customers are an indication that CLCS has delivered effective training at a good value. With new initiatives geared towards increasing its repeat customers (creating course series' leading to a certification, for example), CLCS expects to achieve its ambitious target of 9.9% repeat customers. ?? 1.2 Overall customer satisfaction - Individual. Even during its transitional year, CLCS customer satisfaction was very high in FY2006. Participants reported an average satisfaction level of 4.67 on a 5-point scale in FY 2006, well above the 4.60 target and the FY2005 result of 4.61. CLCS is improving its courses and facilities in a effort to meet or exceed its FY 2007 target of 4.68. ?? 3.1 Earned revenue. This is the amount of revenue earned by the program. CLCS had targeted revenues of $51.6 million and achieved revenues of $43.6 million in FY06. The gap was primarily due to reduced participant numbers for reasons noted above. CLCS is targeting $47.1 million in FY07, an increase of 8% and sufficient for full cost recovery as its new initiatives begin to take hold. In FY08, it is targeting a similar increase to $50.6 million. ?? 4.2 PMF Customer Satisfaction. In order to best meet the needs of its customer agencies, CLCS must be able to satisfy their needs efficiently and effectively. Agencies reported a satisfaction level of 4.08 on a 5-point scale, just below the target level of 4.10. If customer agencies are more satisfied, then this may positively influence the number of PMF hires converted into permanent positions. As previously stated, CLCS's other annual measures are new and while historical data exists, there were no targets set for FY06. Data for these measures is being tracked quarterly in FY07 (where applicable) and CLCS is committed to achieving its ambitious targets for these performance measures.

Evidence: The lower participant levels in FY 2006 can be directly attributed to the following events: a) an extended period of Continuing Resolution for most agencies in the first quarter, b) guidance provided by the President to reduce travel expenditures due to hurricane Katrina, c) budget uncertainties due to the continued Global War on Terrorism, d) CLCS's inability to provide catalogues to its prospective clients during the latter part of the previous fiscal year, e) the inherent reality of people taking "use or lose leave" in conjunction with the holidays, and f) lack of a coordinated marketing/sales strategy to promote IAR programs. Each line of business has created a marketing/sales strategy for FY07 which are currently being implemented. Program perfromance is also demonstrated by the FY2005 and 2006 OPM PAR, CLCS Strategy map and strategic objectives, the 2007 CBJ, and in the attached PART measures tab.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: CLCS has demonstrated improved efficiencies in FY2006 by reducing direct and indirect costs. Direct costs are defined as costs that can be specifically identified with the production and delivery of the output like employee salaries and rental expenditures. Indirect costs are defined as the costs of resources that are jointly or commonly used to produce two or more types of outputs but are not specifically identifiable with any of the outputs.

Evidence: CLCS measures its direct costs (those specifically identified with CLCS products and services) as a percent of revenues. In FY06, direct costs were 71.7% of revenues, down almost 1% from the FY05 level of 72.6%. Indirect costs (those costs not specifically identifiable with an output) fell from FY2005 to FY 2006 by almost $165,000. This had the combined effect of reducing program losses from FY05 to FY06 by 19%, even though program revenues fell during the same period. IT-based systems are continuously being improved to increase efficiency and meet customer needs. The new Learning Management System, which tracks participant traning, and electronic registration has and will continue to improve efficiencies and customer service, as indicated by the increase in overall customer satisfaction from FY05 to FY06.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: CLCS is a new reimbursable program, established during OPM's reorganization in 2003, and stopped receiving appropriated funding in 2004 in order to become 100% reimbursable. ??CLCS is the only federal provider of classroom based leadership development programs. Its customer satisfaction results compare favorably to industry standards. ??CLCS's cost per ACE accredited courses (course hour equivalent) for comparable categories is lower than that of private-sector providers. ?? CLCS's direct costs as a percentage of revenues are within 5% percentage points of industry averages, even though its prices are substantially below that of private sector providers.

Evidence: CLCS sent its first standardized Customer Satisfaction Survey organizational customers in FY 2006. The survey was designed and conducted by OPM's Center for Talent Services and will be repeated each year. It is able to be benchmarked against the American Consumer Satisfaction Index (ACSI). The FY06 survey yielded a very high result of 4.55 on a 5 point scale well above the ACSI benchmark of 71.3. Customer satisfaction is very high among CLCS course participants as well. On a 5 point scale, for FY 04, 05 and 06 CLCS scored 4.69, 4.61 and 4.67 respectively, also comparing very favorably to the ACSI (aggregate on a 100 point scale). FEI's programs are accredited by the American Council on Education (ACE) at the college program graduate level. FEI participants compare FEI's LDS program to executive programs at Harvard Kennedy School and Center for Creative Leadership. Comparative Evaluation of MDC programs Originally established as a delivery organization in which information regarding public policy could be shared with Federal managers and executives on an interagency basis, and through which networking relationships across agency lines could originate and develop, the Management Development Centers have been addressing the question of how to confirm the effectiveness of their Seminars practically since the inception of the centers. Some "open enrollment program" competitors' measure results by means of downstream surveys of participants ("did your behavior change as a result of having taken this course?"??Kirkpatrick level 3), or by studies of Return on Investment ("Measured in dollars or time saved, what was the impact of this training on your organization?"??Kirkpatrick level 4). ROI, which compares the cost of training against the benefits that the training can produce, can yield suspect results: ROI is rarely less than 1,000% and frequently much more??after all, if a manager in a very large agency attributes to a training course his insight that the Agency can do without one report it currently requires, then the saving in lost person-hours, lost opportunity-costs, wasted paper, etc. is likely to dwarf the cost of the training into utter insignificance by comparison. This data can appear useful, but it is a byproduct of what leadership training is FOR: transforming organizations in ways beyond ordinary cost-savings to meet the challenges of a changing environment. The private sector benchmark in the business training services industry for direct costs as a percent of revenues is 64.9% (Bizminer, June 2006).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Currently, the independent evaluation of CLCS comes from a variety of different sources: (1) Customer feedback from surveys (2) Sales volume for CLCS's programs (3) Government wide and private-sector comparisons of CLCS using a variety of measures, including financial and customer satisfaction (4) American Council on Education (ACE) course accreditation.

Evidence: There is no comprehensive, independent evaluation of CLCS.