Detailed Information on the
Center for Talent Services - HR Products and Services for Federal Agencies Assessment

CTS will submit an organizational restructuring plan including timeline and results. CTS will also submit a periodic status report of implementing the restructuring plan.

OPM will address the deficiencies discussed in section 2.7 of CTS PART concerning reflecting full costs such as indirect costs.

CTS will explore ways to measure the direct impacts of its products and services. For example, CTS should examine whether it is useful to tract pre- and post-"treatment" (before and after CTS services rendered) results for certain measures.

CTS will submit by September 9, 2005, a draft 3-year evaluation plan that covers most appropriate independent evaluations for different lines of CTS business.

Measure: Full cost recovery maintained over 3 years

Explanation:Full cost recovery is a function of retained earnings which are calculated with total earned revenue and all program costs.

Measure: Full cost recovery in any given year

Explanation:This is the total earned revenue divided by all program costs in any given year.

Measure: Providing good monetary value to CTS customers

Explanation:CTS business lines are likely to be cost-effective if customers perceive they are receiving good value.

Measure: Quality of CTS products and services

Explanation:If customers agree that they received high-quality products and services, they are more likely to consider them good value for the money.

Measure: Repurchase intention

Explanation:If customers express repurchase intention, it indicates that CTS provides good value for the money.

Measure: Timeliness of CTS products and services

Explanation:If customers agree that they received high-quality products and services, they are more likely to consider them good value for the money.

Measure: Overall customer satisfaction with CTS products and services (ACSI equivalent)

Explanation:The equivalent of the American Customer Satisfaction Index (ACSI) can be used for comparisons with Government and private industry and indicates whether CTS is meeting customer expectations and providing good value. Increased customer satisfaction leads to increased repeat business and higher revenue.

Measure: CTS contribution to organizational effectiveness

Explanation:If CTS sustomers agree that CTS products and services increase organizational effectiveness, agencies are more likely to become high-performing.

Measure: Mean on OAS dimensions for repeat vs. all OAS customers

Explanation:If we reduce time to recruit, assess and/or refer candidates, agencies are more likely to reach high-quality candidates and fill competency gaps, thus increasing agency workforce quality.

Measure: Satisfaction with candidate quality

Explanation:If CTS recruitment and assessment results lead to higher satisfaction with candidate quality, agency workforce skills should improve.

Measure: Percentage of high-quality eligibles out of all qualified applicants per vacancy at CTS client organizations

Explanation:A higher proportion of high-quality eligibles from which to select will increase agency workforce qaulity and indicates that job announcements are targeting required competencies.

Measure: Percentages of vacancies that attract at least 10 qualified applicants

Explanation:Vacancies with fewer than 10 qualified applicants are less likely to include sufficient numbers of highly qualified applicants than vacancies that attract more qualified applicants.

Measure: Examining support timeliness in days

Explanation:This measures how many days it takes to issue a certificate list of eligible candidates to a client agency. If CTS reduces time to recruit, assess and/or refer candidates, agencies are more likely to reach high-quality candidates and fill competency gaps, thus increasing agency workforce quality.

Measure: Workforce skills for CTS customer agencies that received competency-based assessment and customized training services vs. all agencies governmentwide

Explanation:If this metric improves, the workforce has improved knowledge and skills to accomplish organizational goals in part due to more effective recruitment/selection and training assistance from CTS.

Measure: Satisfaction with quality of training services provided by TMA

Explanation:If agencies agree that training services provided by TMA contractors are of good quality, workforce skills should increase.

Measure: TMA acceptance rate

Explanation:This represents the percentage of client-accepted deliverables (or products) based on the total number of deliverables submitted and invoiced by TMA contractors.

Measure: USAJOB's user's satisfaction with federal employment application process.

Explanation:Increased satisfaction with federal application process will increase the number applying for federal jobs and increase likelihood of attracting more qualified applicants.

Measure: USAJOBS users' satisfaction with USAJOBS website - Index score

Explanation:Increased satisfaction with USAJOBS website will increase the number applying for federal jobs and increase likelihood of attracting more qualified applicants

Is the program purpose clear?

Explanation: The purpose of the Center for Talent Services (CTS) at the Office of Personnel Management (OPM) is to provide Federal agencies with relevant, cost-effective reimbursable human capital products and services, based upon merit system principles, needed to build a high-quality workforce and become high-performance organizations.

Evidence: Title 5 USC Section 1304(e)(1) authorizes OPM to establish a revolving fund for financing, among other services, personnel management services performed at the request of individual agencies. The title also states that such requested services are those that would otherwise be the agencies' responsibility to provide. Since these services are at the request of agencies and would otherwise be the responsibility of the agency to provide, the services provided by CTS are, by definition, relevant to its customers. Furthermore, Title 5 USC Section 1304(e)(1) states that OPM is required to provide services "to the maximum extent feasible . . . on an actual cost basis". Since OPM is required to provide services on an actual cost-basis, its services are cost-effective in that "profit" is not authorized. OPM's Strategic Plan also drives CTS's purpose. Strategic Goal 3 of the plan sets the expectation for the delivery of efficient and effective products and services. Objective 1 of that goal calls for those products and services to be effective, relevant, and useful to agencies. The work of CTS ultimately supports OPM's Strategic Goal 1, which is to help agencies improve their ability to build high-performance organizations, and especially Objective 4, which is to increase the effectiveness and efficiency of the Federal workforce by attracting high-quality applicants of diverse backgrounds. Therefore, CTS's purpose??to provide Federal agencies with relevant, cost-effective reimbursable human capital products and services, based upon merit system principles, needed to build a high-quality workforce and become high-performance organizations??is authorized by statute and supported by OPM's Strategic Plan goals and objectives.

Does the program address a specific and existing problem, interest, or need?

Explanation: CTS's products/services are designed to help Federal agencies: (1) transform the management of Federal human capital (as outlined in the President's Management Agenda); (2) accomplish their missions, by having the right person in the right job at the right time; and (3) successfully implement effective personnel systems. The human capital problems and needs that CTS addresses range from those that are agency-specific to those faced by all agencies. CTS's 2004 Customer Satisfaction Survey (CSS) revealed that 94% of respondents agreed that CTS services are designed to meet customer needs. CTS has seven related lines of business: 1. USAJOBS. Through USAJOBS, job seekers can search a database of Federal jobs, apply on-line, and track the status of their applications. This program includes: (a) the www.usajobs.gov website, StudentJobs website, and USAJOBS by Phone; and (b) customer support services for job-seekers and recruiters. 2. USA Staffing. This is the only Government off-the-shelf hiring software that generates vacancy announcements, posts them on USAJOBS, accepts applications, rates applicants, produces lists of candidates, notifies applicants of their status, and maintains records. 3. Technical Services. CTS can modify USA Staffing to address agency customization needs. CTS also runs three benefits information technology (IT) systems -- Employee Express, myPay, and Services Online -- that allow employees and annuitants to manage their own payroll and personnel transactions (change of address, withholding amounts, etc.). 4. Nationwide Testing Service (NTS). With a national network of test administrators and state-of-the-art scheduling and processing systems, CTS administers approximately 40,000 pen-and-paper test sessions annually to nearly one million applicants seeking employment with Federal agencies or acceptance into the Armed Services. 5. Individual and Organizational Assessment Services. CTS develops competency assessments for employee selection, promotion, and career development. Individual assessment services include a Leadership 360 assessment instrument, skills gap analysis, selection and promotion tests, and technical consulting. CTS also offers two standardized surveys that assess (1) organizational culture and climate, and (2) customer satisfaction. Services include survey customization or design, sampling, survey administration, data analysis, and assistance with action planning based on survey results. Finally, CTS provides program evaluation services (e.g., evaluation of HR demonstration projects). 6. Examining and Consulting Services (ECS). The examining units offer a range of hiring services - from accepting and evaluating applications to identifying eligible candidates. Consulting services include workforce and succession planning; organizational design and restructuring; recruiting services and employer branding (creating employer brands and agency logos); and human resources capacity building (filling short-term vacancies and conducting long-range staffing initiatives). 7. Training and Management Assistance (TMA). CTS offers a list of pre-qualified vendors for training and human capital (HC) management (such as compensation and performance management), which agencies can use through an expedited contracting process. CTS, acting as a contract facilitator, works with a client to develop statements of work and select a vendor. Once the contract is signed by the client agency and vendor, CTS provides contract management and oversight on the client's behalf.

Evidence: Many observers have noted the HC challenges facing Federal agencies. In early 2001, the General Accountability Office (GAO) designated the Federal Government's strategic HC management as being in the "high risk" category. In August 2001, the President's Management Agenda (PMA) was launched, and one of its five Government-wide initiatives is Strategic Management of Human Capital (SMHC). All initiatives on PMA represent, as the President put it, "the most apparent deficiencies where the opportunity to improve is greatest." The PMA tracks agencies' status against established Standards for Success as well as their progress toward meeting standards. At present, the SMHC scorecards indicate that 65% of agencies need to improve human capital management (17 out of 26 agencies scored red or yellow in status, as of December 2004). The areas of effective hiring, succession planning, and leadership development are frequently cited as requiring urgent improvement across the Federal Government, as in the following reports: o General Accounting Office. (2003). Human Capital: Opportunities to Improve Executive Agencies' Hiring Processes. Concludes that to reduce time to hire, OPM needs to assist agencies in developing improved assessment tools. o Partnership for Public Service. (2004). Asking the Wrong Questions: A Look at How the Federal Government Assesses and Selects Its Workforce. Concludes that the Federal hiring process is broken and in need of dramatic reform, and that effective assessment tools are essential to guarantee a high-performing Federal workforce. o U.S. Merit Systems Protection Board. (2003). The Federal Selection Interview: Unrealized Potential. Identifies wide variation in how Federal agencies and managers conduct job interviews, and recommends greater use of structured (vs. unstructured) interviews. Finds that structured interviews are better predictors of on-the-job performance than unstructured interviews. Two concrete examples illustrate how CTS meets agency-specific needs. First, to fill 120 positions in 10 different IT specialties across several agencies, CTS conducted a virtual job fair in conjunction with Federal Chief Information Officer Council. CTS created two on-line IT assessments and then partnered with a private-sector vendor on the administration of on-line IT knowledge tests. CTS also posted vacancy announcements, processed applications using USA Staffing, and housed the system's infrastructure. The job fair generated 18,000 applicants, and CTS delivered certificates of eligible candidates to client agencies just 10 days after the job fair. After CTS completed its role in the process, its client agencies??at their own pace??filled all 120 positions, two-thirds of them within three to four weeks. Second, CTS recently worked with NASA Langley Research Center (LaRC) to design and implement executive and managerial assessments for personnel placement after a re-organization. CTS developed lists of the executive and managerial competencies needed in the new organization. The LaRC Director used this information to make placement decisions for the executives, and the executives used the assessment information to select managers. The Director and other executives reported high satisfaction with the process and the resulting placement decisions.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The market for HR products and services for 1.8 million Federal civilian employees is very large and allows for many service providers to meet the demand. While many of CTS's products/services are duplicated elsewhere in this vast market, CTS's presence adds value to both its agency customers and to the market as a whole for three key reasons: 1. CTS provides its products/services at lower prices than those charged by other vendors in the market. To offer its reimbursable services/products on a legally-required "actual cost basis," CTS charges customers only enough to cover costs and maintain security through minimal retained earnings, compared to profit-maximizing prices in the private sector. 2. In comparison to most private-sector companies, CTS serves the broader government good. For instance, while private-sector companies develop tools and knowledge in one agency and resell these at full price to other agencies, CTS reduces development costs for the next customer or ?? in the case of its Training and Management Assistance line ?? negotiates with its contractors to do so. Also in the case of assessment services, CTS's systemwide perspective and extensive Federal HR experience enables CTS to create valid Governmentwide HR benchmarks not found in the private sector. 3. CTS employees have significant subject-matter expertise in public HR (merit-based, legal framework), accumulated through years of Federal HR work. CTS also has access to the latest Federal HR policy and HC leadership guidance through its close relationships with OPM's policy and compliance divisions. Below are two specific examples of how CTS compares with its competitors: A. USA Staffing. Private-sector software packages may require customization to meet statutory Federal requirements. By contrast, USA Staffing is designed for Federal hiring and reduces technical problems and potential interruption in service that may otherwise be incurred through post-purchase adaptation of private-sector off-the-shelf solutions. B. Training and Management Assistance. CTS's training and management assistance services are similar to those offered by the General Services Administration (GSA), as both provide a contracting vehicle for agencies looking to acquire HR products/services. However, the two services use different delivery models. GSA only lists service providers and their prices. On the other hand, for a slightly higher contract management fee, experienced CTS project managers provide comprehensive contract facilitation and contract oversight, a skill that is in short supply in many agencies. CTS claims to be the only provider with a money-back guarantee if the client is not satisfied.

Evidence: Claim 1 - Providing HR products/services at lower prices A. CTS won three A-76 competitions, which are competitions with the private sector to provide the best value for the money. These were the three competitions: Nationwide Testing Service (NTS) conducted in 2003 (for 1,000 positions); Teleservice and Transcription (TT) in 2002 (for 32 positions); and Computer and Building Management Assistance (CBMA) in 2004 (for 22 positions). In each competition, CTS costs were considerably lower than those of private competitors (evidence provided to OMB). B. CTS has compared an average cost per job posting for USAJOBS to that of different private companies, using the advertised/published high and low rates as of March 2005. The cost comparison shows that USAJOBS offers more features and functionality to customers at a lower price than potential competitors. C. CTS's billable labor rates for IT technical services and HR consulting services are much lower than its competitors'. For instance, CTS's rate for the HR consulting services of advanced staff with extensive Federal experience is comparable to that of a junior consultant in a major private-sector consulting firm. D. According to CTS Customer Service Survey (CSS), 90 percent of respondents view CTS performance as superior to that of private sector contractors for similar work. Claim 2 -Serving the broader government good A. CTS's competency-based assessments and standardized survey are distinguished from similar private-sector tools in several ways. CTS uses a common competency language based on Governmentwide job analysis studies for its individual assessments. It has also developed a longitudinal, benchmarking database of Federal agency results for the Organization Assessment Survey (assessing organizational climate/culture), Customer Satisfaction Survey, and OPM Leadership 360; these allow CTS survey customers to compare their results directly to the Governmentwide benchmarks. For the past 8 years, Fortune 500 companies have agreed to share common data from organizational climate surveys, so that CTS clients can benchmark with both the public and private sectors. Claim 3 - Possessing subject matter expertise in HR A. In fall 2002, the Society for Human Resources Management devoted an entire issue of its journal, Human Resources Management, to the work of the staff of CTS's predecessor organization, with the editor commenting on its world-class HR practices in the public sector. B. According to the CTS 2004 CSS, 98 percent of respondents agreed that CTS staff is knowledgeable, and 96% indicated that they would use CTS services again.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: CTS's program design has four potential issues. First, it might be argued that OPM has too much protection from market competition, at least in the short term. OPM's revolving fund authority has flexibility for CTS to operate on a cost-recovery model in that it requires CTS to "break even" over a three-year period. While such flexibility allows CTS to offer its services without having to raise its prices in a market downturn, one could argue that it would also make CTS less responsive to market changes or client demands. CTS contends that given the competition it faces with private companies, it must constantly evaluate and improve its services to maintain or expand its market share. For example, in the first full year of operation after OPM's 2003 restructuring, CTS did not reach its break-even point. But rather than taking advantage of the three-year window, CTS has already diagnosed the problems (e.g., skill mismatch of staff transferred from other parts of OPM), made recommendations, and started implementing some solutions (e.g., reducing space, proposing an internal reorganization). Second, a potential conflict of interest arises when OPM serves both as the problem identifier (through the Division for Human Capital Leadership and Merit System Accountability's (HCLMSA) agency audits or PMA scorecard evaluations) and as the solution seller (through CTS). Conceivably, OPM could purposely generate business for CTS by flagging problems at agencies and recommending CTS's products to address those concerns. However, several programmatic safeguards prevent the exercise or appearance of such inappropriate influence. First, HCLMSA and CTS are financially and structurally separated so that the former S&E-funded organization has no incentive to generate business for the latter reimbursable entity. Second, HCLMSA staff is fully aware of possible conflicts of interest, are trained to avoid even the appearance of them, and are careful to offer private-sector-sourced solutions alongside any OPM options. Third, because CTS offers services on an actual cost basis, it cannot do much for agencies that lack funding but are in dire need of HR products/services. At present, CTS is considering whether it could partially subsidize work for those agencies. Doing so would require CTS setting aside a small percentage of its revenues to create a fund for such underwriting. CTS is exploring whether this option is possible under its current authority. Fourth, through its association with OPM, CTS tends to incur sizable indirect costs (overhead, rent etc.) that are higher than those incurred by private companies. At present, CTS is examining two options to address these disproportionate indirect expenses. One option is to ally with one or more of the service centers, as defined under the Human Resources Line of Business (HR LOB) model. The hope is that such a center may better support CTS's entrepreneurial activities, by reducing its indirect costs, improving its financial information systems, and improving its service levels through more efficient administration. Another would be for CTS to negotiate a service-level agreement (services provided with standards and costs for those services) with OPM to achieve the same effect.

Evidence: OPM is charged to lead the HR LOB project under the PMA e-Gov initiative. Earlier this year OPM submitted a business case justification for OMB's review. The objective of the HR LOB is to create a framework for Governmentwide, modern, cost-effective, standardized, and interoperable HR solutions to support the strategic management of human capital. The HR LOB initiative will establish shared Service Centers to provide technology solutions to support multiple agencies with HR management and back office activities. Multiple Service Centers will be established to leverage economies of scale, reduce costs, and increase the quality and consistency of services provided. The services currently provided by CTS??or ones that could be provided by CTS??cover 85% (29 out of 34) of the processes identified in the HR LOB business reference model that could be outsourced. The selection process will focus first on the Federal service centers, followed by private-sector competitions. At this point, it is sensible for CTS to explore how it would operate in the HR LOB environment and to be part of the HR LOB discussions.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: CTS's resources are reaching intended beneficiaries. It primarily serves Federal agencies but also some state and local entities. Since agencies reimburse OPM for services requested and received, and the prices for CTS products/services are supposed to reflect the real cost of operation, there are no unintended beneficiaries. In addition, CTS targets the right beneficiaries because its clients purchase services that they must (by law) either provide for themselves or purchase elsewhere. Further, CTS does not provide products and services unless it is compensated. For more effective marketing, CTS has improved its website (www.opm.gov/Products_and_Services) and developed a marketing brochure that is distributed at Federal conferences and to interested agencies. Also, CTS has periodically provided OPM's human capital officers with updates about its projects with various agencies.

Evidence: CTS's intended beneficiaries are paying customers (primarily Federal agencies but occasionally state and local entities), through which CTS's products/services reach employees and, in some cases, the public. See three examples below: ?? Beneficiaries of USAJOBS: CTS is running a Governmentwide program for 3,000 Federal agency recruiters worldwide and all U.S. citizen job seekers. There are 15,000 to 17,000 job postings available daily, and USAJOBS logs in excess of 300,000 daily visits by job seekers each weekday (over 1 million job searches per week) and 2,000 new resumes per day. A Nielsen report cites USAJOBS as one of the top five job sites on the Internet. ?? Beneficiaries of Technical Services: Over 90% of non-DoD Federal agencies use Employee Express, covering more than 840,000 employees. Additionally, 100% of Federal annuitants, 2.3 million civilian retirees and annuitants have access to Services On-line, and 100% of DoD civilian employees, military service members, and military annuitants (5.5 million) have access to MyPay. ?? Beneficiaries of Nationwide Testing Services: CTS's service provides 95% of military entrance tests to prospective recruits. Additionally, it delivers entry-level and promotional tests for civilian positions with many Federal agencies, principally DHS. To develop and market its products/services for the right beneficiaries, CTS does the following: ?? CTS regularly convenes stakeholder and partner meetings, including agency HR groups for Employee Express (monthly) and USAJOBS (as needed), a USA Staffing user group (quarterly), and a Performance America network for agency survey users of OAS, CSS, and OPM Leadership 360 (annually). ?? CTS issues a quarterly update to inform HCLMSA staff of its latest work with client agencies.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: CTS's seven long-term measures reflect its purpose to provide Federal agencies with relevant, cost-effective reimbursable human capital products and services needed to build a high-quality workforce and become high-performance organizations. Of the seven measures, three focus on cost-effectiveness of CTS products/services, and four concern the impacts of CTS products and services on client agencies' workforce quality and organizational performance. The three efficiency measures include: (A) full cost recovery over 3 years, which examines a ratio of total earned revenue to all direct and indirect costs over a 3-year time period; (B) total earned revenue in a given year; and (C) customers' perceptions of whether CTS is providing good monetary value. The remaining four measures concern high-performing organizations and high-quality workforce. Given the difficulty of obtaining a stable, long-term measure of impact on organizational effectiveness and workforce quality. CTS uses four intermediate outcome measures based on the results of its Customer Satisfaction Survey (CSS), Federal Human Capital Survey (FHCS), and the American Customer Satisfaction Index (ACSI) for USAJOBS. Using multiple indirect outcome measures could reveal trends that indicate the impact of CTS's products and services. The four measures are: (1) customer views on CTS's contribution to organizational effectiveness; (2) customer satisfaction with the quality of candidates (whom CTS recruited and screened); (3) customer opinions on workforce knowledge and skills (after using CTS competency-based assessment products/services); and (4) USAJOBS users' satisfaction with Federal application process.

Evidence: Three of the seven long-term measures described above have been reported in OPM's Performance and Accountability Report (PAR). CTS is willing to report all seven measures in OPM's future PAR and/or budget documents. All of the seven measures have 2004 baseline information, the first full year after CTS was created in mid-March 2003. Whenever possible, cross-sectional as well as longitudinal comparisons are used, including private-sector comparisons. Established in 1994, the American Customer Satisfaction (ACSI) tracks trends in customer satisfaction and provides benchmarking insights of the consumer economy for companies, industry trade associations, and government agencies. The ACSI reports scores on a 0 to 100 scale at the national level. It measures 10 economic sectors, 41 industries (including e-commerce and e-business), and more than 200 companies and federal or local government agencies. Data are collected at the individual customer level, with scores for a company's customers aggregated to produce the company-level results. The score for a particular industry consists of an average of its company scores, weighted by the revenues of the companies included. Sector scores consist of industry scores, weighted by industry revenues. See performance measure tabs for selected measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: CTS's targets and timeframes for its long-term measures are realistic yet ambitious. CTS reports that the improvement targets are set empirically, based on past trends. The targets also require continuous improvement or maintenance of already high levels of customer satisfaction. Using either internal longitudinal or external cross-sectional comparison with the Federal governmentwide results or private industry, CTS also aims to exceed those comparative values. Of the seven long-term measures, it may be most challenging for CTS to achieve the two financial targets. CTS was created in FY03 as part of OPM's reorganization separating policy and HR leadership work from reimbursable work. (Prior to 2003, CTS's predecessor organization received both annual appropriations and fees collected from other agencies through sales of HR products/services.) In FY04, CTS became a 100% fee-for-service entity. In that year, CTS's total earned revenue did not cover all of its operating costs, thus reducing its cumulative retained earnings. CTS aims to achieve full cost recovery by FY07. CTS also aims to secure additional cumulative retained earnings by FY09, which CTS considers is sufficient to cover annual business fluctuations and potential shutdown costs of the program. (CTS's specific strategies for achieving these financial targets are further discussed in section 2.8.) Three customer satisfaction measures listed in section 2.1??the customer survey-based measures of (C) Providing Good Value to CTS Customers, (D) CTS's Contribution to Organizational Effectiveness, and (E) Satisfaction with Candidate Quality??already show very positive results. The 2009 targets are set at 95% (C) or 90% (E) level of satisfaction, which would require continuous improvement of CTS's services. Measure (F), Workforce Skills, is derived from the Federal Human Capital Survey (analyzed by CTS customer agencies that received competency-based recruitment/assessment-related services). This measure has increased from 73% positive for CTS customers in 2002 to 77% in 2004; in both years, CTS's scores exceeded the Governmentwide average (71% positive in 2002 and 74% in 2004). CTS aims to maintain a statistically significant difference between its positive score and that of all Federal agencies, and it has set a 2009 target of 78% positive. Finally, increasing Measure (G), USAJOB Users' Satisfaction with the Federal Employment Application Process, from 56% in 2004 to 61% in 2009 could be challenging as well. Making the Federal application process more user-friendly will require continuous improvement efforts not only by CTS but also by other parts of OPM (HCLMSA and Strategic Human Resources Policy) and other Federal agencies.

Evidence: See measures tabs for selected performance measures.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: For each of CTS's seven long-term measures, one to four annual measures are established and monitored. All sixteen annual measures together would provide a comprehensive picture of if and how CTS becomes cost-effective and its products/services contribute to improving client organizations' performance and workforce skills. The long-term Cost Efficiency goals are tracked through fiscal as well as customer satisfaction indicators. Three fiscal indicators are: (1) Earned revenue per full time equivalent (FTE); (2) Ratio of direct costs to earned revenue; and (3) Ratio of retained earnings to earned revenue. Five measures gauging customers' satisfaction with relevancy and monetary value of CTS products/services include: (1) customer perspectives about CTS's services meeting their needs; (2) customer views on quality; and (3) timeliness of CTS products/services; (4) intention to repurchase from CTS; and (5) overall satisfaction. To assess CTS Contribution to Organizational Effectiveness, CTS compares grand means (an average of 17 dimension means) on high-performance dimensions in the Organizational Assessment Survey from two groups of its clients - one-time customers who purchased the OAS instrument and repeat customers who worked with CTS to implement the OAS results. CTS also tracks an annual contract value of CTS products/services related to organizational performance improvements, including organizational development (OD). Three drivers for Satisfaction with Candidate Quality are tracked: (1) Percentage of high-quality eligibles out of all qualified applicants; (2) Percentage of vacancies that attract 10 or more qualified applicants; and (3) examining support timeliness. The higher proportion of high-quality eligibles may indicate successful targeting of applicants (based on its experience, CTS defines those scoring 85 or greater on an assessment as "high quality eligibles"). Also, from its experience, CTS concludes that vacancies with fewer than 10 qualified applicants are less likely to provide sufficient numbers of quality candidates. Examining support timeliness, measured in days spent on preparing and posting the vacancy announcement and applicant screening, is monitored, since untimely hiring processes are likely to result in loss of quality candidates. Workforce Skills are monitored through measuring: (1) Satisfaction with quality of training services; and (2) contract value of competency-based assessments. An indicator for USAJOBS users' satisfaction with the Federal employment application process is user satisfaction with the USAJOBS website. Increased satisfaction with the USAJOBS website could increase the number of applicants for Federal jobs, which could then increase the likelihood of attracting more qualified applicants.

Evidence: Many annual measures that CTS uses are intermediate, rather than ultimate, outcome measures. Using multiple indirect outcome measures should allow trends to become apparent that will provide an indication of the impact of CTS's work. About a third of the measures (6/17 or 35%) are customer satisfaction type measures. Customer satisfaction is an important measure because there is a body of research linking it to ultimate outcomes, such as financial results and organizational effectiveness. ?? Research published in the Harvard Business Review (Rucci, Kearn, and Quinn 1998) has shown that increased customer satisfaction leads to increased repurchasing and higher revenue. A study conducted for Sears found that a 1.3 unit increase in customer satisfaction led to a 0.5% increase in overall revenue growth. This translated into $200 million additional annual revenue. ?? The same article also demonstrated that employee satisfaction/morale and customer satisfaction are strongly and positively linked. A 5 unit increase in employee satisfaction led to a 1.3 unit increase in customer satisfaction. ?? CTS replicated this linkage research with DFAS and found a strong link between employee satisfaction and customer satisfaction (r=.67). In addition, some of CTS's annual measures are drawn from individual competency research, which shows that competency-based approaches to employee selection can add measurable value to an organization. Spencer and Morrow (Spencer, 1997) found that implementation of competency-based employee selection procedures led to substantial improvement in overall job performance (5 to 25% in non-sales jobs; 30 to 60% in sales jobs). Morrow, Jarrett, and Rupinski (1997) showed that competency-based approaches to training and performance management also led to improvement in overall job performance (11% to 30% in non-sales jobs; 30% to 60% in sales jobs). Hunter, Schmidt, and Judiesch (1990) found that organizations can realize economic gains by adding superior (vs. average) performers to the workforce. Their research demonstrated that superior performance is directly linked to increases in productivity (19% to 48% in non-sales jobs; 48% to 120% in sales jobs). The costs of poor selections were cited by the HayGroup (2003): 1. Dollars wasted in training and development 2. Low productivity and work quality of poor performers 3. Lost opportunities (e.g., not closing a deal, poor interaction with clients) 4. Poor morale among staff members who have to pick up the slack 5. Dissatisfied customers HayGroup, Inc (2003). Working paper. HayGroup, Inc. Hunter, J. E., Schmidt, F. L., and Judiesch, M. K. (1990). Individual differences in output variability as a function of job complexity. Journal of Applied Psychology, 75(1), 28-42. Morrow, C., Jarrett, C. Q., and Rupinski, N. T. (1997). An investigation of the effects and economic utility of corporate-wide training. Personnel Psychology, 50(1), 91-119. Schmidt, F. L., and Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274. Spencer, L. M., Jr. (1997). Competency assessment methods. In L. J. Bassi and D. Russ (Eds.), What Works: Assessment, Development, and Measurement (pp. 1-36). Alexandria, VA: American Society for Training & Development.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Because CTS was established in its current form in mid-2003, CTS uses 2004 information as the baseline for all of its annual measures. CTS's annual targets are ambitious, in that they require improving current performance levels or maintaining already high levels of customer satisfaction. Also, where available, CTS uses cross-sectional comparisons from other Federal agencies or the private sector, and strives to exceed those comparative values. Finally, CTS continues to refine or enhance its measures to capture as many direct (rather than indirect) outcomes as possible. Below are some highlights of CTS's ambitious targets. First, given CTS's current finances and its long-term goal of securing sufficient retained earnings for a "safety net," three cost- efficiency targets aim for a sharp turnaround from increasing losses to decreasing losses and then to positive retained earnings in FY07. Second, five measures of customers' satisfaction with the relevance and monetary value of CTS products/services show very positive results (94% or greater level of customer satisfaction). CTS attempts to improve or maintain this high level of customer satisfaction each year, with a goal of ensuring that all five measures reach or exceed 95% satisfaction. In the case of Overall Customer Satisfaction (OCS), CTS has particularly ambitious targets. The OCS is an equivalent measure of American Customer Satisfaction Index. In 2004, CTS's OCS was already high at 84, compared to 74 for the private sector in 2004; the 2006 target of 86 would be 12 points above the private-sector benchmark. Third, user satisfaction with the USAJOBS website is targeted at 80% positive for 2006. This index score, ASCI on OPM's e-Government, rose to 76 in 2004 from 71 in 2003, exceeding the Governmentwide e-ACSI score of 72 and the e-business ACSI for the private sector of 72.5. The target of 80 is likely to be well above the private-sector average. Fourth, customer satisfaction with the quality of training services is already high (99% in 2003 and 2004). Maintaining this level requires a continuous focus on customer service and quality. Recently CTS has decided to strongly encourage agency clients to include direct measures of learning in their statements of work for the development of training products.

Evidence: As for financial efficiency measures, CTS reports that while it has no information to compare itself with other reimbursable organizations in the Federal Government, it found information for cash-after-operations (sales less operations costs analogous to retained earnings expressed as a percentage of sales) for private companies classified in NAICS 541612 Human Resources and Executive Search Consulting Services. The range for 2003/2004 was from -8.0% for the lowest quartile to +5.8% for the highest quartile. CTS's 2007 target could put CTS close to the highest quartile. From random-digit-dial (RDD) telephone samples (and Internet samples for e-commerce and e-business), more than 65,000 consumers are identified and interviewed annually. Since the baseline study in 1994, a sample has been amassed of more than 700,000 respondents. According to ACSI researchers, the ACSI model is a set of causal equations that link customer expectations, perceived quality, and perceived value to customer satisfaction (ACSI). In turn, satisfaction is linked to consequences as defined by customer complaints and customer loyalty - measured by price tolerance and customer retention. For most companies, repeat customers are major contributors to profit. Thus, customer retention is a key to financial performance. By combining it with certain financial data, ACSI corporate subscribers are able to calculate the net present value of their company's customer base as an asset over time. See measures tab for selected targets and timeframes.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: A majority of CTS's work (as defined by revenues) is performed by contractors. CTS personnel exercise stringent oversight over all contractors. For example, Monster Government Solutions, the contractor for USAJOBS website, is constantly monitored through its daily submission of performance metrics. Furthermore, in all of its contracts with significant partners, CTS includes clauses concerning performance expectations and remedies if expectations are not be met. Because CTS project managers exercise strong oversight and because most partners share a stake in financial and performance outcomes, these clauses rarely need to be invoked. Other CTS partners include OPM's HCLMSA, SHRP, and HRPS's Center for Leadership Capacity Services. CTS often works with these groups to keep abreast of the latest HR policy developments so that it can provide relevant HR products/services and build high-performing organizations and a high-quality workforce. CTS also informs these partners about projects it is working on with other Federal agencies, so that other parts of OPM are fully aware of Federal agencies' needs and challenges and the ways in which CTS is assisting agencies in meeting their HC objectives.

Evidence: The sample contracts submitted by CTS all contain detailed performance clauses, supporting the agency's assertion that this is standard practice. CTS can also provide customer satisfaction survey results for each line of its business, and it uses this information to provide contractors with client feedback. Up to this point, CTS has not encountered any problems with contractor performance that have threatened CTS's ability to achieve its long-term goals.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: To date, CTS has not undergone any systematic independent evaluation, such as one with an experimental design and a randomized sample. Arguably, because CTS is a 100% fee-for-service organization, its customers ultimately perform the most significant evaluation of CTS's products/services when they choose CTS over its many HR service competitors. Nevertheless, CTS has carried out a number of different assessments of its cost and operational efficiency. First, after every major contract (those valued $20,000 or more), CTS conducts a Customer Satisfaction Survey. It uses the survey results to create and monitor constantly many long-term and annual satisfaction measures. For example, as mentioned earlier, CTS generates a score equivalent to the ACSI and tracks its performance. Current scores compare well with both CTS' past performance and with private-sector comparators. Second, in the past, CTS and its predecessor organizations won three A-76 competitions against the private sector. These areas were chosen under OPM's OMB-approved Competitive Sourcing Plan. CTS won all three by having a lower cost than comparable private-sector providers. Pending finalization of a proposed reorganization, CTS will compete many of its positions. Third, in 2004, CTS contracted with KPMG to conduct a Statement on Auditing Standards (SAS 70) test of the operational effectiveness of one of its IT systems (Employee Express program). A SAS 70 audit is widely recognized because it demonstrates that a service organization has been through an in-depth audit of its protection of data belonging to other agencies but hosted and processed by the service organization. The audit result indicated that CTS's Employee Express program has satisfactory controls and safeguards. In the coming months, CTS will develop a 3-year external evaluation plan. Since CTS is responsible for a wide range of business lines covering the entire HR cycle, no single evaluation would be adequate. The plan envisions different types of external evaluators for the various business lines. For example, a SAS 70 review will be conducted of all IT business lines: USAStaffing, Employee Express (first review conducted in 2004), and, if possible, USAJOBS technical operations by Monster Government Solutions. CTS will arrange for external review of ECSG and ATAS by a not-for-profit entity such as the Merit Systems Protection Board or a school of public administration. In addition, an overall assessment of CTS as an integrated provider of HR products and services is planned. Among other areas, that assessment will cover CTS's business operation and marketing for HR products and services.

Evidence: ?? CTS has detailed information on customer satisfaction survey results and is capable of conducting various analyses. ?? The KPMG SAS 70 audit result is available for review. ?? CTS is committed to developing a comprehensive external evaluation plan in coming months; CTS will share a draft plan with OMB by mid-August.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: CTS is a 100% reimbursable organization (other than USAJOBS, which will become 100% reimbursable in FY06). Without congressional funding, all of CTS's resources come through reimbursements from its customers. CTS currently sets its budget levels based upon past performance and a reasonable projection of future costs and customer demand for its products and services. CTS's financial measures and targets help drive these projections. In addition, CTS budgets for all direct costs (salary, benefits, travel, contractor costs, etc.) and indirect costs (OPM's common service/IT charges, administrative support, management salaries, etc.); and CTS reports performance data. The revenue, cost, and performance information is used by CTS to improve program effectiveness and efficiency in achieving performance targets, evaluating managers, preparing all budget-related reports (CBJ, PAR, operating budget and quarterly reports to OPM's CFO, etc.), and setting new performance targets. CTS's budget documents contain, however, very little information on indirect costs. CTS's indirect costs include common service charges allocated by Office of Chief Financial Officer. Common service charges cover costs incurred by the Management Services Division, Office of the Chief Financial Officer (OCFO), Office of the Director, Office of the General Counsel, Office of Congressional Relations, and Office of Communications and Public Liaison, and these costs are allocated to OPM's Salaries and Expenses, Trust Funds, Revolving Fund accounts, and the Office of Inspector General. OPM's CFO reports that it bases the allocation of common service charges on the historical percentages of common services usage by each of these programs. The Revolving Fund share is approximately 35% but may vary slightly from year-to-year. The specific portion allocated to each CTS business line is determined based upon the same allocation methods used by OCFO. CTS reports that the allocation of current common services is being reassessed by OCFO, which plans to develop a methodology that is more equitable. At this point, due to a lack of information available on OPM's historical trends, calculation methodology, or allocation formula of common service charges, OMB is unable to assess why CTS's indirect costs are what they are.

Evidence: Since CTS is fully financed by fees collected in exchange of services, CTS's annual budget "request" in OPM's budget justification documents to Congress or OMB represents a maximum amount of revenue CTS can bring in for a fiscal year. (This practice is adopted by OPM OGC's interpretation of authorizing statutes.) CTS's performance plans in the budget documents reflect the agency's strategic goals, the resources necessary to accomplish these goals, various financial and non-financial performance measures, and activities performed by CTS in support of the goals. Performance indicators per OPM's 2006 CBJ include the following types of measures and longitudinal results: financial (full cost-recovery), efficiency (USAJOBS metrics), and effectiveness (customer satisfaction, CTS's contribution to improving agency organizational effectiveness, repurchase intention, customer satisfaction with USAJOBS, and customer satisfaction with Federal application process). As mentioned earlier, no evidence has been presented to OMB regarding OPM's analytic methodology of calculating/allocating common services charges.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Following OPM's restructuring in 2003, CTS extensively revised its five-year strategic plan and created a 2004 Business Plan, which CTS has updated in 2005. CTS has already taken steps to mitigate its projected fiscal shortfall, including increasing prices for its products and services where possible, improving management control, hiring a dedicated business development person (to be shared with CLCS), undertaking a targeted marketing campaign and increasing attention to performance measures (financial, quality, and customer satisfaction). (1) Expenditures in the Examining and Consulting Services and Testing business lines exceeded their earned revenue in 2004 due to a combination of factors??the elimination of appropriated funds, unanticipated increases in operating overhead (OPM's common service and GSA space charges), excess staff capacity, and less-than-planned reimbursable sales and delivery. A restructuring plan has been proposed to cut staff levels and costs, realign selected functional units, rationalize organizational structure, optimize employee competency base, adjust pricing, and examine pricing methodology. If implemented, the results should make the lines viable and result in full cost recovery. (2) The Technical Services Group (TSG) also had expenditures significantly exceed its earned revenue in FY04, and consequently steps have been taken to reduce staff and streamline operations. Also, as part of the CTS restructuring effort, CTS is proposing to merge TSG with OPM's Center for Information Services and Chief Information Officer (CISCIO). The proposed consolidation would bring OPM more in line with the spirit of the Clinger-Cohen Act of 1996 (which mandates that agency CIO's monitor all IT spending and ensure interoperability), reducing redundancies and increasing the technical capability of the CISCIO. (3) As referred to earlier, as a result of the reorganization, not all staff assigned to CTS have the competencies required to optimize CTS performance. Therefore, Individual Core Competency Development plans (ICCDPs) have been developed for each staff member. As part of this effort, some staff may be reassigned to positions that better fit their existing competencies (workforce realignment). (4) When CTS received appropriated funds in the past, it was able to partially fund work for internal OPM projects. As a fully reimbursable organization, it is no longer able to do so, and it has instituted a policy that an agreement must be signed for all work (including internal OPM projects) with prices set at levels that ensure that all costs (direct and indirect) are recovered. The result is that there are no unaccounted-for outflows of funds. (5) CTS believes it has a unique offering of integrated solutions for agencies to address HC problems. One challenge is to properly educate all agencies on what CTS has to offer and how it can help them build a high-quality Federal workforce and increase organizational performance. CTS is exploring different avenues to address this challenge, including having recently hired the previously mentioned business development staff person and exploring partnership opportunities with proposed HR LOB service centers. The added sales and marketing capacity should result in CTS reaching more agencies.

Evidence: CTS's strategic planning included (1) refining the short- and long-term goals so that they focus adequately on outcomes, (2) integrating all goals and measures with CTS-wide reporting, and (3) ensuring that work reporting and tracking systems generate data needed to measure performance against goals. CTS has recently awarded a contract to provide additional budget and performance integration support services. Other evidence of the meaningful steps CTS has taken to bolster its strategic planning includes the recent production of a marketing brochure and hiring the aforementioned new senior manager for marketing and business development.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: CTS regularly collects timely and credible performance data related to program goals from its key program partners: its customers and its contractors. As discussed earlier, CTS is able to assess the performance of USAJOBS by monitoring the American Customer Satisfaction Index (ACSI). The ACSI, which measures customer satisfaction in a variety of industries, rates how satisfied customers are with job posting tools offered by various private-sector firms and USAJOBS. CTS receives USAJOBS ACSI results on a quarterly basis, along with industry-wide benchmarks. For all other business lines, CTS tracks its performance by (1) closely monitoring contractor products and services for compliance with performance-based contract clauses and (2) soliciting suggestions for improvements and future needs through its Customer Satisfaction Survey (CSS). CTS collects its CSS information in the following way. For all products and services in amounts exceeding $20,000, branch managers at CTS forward the survey to customers upon delivery. Customers may then return their completed surveys directly to a GPRA coordinator outside of CTS to ensure that customers can provide immediate feedback without fear of influence by those who delivered the products or services. (Customer response rate for 2004 was 42%. CTS is trying to increase the response rate.) The coordinator enters survey information into a CTS-wide database, and copies of each completed survey are made available to CTS managers. Thus, CTS continuously monitors customer feedback, which allows CTS management to address issues immediately by bringing them to the attention of staff. In addition, CTS research psychologists compile CSS results and formally provide them every six months to CTS managers and supervisors, who use this information to make program adjustments to improve customer satisfaction. CTS also carries out closeout sessions with customers at the end of each project to review what went well and what could be improved. Financial performance data on all CTS projects and contracts are collected and analyzed through CTS's own internal tracking systems (so-called CTS "cuff" systems). Information is collected in real time and used as the basis for ensuring project cost recovery, setting annual revenue and cost targets, measuring specific project outcomes, and prioritizing allocation of resources. Finally, CTS communicates with its clients and other stakeholders through client user groups, focus groups, and contacts made in person or by phone or email. In these ways, CTS identifies needs for new services or IT product features.

Evidence: Stakeholder Communication ?? CTS regularly convenes stakeholder and partner meetings, including agency HR groups for Employee Express (monthly) and USAJOBS (as needed), a USA Staffing user group (quarterly), and a Performance America network for agency survey users of the CSS, and Organizational Assessment Survey (OAS). Several examples show how CSS data and other feedback have led to CTS management actions or improvements in performance: ?? USAJOBS: Feedback has resulted in continuous system improvements, including job search revisions, design changes, the addition of a site map, resume builder streamlining, job announcement builder and display changes, and job search results list changes. ?? Technical Services: Feedback on the 2004 CSS triggered the Technical Services Group to follow up with a customer regarding problems with product usability. In response to the customer's comments, CTS improved the product and retained the customer, and in the FY05 CSS the customer provided a fully satisfactory rating on the product. ?? Nationwide Testing Service: A recruitment branding project with the Animal and Plant Health Inspection Service (APHIS) came about when CTS staff visited APHIS and learned that many applicants do not show up for an entry test shortly after sending in their application forms. CTS suggested that APHIS revamp its job announcement to highlight the many benefits of working for APHIS, and also suggested that before each test, APHIS should mail registrants a letter or glossy brochure underscoring the agency's value and career growth potential. APHIS has since allocated funds for these branding tools and is working with a CTS team to build them. ?? Assessment Services: Feedback from users of the OPM Leadership 360 product led to the automation of the product administration, and feedback from OAS and CSS customers led to developing automated content analysis of survey comments, which costs significantly less than the non-automated approach.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: CTS program managers are held accountable for cost, schedule, and performance results through the annual performance review, mid-year performance review, and quarterly financial reviews. Managers in turn hold their employees accountable through their performance plans for providing work products that meet standards for quantity, quality, and timeliness, as well as customer service and other competency-based criteria. Incentives for performance of program partners are built into contracts, and CTS continuously monitors partner performance using project plans that outline deliverables, delivery dates and milestones, and associated costs. CTS also ensures contractor accountability at the task-order level, through continuous review of performance quality, timeliness, and cost of deliverables.

Evidence: ?? Managerial Performance Agreements include the SES Standard for Excellence (including the five executive core qualifications) and fulfillment of annual performance plans, financial targets, and/or annual work plans as key elements on which CTS managers and supervisors are evaluated. ?? As mentioned earlier, sample contracts provided by CTS all contain performance clauses and detailed project plans and deliverables along with delivery dates. ?? CTS managers and staff use a number of reports from OPM's central financial and payroll IT systems, such as Summary Reimbursement Reports which present revenues and expenses by initiative and project/contract); and year-to-date payroll. To provide a comprehensive status and accounting of a particular project, business line, or CTS as a whole, managers and staff combine the data from various reports to produce customized reports.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: OPM's Government Financial Information System (GFIS) is a transaction-driven financial management system that permits budgetary transactions to be recorded in a timely manner. CTS funds are obligated and accounted for in GFIS once program staff obligate/expend funds for each business line's individual contracts and/or projects after checking the relevant legal authority, contractual terms, the OPM Strategic Plan, and associated annual performance plans. A typical CTS transaction would be processed as follows: 1. CTS enters into a contract with a client agency, typically using an Interagency Agency Agreement (IAA). Upon signatures by both the customer and OPM, the agreement is assigned a project code (to track revenues and expenses) and entered into GFIS. 2. Next, a project plan is created, complete with expected revenues and expenses. If a contractor is to be used, CTS issues a task order to the selected vendor, obligating funds against its customer agreement funding. 3. CTS personnel complete the work, and then record their time expended in working on agreement tasks on their time cards against the appropriate project code. 4. CTS personnel salary cost is categorized as a project expense, along with an "overhead" charge. Reports from GFIS give CTS financial and performance data in real time. These are supplemented by labor reports from the GSA website (http://finance-kc.gsa.gov/labor), Protrac (TMA), and CTS sub-program cuff records. These systems allow CTS to track expenses against agreement funding, and thereby to ensure that expenses are incurred consistent with the overall project plan and each project contributes to earned revenue targets. In this way, CTS can create schedules for obligations that correspond to the project plan, obligate program funds consistent with the overall program plan, and compare actual expenses to actual funding levels. Both CTS and Office of Chief Financial Officer (OCFO) analyze GFIS transactions daily to identify and resolve any problems with funds that are being obligated and expensed against the seven CTS business lines. OCFO staff post any required corrective entries in GFIS, ensuring that the financial information received by management is complete and reliable at any point in time. While there exist certain long-term problems for reconciling the Revolving Fund (RF) balances with related balances maintained by the Department of Treasury, data and staff analyses indicate that the impact of the RF problems on CTS operations is "minimal": it turns out that most of the discrepancy relates not to CTS, but to other RF programs.

Evidence: ?? To date, CTS has not heard any complaints from contractors or vendors not receiving their payments from OPM in a timely manner. ?? While there are a number of issues with GFIS, CTS staff reports that fund obligation is not a problem. To CTS staff, financial reports generated by GFIS are problematic because they do not have all the information that is useful to CTS business managers or they are not presented in user-friendly ways.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: CTS routinely uses a number of performance measures concerning cost-effectiveness and timeliness of processes. For example, it monitors earned revenue per FTE, the direct cost/earned revenue ratio, ratings of value-for-money and service timeliness, and the extent to which it achieves full cost recovery. As a fee-for-service organization, CTS also monitors competitor costs regularly to ensure its products and services remain competitive; and it regularly reviews its pricing models and practices. In addition, CTS monitors project costs on a project-by-project basis and ensures that it is achieving full-cost recovery. Once projects begin, CTS managers receive and/or compile regular reports of expenses against these projects to ensure that costs do not exceed projections. CTS also advises customers periodically of the status of contract and fund activities. Furthermore, to ensure efficiency, CTS makes use of competitive sourcing, performance-based contracting, and management performance agreements, whenever necessary and appropriate. As discussed earlier, over the past three years, three units of CTS have undergone A-76 competitions; CTS adds strong performance clauses to each of its business contracts; and CTS managerial performance agreements include financial targets as key elements on which managers and supervisors are evaluated.

Evidence: ?? Competitive sourcing plans (Copies of RFPs provided upon request) ?? Performance-based contract statements. ?? Managerial Performance Agreements include financial targets as key elements on which CTS managers and supervisors are evaluated. ?? CTS customer satisfaction surveys results

Does the program collaborate and coordinate effectively with related programs?

Explanation: Of seven lines of CTS business, "related programs," as defined in the PART Guidance, are scarce. Most of CTS services have competitors in the private sector. To improve its products and services, however, CTS regularly collaborates with various policy and program offices in OPM (SHRP, HCLMSA, CLCS) and with its program partners, Federal consortiums, and groups in other agencies. Here are three examples of this collaboration. 1) CTS has collaborated with a consortium of Federal agencies to offer a self-service payroll and benefits system called Employee Express. The consortium was initially formed by 17 Federal agencies, and it covered 120,000 Federal employees during a 6-month pilot; it has now grown to include more than 50 Federal agencies with more than 1 million employees. This employee self-service approach has freed agency HR personnel from routine data entry tasks, allowing them to handle more value-added tasks and to better serve their employees. Employee Express has also improved employee satisfaction by empowering employees to control their payroll and personnel information. 2) CTS collaborates with CLCS to support the Presidential Management Fellows (PMF) Program. CTS has designed the assessments and provided program support for the PMF program for eight years. The system has evolved from the paper-based application developed in 1996, and from the program's early, primarily informational website. The current PMF IT system supports: online application acceptance and processing; nomination coordinator approval of candidates; oral and written assessment processing; accomplishment review and rating; email notifications to finalists, non-finalists and agencies; posting of finalist resumes for agency review; and posting of job opportunities and rotational assignments for finalist review. These expanded functionalities benefit not only PMF applicants but also PMF program offices as well as hiring agencies. CTS continues to work with CLCS to update and streamline system functionality as the program evolves. 3) CTS worked together with HCLMSA and SHRP in conducting focus groups of agency HR personnel to assess satisfaction with OPM policies and services. The result was a comprehensive report that both groups used for internal discussion and as the basis for discussion at the OPM Senior Leadership Conference.

Evidence: While it is hard to find "related programs" to CTS, as defined in the PART Guidance, CTS's Training and Management Assistance unit is somewhat similar to GSA's Management, Organizational, and Business Improvement Services (MOBIS) in some sense. As discussed earlier, the TMA unit offers a list of vendors and manages a contract between a vendor and a client agency, whereas GSA makes a vendor list available but provides no or little contract management in the HR arena. It seems that the two different approaches would appeal to different customers - depending on how much project management/oversight needs a client agency has. Nevertheless, since CTS and GSA list many of the same vendors on their respective acquisition lists, it may be beneficial for CTS and GSA explore possible collaboration and coordination in the future.

Does the program use strong financial management practices?

Explanation: CTS uses OPM's GFIS, FEDDESK, the GSA website (http://finance-kc.gsa.gov/labor) and Financial Data Portal (FDP) to account for its revenues, expenditures, and obligations. GFIS is OPM's accounting software package that records all of CTS's project, revenue, and expense transactions. From FEDDESK and the GSA website, CTS managers use data and reports from the following applications: ?? Electronic Time and Attendance Management System. ?? Travel and Miscellaneous Employee Reimbursements. ?? Incentive award processing. ?? Labor distribution cost by CTS employee and project code. In addition, CTS frequently accesses reports from FDP (which draw GFIS data) regarding revenue and expense (by organization, by project, and/or by accounting code), customer activity, and trends. CTS uses all of these reports to monitor project profitability, labor hours billed to contracts or other overhead project codes, and to track its progress towards financial targets. They are available around the clock and have information up-to-date as of the previous day. Nevertheless, there are some issues with the major systems. First, FEDDESK and GFIS are not integrated, so getting certain reports in a timely manner can pose an issue. For instance, CTS manages are not able to obtain individual staff labor costs, other project costs, and project balances from one system in a user-friendly format. Individual labor costs are obtained through downloads of raw data from FEDDESK. Overall project costs and balances are obtained from GFIS. Managers then have to reconcile and compile the necessary information manually. Second, because some reports generated by GFIS suffer from weaknesses in data quality and usability, CTS managers need to maintain secondary (cuff) records to support day-to-day operations and ensure the integrity of GFIS information. In GFIS, some critical metrics are not readily available and accurately captured. For instance, there is no ability to segregate current year available funding balances; no report is available to distinguish direct versus indirect costs; and there is no readily available and reliable line-item report of revenue accruals and subsequent earned revenue to satisfactorily validate revenue numbers at sub-fund, project, and project level. Because of these gaps in GFIS, CTS managers have maintained "cuff records" for their respective business lines. CTS's TMA unit, for example, uses a program called Protrac, which gives it detailed information about all project expenditures and revenue collections. To use all the financial systems effectively and minimize financial risks, CTS routinely: ?? undertakes only projects that will enable full cost-recovery (unless approved by the agency); ?? recognizes revenues on a quarterly basis; ?? bills customers as appropriate, given the project agreement (e.g., quarterly for long-term projects, and on completion for short-term); ?? assigns a financial code for projects and charges costs to the appropriate codes; ?? spreads indirect costs across projects; ?? updates customers regularly on the financial status of projects; and ?? monitors collections regularly, reminding customers of payment problems. In FY05 CTS provided financial training for all of its managers to ensure that they had a common understanding of the CTS business model and financial principles important to achieving financial goals.

Evidence: CTS has submitted project revenue and expense reports, Protrac reports, and information about its financial training.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: As a reimbursable program, CTS must maintain constant vigilance over revenues and expenses. It is supported by OPM's CFO group and the Resource Management Office of the Human Resources Services and Products Division, but given the competing demands on these offices' resources, CTS has contracted with outside support to ensure that critical financial support is available, particularly in the area of collections and internal reporting. The result is more focused, timelier information used by management to ensure that problem areas are quickly identified and addressed. CTS also provided financial training to all CTS professionals so that they may better understand the information available to them. During FY05, OPM continued to improve its financial management, internal controls, operations, and reporting, particularly as they apply to the Revolving Fund (RF) business lines. OPM has tried to address the causal factors contributing to a material weakness identified and reported by KPMG during its FY04 financial audit of the Agency. The material weakness was related to internal control over financial reporting and its operations within the RF and Salaries and Expenses (S&E) accounts. The reported weakness primarily stems from long-standing unreconciled differences between OPM fund balances and comparable balances maintained by the Department of the Treasury. But based on corrective actions since taken by OPM, OPM's CFO fully expects that the independent audit of the Agency's FY05 consolidated financial statements will confirm that this problem has been resolved. To date, OPM's completed corrective actions include: ?? Implementing standard procedures for reconciling the RF and S&E fund balances ("cash") with related balances maintained by the Department of the Treasury. At the beginning of June 2005, OPM had reduced the number of unreconciled months from 33 (in September 2004) to 5 months for deposits, and from 26 to 15 months for disbursements. OPM's goal is to reduce both to 3 months so that there are no differences with Treasury for more than 90 days. ?? Liquidating $77 million of open receivables documents for all OPM programs, which dated back to the implementation of GFIS in FY01 and prior years, and the outstanding conversion balances from the GFIS' predecessor system. Thus, the amount of receivables reflected in OPM's financial system is now significantly lower and more accurate. ?? Reviewing and taking the necessary actions to de-obligate $266 million of open obligations from prior fiscal years. Thus, the financial data reflected in OPM's financial system for FY01 and FY02 is now significantly more accurate. ?? Developing and implementing regular status-of-funds and full-time equivalency management reports, which are to be distributed to program managers monthly and quarterly. Thus, program managers and OPM leadership are now receiving more timely, accurate, and reliable financial management information to support the decision-making process.

Evidence: ?? Training Information ?? Copy of Financial System Remediation Plan (Corrective Actions) for FY04 Audit Findings

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: CTS is a new organization, created in FY03 as part of OPM's reorganization, and it did not become 100% reimbursable until FY04 (the baseline year for all CTS measures). For this reason, it is somewhat difficult for CTS to demonstrate "adequate" progress on its long-term performance goals. Nonetheless, from the information available to date and anecdotal feedback from customers, CTS is on its way to achieving its long-term goals. For the cost-efficiency measures, there are indications that CTS will meet its target, although it is too early in the fiscal year to say for certain. In 2004, the first full year after CTS became 100% reimbursable, CTS's total earned revenue did not cover all of its operating cost. If current trends hold, CTS should be able to achieve full cost recovery by 2007 and to increase ratio of total earned revenue to total cost over time. As discussed in section 2.2, for most customer satisfaction measures (indirect measures of organizational performance and workforce quality), CTS had very positive baseline results in 2004. Based on the customer feedback that CTS has so far received, CTS does not expect any problem in meeting 2005 annual and long-term goals.

Evidence: See measures tab, attached measures chart and 2006 CBJ for long-term measures. ?? Links to new CTS marketing Web pages ?? Copies of new marketing brochures ?? Letters of commendation for Competency Assessment Branch ?? Articles related to value of competency-based assessment

Does the program (including program partners) achieve its annual performance goals?

Explanation: CTS has met all annual PAR goals. Program partners, primarily contractors, contributed to meeting the goals. CTS has 16 annual performance measures that drive the seven long-term measures. Since CTS is a new organization financially, it is too early to demonstrate through historical measures that CTS is meeting its financial targets. For most of the non-financial measures, longitudinal or cross-sectional comparisons are available, and these comparison show progress. All financial measures have baselines for 2004, the first year CTS was 100% reimbursable. Earlier financial measures do not offer appropriate comparisons, since funding was a mix of S&E and reimbursable money, and CTS's organizational structure was different. However, data available to date indicate that CTS is improving on all fiscal measures (such as Earned Revenue per FTE and Direct Costs/Earned Revenue) and that it is on the way to meeting its long-term full cost recovery target by 2007. For many non-fiscal measures on which CTS has been collecting results, there are trends toward higher customer satisfaction each year. For example, 96% of customer respondents said that they had received high-quality products in 2004, compared with 95% in 2003. Also, 94% of CTS customers in 2004 and 93% in 2003 agreed that CTS products and services are designed to meet customer needs, and are therefore relevant. Moreover, CTS customer satisfaction with quality of training services provided by TMA contractors, was 100% in both 2003 and 2004. For those measures that are newly adopted, CTS attempts to make progress toward the long-term goals. For instance, the percentage of all eligible candidates who were rated as being "high-quality" (defined as those having scores of at least 85), an indirect measure of CTS's impact on building high-quality workforce, was 41% in 2004. CTS aims to increase it to 45% by 2006. Another new measure, the percentage of vacancies that attract at least 10 qualified applicants, was 37% in 2004; CTS's goal is to increase that figure to 39% by 2006.

Evidence: See 2006 CBJ, measures tab and attached measures chart.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: CTS is achieving its annual goals for the most part. As noted earlier, CTS is a new organization financially and does not have a complete financial year to compare to its base year of FY04. This makes it difficult to demonstrate improved financial results over a prior year. The non-financial efficiency indicators show positive results and trends, as available. It is too early to assess results on efficiency measures Full Cost Recovery, Earned Revenue per FTE, and Earned Revenue/Direct Costs by comparing 2004 to 2005. As a reimbursable program, CTS must continually strive to recover its costs and provide its customers with good value. If the program were operating inefficiently, it would have to charge higher prices for its products and services to recover its additional costs. If this resulted in prices above the market, it would lose sales and market share, resulting in further losses. Therefore, if CTS is able to achieve full cost recovery, it is very likely that it is operating efficiently. By the end of FY05, CTS will still not reach a "break even" point, but its annual loss is expected to be much less than in FY04. If current trends hold, CTS should improve upon its goal of full cost recovery and reach its FY05 retained earnings target. Several efficiency measures - Providing Good Value to CTS customers (93% positive), Timeliness of CTS Products and Services (96% in 2004 vs. 95% in 2003), and USA Staffing Examining Support Timeliness (31 days) - all show highly positive results that would be difficult to improve on, indicating that customers already receive good value for their money and in a timely manner. CTS also won three A-76 competitions and is taking steps to increase its cost-effectiveness since becoming 100% reimbursable in 2004 (see earlier responses). IT-based products are continuously being improved to increase efficiency and meet customer needs. USAJOBS postings are significantly less costly compared to the private sector; see earlier response under 1.3 Evidence.

Evidence: See attached measures chart and 2006 CBJ and response to 1.3 Evidence. ?? 92% of CTS customers agree that CTS provides good value for the money (CSS #8) ?? USA Staffing migration to Internet (USA Staffing reports for FY03 vs. FY04 trends) ?? USAJOBS upgrade (Monster contract and FY03-04 changes in metrics) ?? OPM Leadership 360 automation

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ?? CTS is a new reimbursable program, established during OPM's reorganization in 2003, and stopped receiving appropriated funding in 2004 in order to become 100% reimbursable. ?? CTS is the only interagency one-stop shop for federal HR services. Its customer satisfaction survey includes the three items used in the ACSI for an equivalent index, and results exceed the Governmenwide index and private-industry comparison. ?? CTS won three A-76 competitions. ?? CTS's labor cost for comparable categories is lower than that of private-sector consultants. ?? Results from the CTS 2004 Customer Satisfaction Surveys (CSS) indicate: (1) CTS performance is superior to that of private-sector consultants who provide similar services, (2) CTS products and services are a good value for the money, (3) and customers intend to purchase CTS products and services again in the future. ?? CTS's staff is rated as highly knowledgeable. ?? GAO report cited TMA exemplary practices and cost savings from OAS. ?? The work of CTS psychologists is published in refereed professional journals and continues to result in the development of innovative assessment approaches.

Evidence: ?? General Accounting Office. (2004). Human Capital: Selected Agencies' Use of Alternative Service Delivery Options for Human Capital Activities. Washington, DC. The report cites OPM's TMA program as an effective example of using outside expertise to deliver training and human capital solutions (pp. 22-23). The report also cites savings of $300,000 by the U.S. Coast Guard from using the CTS Organizational Assessment Survey (OAS) (p. 14). ?? CTS won three A-76 competitions (nationwide testing, teleservice and transcription, and computer assistance) saving the Government millions of dollars over 5 years. ?? 90% of respondents in the 2004 CTS CSS stated that CTS contract performance is better than that of private-sector consultants who provide similar services (#10). ?? CTS labor rates are generally below those of its private-sector competitors (evidence submitted to OMB). ?? According to results of the CTS 2004 CSS, 92% agreed that CTS products and services are a good value for the money (#8) and 96% expressed repurchase intention (#11). ?? USAJOBS is the only consolidated Governmentwide job posting service of its kind, providing job seekers with one-stop access to all federal jobs. Its e-ACSI score exceeds the private sector e-business score (76 vs. 72.5). The CTS ACSI-equivalent result was 84, above the Government and industry averages of 72 and 74, respectively. ?? USA Staffing is the only Government Off-the-Shelf (GOTS) automated staffing system of its kind. USA Staffing is superior to other automated staffing because its functionalities provide the best and most diverse pool of applicants based on statute-mandated merit systems principles. Because OPM personnel specialists developed USA Staffing, its users??federal HR specialists ??find it more intuitive than software from private vendors. In the automated staffing market in which USA Staffing operates, one competitor has announced that it will not be supporting its product beyond February 2006, and the remaining large private-sector provider recently had technical problems forcing an interruption in service (since March) to some clients, some of whom switched to USA Staffing to post their job announcements. ?? CTS has promoted a common language Governmentwide with standardized assessment tools to reduce cost, avoid duplication of work, and maximize the benefit to the Government (HR Manager, OAS, CSS, OPM Leadership 360 ). Large private-sector companies who conduct surveys cannot offer Governmentwide benchmarking, and the numerous smaller consultant organizations come and go and/or frequently change their business models.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: CTS is a new organization that became 100% reimbursable in 2004. Currently, the independent evaluation of CTS comes from a variety of different sources: (1) Customer feedback from a survey benchmarked as a valid and reliable GPRA measure by GAO. The instrument assesses nine service quality dimensions: Access, Courtesy, Knowledge, Timeliness, Reliability, Choice, Tangibles, Recovery, and Quality of specific business line products and services. The instrument also provides an equivalent of the ACSI, a private-industry standard. (2) Sales volume for CTS's seven business lines. (3) Governmentwide and private-sector comparisons of USAJOBS, using a variety of measures, including financial and customer satisfaction for e-business enterprises. (4) Three successful A-76 competitions for Customer Service Representative positions and Computer Assistant positions at TSG, and for the Nationwide Testing program. (5) A SAS 70 evaluation on the key internal controls placed in operation and test of operational effectiveness related to the Employee Express program. (6) A KPMG review of TMA's fee structure. (7) Citations of CTS's work in a GAO report as appropriate practices for reimbursable business in Government. (8) Peer review of the work of CTS's personnel research psychologists who design innovative assessment approaches.

Evidence: ?? 90% of respondents in the 2004 CSS stated that CTS contract performance is better than that of private-sector consultants who provide similar services (#10). ?? CTS labor rates are generally below those of its private-sector competitors (evidence submitted to OMB). ?? According to results of the CTS 2004 CSS, 92% agreed that CTS products and services are a good value for the money (#8) and 96% expressed repurchase intention (#11). ?? According to results of the CTS 2004 CSS, 98% agreed that CTS staff is knowledgeable (# 6b). ?? In CTS's 2004 CSS, 95% agreed that CTS services contribute to improving organizational effectiveness (Item # 13a, excluding "don't know" responses). ?? Using the ACSI E-Government Web Site Index, CTS has been tracking USAJOBS customer satisfaction since March 2003 and has seen satisfaction increase from a baseline score of 71 to a current index rating of 76, outperforming both the Governmentwide and overall ACSI aggregates. ?? U.S. Merit Systems Protection Board. (2002). Assessing Federal Job Seekers in a Delegated Examining Environment. Washington, DC. This report raises questions about agencies' varying assessment capabilities and the negative outcomes of using inadequate assessment tools, i.e., poor employee selections. Such outcomes undermine merit system principles and increase the cost of Government. The report recommends that agencies cooperate with OPM in their efforts to develop valid candidate assessment tools. ?? "Software glitches force shutdown of agency hiring Web sites." GovExec.com. 18 April 2005. The article discusses a private-sector company's service interruption to customers and CTS's ability to assist them through USAStaffing. ?? General Accounting Office. (2004). Human Capital: Selected Agencies' Use of Alternative Service Delivery Options for Human Capital Activities. Washington, DC. This report recommends that the OPM Director work with Chief Human Capital Officers to share and distribute information about Alternative Service Delivery options. The report cites OPM's TMA program as an effective example of using outside expertise to deliver training and human capital solutions (pp. 22-23). The report also cites savings of $300,000 by the U.S. Coast Guard from using the CTS OAS (p. 14). ?? A-76 results ?? SAS 70 report ?? KPMG LLP. (2002) TMA Cost Analysis Services Project. Washington, DC. ?? In 2002, an entire issue of the Society for Human Resource Management's journal, Human Resources Management, was dedicated to the work of CTS's staff, with the editor commending CTS for achieving its world-class HR practices in the public sector.